Open Akshay-Venkatesh opened 4 years ago
@Akshay-Venkatesh seems like send type and recv type are created with different 'count' (1024 vs 512), is this expected?
i don't think OpenMPI supports it, and it works with tcp/ud only because their segment size is a power of 2
@Akshay-Venkatesh seems like send type and recv type are created with different 'count' (1024 vs 512), is this expected?
@yosefe sorry missed this earlier.
The send side stride and receive side strides are different. Send side is stride of 512 elements and receive side has a stride of 1024 elements but the sizes in terms of bytes match at both ends and so do the corresponding basic element types. Basically (512 sizeof(double) ) 512 bytes at both ends but receive side has gaps because stride is 1024 between 512-blocklengths. Also, MPI semantics does permit this usage if I'm not mistaken. page 112, line 28 of the MPI 3.1 states the following:
Type matching is defined according to the type signature of the corresponding datatypes, that is, the sequence of basic type components. Type matching does not depend on some aspects of the datatype definition, such as the displacements (layout in memory) or the intermediate types used.
Please correct me if you think I'm mistaken here.
Whether this was stride difference was intended by the user is something I can confirm. The user who reported this issue also reported the following cases work:
@Akshay-Venkatesh this is because the segment size has changed with UCX v1.8.0. Anyway, i think this is an OpenMPI issue and not UCX.
@yosefe taking in account that OMPI works with several flavors of PML (including an older UCX version), the issue might be on UCX side. More specifically, this error indicates that a partial data has been received, something that the datatype tries hard to prevent. So, either some bytes got lost, or the sender is doing a fragmentation that is not coming from the OMPI datatype engine.
@bosilca i think what happens here is that receive stride is larger than sender stride, so the packer side packs by a 512-byte granularity (for example), but the receive expects to unpack by 1024-byte granularity, and fails. Does OpenMPI support such case?
Definitively, as @Akshay-Venkatesh mentionned earlier, communications with datatypes with the same type signature, aka. same sequence of predefined types ignoring the memory displacement, is an MPI requirement. In fact, in terms of communication MPI forbids little: truncation (receiving less than sent), and type conversion (send a float and receiving an int as an example).
Vectors with different strides is really one of the simplest cases. to help understand what is not going on correctly, have your OMPI build in debug mode and provide the following MCA params, mpi_ddt_unpack_debug and mpi_ddt_pack_debug set to a non-zero value. You should get additional info about how the pack/unpack works on both sides.
@bosilca Added debug output of the test from pack/unpack here: https://gist.github.com/Akshay-Venkatesh/e33a9088ec9cfeb2cd052c106dabd821#file-ddt-debug-txt
Describe the bug
The bug was originally reported to not occur with UCX 1.7 but I've not confirmed this. I'm able to repro on 1.8.x and master
Steps to Reproduce
Makefile and source here: https://gist.github.com/Akshay-Venkatesh/e33a9088ec9cfeb2cd052c106dabd821
Setup and versions
Output
Passing case:
Failing case: