Symmetric inplace operations

[lukashuebner]

We have to think about three cases:

1. The user provides the same buffer to send_buf(...) and receive_buf(...). Possibly via a Span<> object or otherwise encapsulated. We could decide to either not support this or to require the passed objects to have an underlying_buffer_equals_that_of(...) if the user wants the possibility for us to detect this. Note, that MPI forbids passing pointers to overlapping memory regions as send and recv buffer; if you want this, you have to use MPI_INPLACE. I would prefer not to use operator== for the comparison, as two object might be unequal even if they have the same underlying buffer (e.g. a std::vector and a Span<> pointing to the same memory region).
2. The user explicitly specifies kamping::in_place or MPI_INPLACE as the receive buffer. Imho, it would be nice to support specifying kamping::in_place as a tag, e.g., comm.scan(send_buf(value), kamping::in_place);
3. The user assigns the recv_buf to the same variable used as the send_buf, i.e., value = comm.reduce(send_buf(value));. This should always yield the expected result, and we can do nothing to optimize this anyway; so we do not need to consider this further.

[Hespian]

1. Can't we just compare .data() for the first case?
2. I like that proposal. It would be even cooler if we could do that in
3. Seems sketchy and I'd have to think about this in more detail to see if it would work. I would advice against it ;)

[lukashuebner]

I like that proposal. It would be even cooler if we could do that in

In ...? :-)

Can't we just compare .data() for the first case?

I thought about that, but wasn't sure if every object has a .data() member. But it probably has, as we're using .data(...) to get the pointer we pass to the MPI function. So :+1:

[Hespian]

In ...? :-)

Hm.. I forgot what I wanted to say there :/

I thought about that, but wasn't sure if every object has a .data() member. But it probably has, as we're using .data(...) to get the pointer we pass to the MPI function. So +1

The important part is that the parameter object always has .data(), which it does because we wrote it ;)

[Hespian]

I like that proposal. It would be even cooler if we could do that in

Actually, let me fix that: I like the idea (which is very close to how the normal MPI interface does it), but I don't like passing a tag without the named parameter syntax ;)

[lukashuebner]

One of the reasons we decided against the (imho) way prettier send_buf = value interface was exactly such that we can use tags instead of the (imho) ugly kamping::in_place() syntax. Also, tags are “more C++” than our named parameters.

[Hespian]

Well yes, but named parameters are always used in kamping. I don't recall that being a reason against the = syntax (in fact, I don't see a difference between a tag in the ()-Syntax and in the =-Syntax.

[niklas-uhl]

Operations supporting inplace:

• [x] alltoall (#626)
  • data is taken from recv_buf and replaced by received data
• [x] allgather (#632)
  • MPI_IN_PLACE must be passed on all ranks
  • each rank has to store it's contribution to the data the correct final destination, send parameters are ignore
• [x] allreduce (#635)
  • pass MPI_IN_PLACE on all processes
• [x] scan (#636)
  • same as allreduce
• [x] exscan (#636)
  • same as allreduce
• [ ] allgatherv
  • same as allgather

Asymmmetric: moved to #666

[lukashuebner]

Thank you, Niklas! :-)

A few small comments:

• Does the MPI standard enforce, that MPI_INPLACE may only be passed on one or on all ranks (e.g., for allreduce)?
• I'd prefer if the user simply passed kamping::inplace and not recv_buf(kamping::inplace).
• Do we explicitly check if the same two buffers are passed to send_buf and recv_buf (either to automatically do the operation inplace or to emit an error message); not zero-overhead

[niklas-uhl]

• Does the MPI standard enforce, that MPI_INPLACE may only be passed on one or on all ranks (e.g., for allreduce)?

This depends on the operation, see the list above

[niklas-uhl]

• I'd prefer if the user simply passed kamping::inplace and not recv_buf(kamping::inplace).
• Do we explicitly check if the same two buffers are passed to send_buf and recv_buf (either to automatically do the operation inplace or to emit an error message); not zero-overhead

I ditched the idea of providing inplace explicitely (see #626). The same semantics can be achieved with a lot less confusion by passing send_recv_buf and no send_buf and recv_buf. When a send_recv_buf is passed, we detect that inplace should be used, no implicit comparison of send and recv buffers. If users pass them explicitely instead of using a send_recv_buf, they might have a reason to do so ...

[lukashuebner]

• Does the MPI standard enforce, that MPI_INPLACE may only be passed on one or on all ranks (e.g., for allreduce)?

This depends on the operation, see the list above

I see, the list is what the MPI standard enforces; I read it as what we want to support. :+1:

[lukashuebner]

• I'd prefer if the user simply passed kamping::inplace and not recv_buf(kamping::inplace).
• Do we explicitly check if the same two buffers are passed to send_buf and recv_buf (either to automatically do the operation inplace or to emit an error message); not zero-overhead

I ditched the idea of providing inplace explicitely (see #626). The same semantics can be achieved with a lot less confusion by passing send_recv_buf and no send_buf and recv_buf. When a send_recv_buf is passed, we detect that inplace should be used, no implicit comparison of send and recv buffers. If users pass them explicitely instead of using a send_recv_buf, they might have a reason to do so ...

Sounds good. It's harder to use the API wrong this way. :-)