As recommended by @hariharan-devarajan, we should add to and reorder operations in the DTL to cause an acknowledgement (ACK) message to be send back to the consumer over RPC just before communication starts. This will allow us to better detect errors in the DYAD module/service from the consumer (which increases resiliency), and it will allow us to stop using the UCX tag probe mechanism.
As recommended by @hariharan-devarajan, we should add to and reorder operations in the DTL to cause an acknowledgement (ACK) message to be send back to the consumer over RPC just before communication starts. This will allow us to better detect errors in the DYAD module/service from the consumer (which increases resiliency), and it will allow us to stop using the UCX tag probe mechanism.