Currently, the tags for UCX's tag-matching communication are generated with:
tag = (producer_rank << 32) | consumer_rank
The benefit of this approach is it makes use of information from Flux to create a tag unique to the producer and consumer applications. However, the downside is that multiple communications between the same producer/consumer pair will not use a unique tag, which introduces potential issues with UCX tag matching.
Instead of the current approach, we could maintain a 32-bit counter (or possibly larger) in the module. Using this counter, the module could generate tags using:
tag = (producer_rank << 32) | counter
This tag will be unique for a particular transfer that is outgoing from a particular producer module. As a result, all transfers will have a unique tag until the counter overflows.
Yet another suggestion from @hariharan-devarajan
Currently, the tags for UCX's tag-matching communication are generated with:
The benefit of this approach is it makes use of information from Flux to create a tag unique to the producer and consumer applications. However, the downside is that multiple communications between the same producer/consumer pair will not use a unique tag, which introduces potential issues with UCX tag matching.
Instead of the current approach, we could maintain a 32-bit counter (or possibly larger) in the module. Using this counter, the module could generate tags using:
This tag will be unique for a particular transfer that is outgoing from a particular producer module. As a result, all transfers will have a unique tag until the counter overflows.