search

ofiwg / libfabric

Open Fabric Interfaces
http://libfabric.org/
Other
561 stars 378 forks source link

API: Add mode or capability flag to determine if remote CQ data consumes an fi_recv operation #666

Closed patrickmacarthur closed 9 years ago

patrickmacarthur commented 9 years ago

(Separating the general API issue from issue #663)

In verbs, receiving a remote CQ data event for a WWI or atomic operation consumes an IB recv WQE. However, we had discussed early on in the requirements gathering phase that there is really no good reason for this to apply to all providers since there is no relationship between the remote CQ data and the receive operation that it consumes.

This should be exposed as a mode/capability flag, since an application will want to know this prior to posting receives, since it must allocate more storage and post more receives if receiving a write or atomic with remote CQ data will consume an fi_recv.

sayantansur commented 9 years ago

I agree with this requirement in general. One shouldn't have to consume a receive buffer if nothing is being placed in the buffer, rather just immediate data is being delivered.

However, from the providers point of view, it might be better if at least some resource were attached to the EP that enables it to generate the completion. There are some resources that are being consumed when a message comes in (e.g. message headers that need to go somewhere).

Is it legal to post a fi_recv with zero length buffer? I wonder if we should have a persistent operation that collects the immediate data? The persistent operation removes the application requirement to post recvs over and over again.

shefty commented 9 years ago

RMA write operations do not consume receive buffers. The headers go into la-la land. So, I do think it's reasonable for an RMA write with immediate data to NOT consume a receive buffer, as the headers are already handled by the HW.

0-length receives can be posted. But this only works if the user has strict control over when an immediate data message is received relative to normal messages. If only immediate data messages are received, then the provider can handle the receive queue internally.

Providers could repost receive buffers that are consumed for immediate data purposes, but an application may need to know that its buffers will be returned to it out of order from how they were posted. The interaction with FI_MULTI_RECV also needs to be defined.

shefty commented 9 years ago

mode bit defined