How to make shm's fi_sendmsg behave the same with fi_inject?

[shijin-aws]

(Some context ...) I am working on the integration of using shm as a peer provider in efa, and I need to make shm do inject when application call fi_inject to efa and the peer is local. I would prefer to call shm's fi_sendmsg API with appropriate flags to make it so I do not have to use different API for inject and non-inject paths.

According to the man page:

Conceptually, this means that the fi_inject function behaves as if the FI_INJECT transfer flag were set, selective completions are enabled, and the FI_COMPLETION flag is not specified. Note that the CQ entry will be suppressed even if the default behavior of the endpoint is to write CQ entries for all successful completions. See the flags discussion below for more details. The requested message size that can be used with fi_inject is limited by inject_size.

I tried to call fi_sendmsg with flag as FI_INJECT | FI_SELECTIVE_COMPLETION &~FI_COMPLETION but it still write cq entry. Later I find enabling selective completion requires me adding FI_SELECTIVE_COMPLECTION to the flags of fi_ep_bind when I bind shm cq to shm ep. Then I could make shm's fi_sendmsg not write tx cq if the flags I passed to fi_sendmsg has no FI_COMPLETION.

Is this the correct and recommended way to make fi_sendmsg behave as fi_inject? (Or it is recommended to call fi_inject for inject path explicitly? I see fabtests and ompi do this way)

Thanks.

[aingerson]

If you want fi_sendmsg to suppress the completion, then you'll have to bind the CQ with the selective completion flag. FI_SELECTIVE_COMPLETION is not a sendmsg flag. I would recommend going through the inject call itself which is an optimized version of the sendmsg path as it omits checks to write to the CQ and minimizes the protocol selection. That said, you can definitely take the sendmsg path if you want but it does mean configuring the shm with selective completion for all calls which could potentially be a headache to coordinate.

[shijin-aws]

@aingerson Thanks for the feedback. I think I will follow your recommendation to go through the inject call itself.