Ambiguity in shmem_ctx_quiet Completion and Visibility

[manjugv]

Assuming context is created by shmem_team_ctx_create operation, on return from shmem_ctx_quiet, the completed operations could be visible:

• (1.) to all PEs in the job, or
• (2.) only to PEs in the team that was used to create the context

What should be the behavior of shmem_ctx_quiet, 1 or (2)?

[manjugv]

Notes from the WG meeting (Jan 10th):

Both approaches have advantages, (1) can have performance benefits for some architectures (GPUs) while (2) providing simplicity to the user. However, the OpenSHMEM memory model doesn’t support scoped memory allocation - scoped completion without scoped allocation introduces asymmetry. Given the constraints of the current OpenSHMEM memory model, one of the resolutions is to introduce a flag to shmem_ctx_quiet t that captures both behaviors.

Next Step:

• Proposal to change shmem_ctx_quiet to include a flag that captures behavior (1) and (2).

[manjugv]

@jdinan @nspark Please feel free to add/edit, if I missed any nuances from today's discussion.

[naveen-rn]

I'm not sure how can (2) be implemented? Irrespective of the TEAM being used, every PE would know the targets that were used for communication using a particular context. Wouldn't getting those operations flushed to target PEs memory make it globally visible to every PE in the job. Not, sure how could we get global visibility on a set of PEs in the job?

Is there an implementation specific feature available for this?