gmegan
commented
5 years ago This seems to be related the way the discussion has gone around whether we can allow collective supporting teams to be created from NO_COLLECTIVE teams and what the utility of NO_COLLECTIVE teams would be if that is not possible. If one cannot split a NO_COLLECTIVE team into a team supporting collectives... then what is the utility of these objects and what are they used for?
Also related, since point to point is based around contexts, and the only way to do rma/atomic using team numbering is to create a context from a team, where does this leave the status of teams that are NO_COLLECTIVE but have requested some num_contexts=N? Should this even be allowed? Is there any use of a team that supports rma/atomic using a team based context but does NOT provide shmem_team_sync (since it is NO_COLLECTIVE)?
Further, once team based heaps are added (not in this proposal), this will be one more resource that will be added during team creation. Can there be a team with just a heap and no contexts or collectives? Seems like that might actually be useful for setting up distributed memory pools.
I mention heaps because as it stands, the team has some set of resources and they have to be tested for. If there are types, then it seems like in the worst case, we might end up adding more types to cover the different legal combinations...
shmem_set_t = just indexing and shmem_team_translate support shmem_team_t = team with collectives, some num_contexts=N, and a heap shmem_team_ctx_t = team with no collectives but just provides private context on some PEs shmem_team_heap_t = team with just a heap but no collectives or contexts shmem_team_comm_t = team with collectives, num_contexts=N, but no heap
However, if the only real world library issue is, for example, needing to know if the team can be used in shmem_team_sync, then all these other types would not be needed, and it would just be a matter of two types...
shmem_team_t = ability to shmem_team_sync shmem_set_t = no ability to shmem_team_sync
Then, all of the other resources (contexts, heaps, etc) could be part of a config that gets tested.
In reviewing the recent WG meeting minutes, I've begun to further question the utility of the NO_COLLECTIVE option.
With respect to library design (i.e., libraries built on top of OpenSHMEM), the use of any team as a function argument may essentially require that function to check whether the team supports collectives before it is used at all. This is maybe nudging me toward wanting distinct types for objects that represent sets of PEs without attached resources (e.g., teams with NOCOLLECTIVE or the earlier ORNL "sets" proposal) and objects with attached resources and functional capabilities (e.g., "regular" teams or the ORNL "groups").