Threading model for collectives on teams... threads that cannot order or sync?

[gmegan]

When going over potential solutions to the problem of synchronizing team creation, the issue was raised about threads that cannot order operations or synchronize.

Since teams encapsulate PE level resources, e.g. NIC resources assigned per PE, the use of the teams object itself for collectives would imply that for any threads on a PE and sharing a team object are also sharing these team resources. So, for the base case of SHMEM_TEAM_WORLD, all threads on a PE share the resources assigned to SHMEM_TEAM_WORLD.

Under the current draft, this would mean that threads sharing a PE must be able to ensure ordering of calls to collectives using SHMEM_TEAM_WORLD - including new team creation collectives.

What are the real world examples of this problem? How would threads that cannot order operations on a PE call shmem_malloc? Is this really a problem for teams to solve or a more fundamental problem that would affect all of OpenSHMEM?

[gmegan]

Here are some 3 minute sketches of how the current draft defines collective behavior and sync requirements.

bcast_scan1.pdf

bcast_scan2.pdf

[nspark]

When going over potential solutions to the problem of synchronizing team creation, the issue was raised about threads that cannot order operations or synchronize.

(Apologies that I've missed the last few meetings and like am missing the context here that would answer my questions.)

Is the concern about "threads that cannot order operations or synchronize" a limitation of the threads due to program design or something more intrinsic (e.g., the threads library itself)? If it is due to program design, is this not just an OpenSHMEM-specific instance of any sharing of non-threadsafe resources in a multithreaded program?

[gmegan]

This would be a concern about something intrinsic to the thread library that would make it incompatible with some requirement in the specification to provide ordering on routine calls.

I think what we are wanting to require for teams is the following:

• Teams routines are thread safe to whatever extent the threading level requires
• The teams object itself is not thread safe under any threading level.

So any number of threads on a PE can call team based collectives at the same time, but a given teams object can only be involved in one collective at a time on a PE. So the program must somehow ensure that if a thread wants to call a collective routine using a team object that is shared across threads (e.g. SHMEM_TEAM_WORLD), that no other thread on that PE is inside a collective routine on that team.

These requirements are all regarding the local PE state only.

As for ordering, that comes down to the global state of the team, not the local thread state. Do we have ordering requirements on collectives operations on a team? As of the last call, it is not specified if we do or not, since we are considering teams safe to reuse in collectives as soon as they return, without an intervening team sync operation.

In the multithreaded case, say two threads both use SHMEM_TEAM_WORLD, one to call reduction, one to call broadcast. Say they put the collective operation in a protected section so SHMEM_TEAM_WORLD is not in use by more than one thread at once. We could have this ordering of atomic sections:

PE0 [thread0: broadcast] [thread1: reduction] PE1 [thread1: reduction] [thread0: broadcast]

Is this supported? According to the spec, the broadcast can return on PE0 even if PE1 has not finished the broadcast. And there is no requirement that PE1 has even entered the broadcast for PE0 to complete the broadcast. It is really just dumping data into some dest buffer that the user is responsible for providing and protecting. Similar for the reduction and all teams collectives; global completion or even global participation is not a requirement. So, If the threading library can only provide protected section, but not ordering, can the OpenSHMEM library be expected to deal with this case of random ordering of collectives on a team?

What if instead of two threads with a reduction and a broadcast, it is N threads with N broadcasts that can occur in any order across PEs? Clearly the threads are providing their own private source and dest buffers, but is there some resource associated with the team that cannot be shared across many ongoing broadcast operations at the same time?

[gmegan]

This issue is resolved in discussion that collectives over teams retain the same model as collectives over active set with shared pSync. Team object is not thread safe. Collectives are matched across a team by ordering on PEs since we do not have tags or other mechanism to match collective calls on a specific team. Text should be added to collectives intro.

[gmegan]

Close per PR #76