Open kent-cheung-arm opened 2 years ago
We have started looking more in-depth at a similar issue filed under PE-43365. Ranks would be stopped before they have hit the barrier release. And because of this, SIGCONT would be consumed when some of the ranks are continued and get stuck in their barrier.
The PALS team is looking into the root cause.
CTI in PE 22.11 will have an environment variable CTI_PALS_BARRIER_RELEASE_DELAY
to delay releasing the applications in the startup barrier. In our testing, setting it to 1
second is usually sufficient to avoid this race condition.
In moderately-sized PALS jobs, the tool sometimes attaches after the barrier even though
cti_releaseAppBarrier
was not yet called.Attaching at barrier as expected:
Attaching at
MPI_Init
after barrier:The call to
cti_releaseAppBarrier
occurred at 59.749728688 in this run.