On Summit (OLCF), when I have 80 servers create a group with a 1 second period, I get errors like these:
SWIM dping req recv error -- group 7706329477877535173 not found
SWIM dping req recv error -- group 7706329477877535173 not found
SWIM dping req recv error -- group 7706329477877535173 not found
SWIM dping req recv error -- group 7706329477877535173 not found
SWIM dping req recv error -- group 7706329477877535173 not found
SWIM dping req recv error -- group 7706329477877535173 not found
SWIM iping req recv error -- group 7706329477877535173 not found
SWIM iping req recv error -- group 7706329477877535173 not found
SWIM dping req recv error -- group 7706329477877535173 not found
SWIM iping req recv error -- group 7706329477877535173 not found
SWIM iping req recv error -- group 7706329477877535173 not found
SWIM dping req recv error -- group 7706329477877535173 not found
SWIM iping req recv error -- group 7706329477877535173 not found
SWIM dping req recv error -- group 7706329477877535173 not found
SWIM dping req recv error -- group 7706329477877535173 not found
SWIM iping req recv error -- group 7706329477877535173 not found
SWIM iping req recv error -- group 7706329477877535173 not found
I actually don't know with certainty if they indicate critical errors or just excessively noisy output, though, because I had a bug in the script that was meant to subsequently use the group. I think the group exited though, based on execution time.
The problem was resolved by increasing the period length from 1 seconds to 3 seconds.
The execution command the triggered it looked like this:
On Summit (OLCF), when I have 80 servers create a group with a 1 second period, I get errors like these:
I actually don't know with certainty if they indicate critical errors or just excessively noisy output, though, because I had a bug in the script that was meant to subsequently use the group. I think the group exited though, based on execution time.
The problem was resolved by increasing the period length from 1 seconds to 3 seconds.
The execution command the triggered it looked like this:
And the example code can be found at https://github.com/mochi-hpc-experiments/ssg-benchmarking/blob/carns/dev-1rank-dump/ssg-launch-group.c.