Replace costly call to 'scontrol show config' with pmix env vars

NVIDIA / enroot

A simple yet powerful tool to turn traditional container/OS images into unprivileged sandboxes.

Apache License 2.0

649 stars 94 forks source link

Replace costly call to 'scontrol show config' with pmix env vars #204

Open nvcastet opened 3 months ago

nvcastet commented 3 months ago

scontrol show config issues RPC calls to master node. When Slurm is configured with per-user RPC rate-limiting (rl_enable), the command can be throttled causing a large variance in rank start times since it is called in a hook executed per rank. This PR leverages env variables to get the info for the PMI hook instead of calling scontrol show config.

CC @flx42 @3XX0

3XX0 commented 1 month ago

I would have to look at it again, but there were good reason not to do this.