AMDResearch / omnistat

https://amdresearch.github.io/omnistat/
MIT License
5 stars 0 forks source link

with multiple node allocation on SLURM, #GPUs is showing only one instance #85

Closed GowriShankarEAAS closed 2 months ago

GowriShankarEAAS commented 2 months ago

with multiple node allocation on SLURM, the dashboard is showing only on system GPUs and not all the GPUs allocated to the job. in the example shown below, we need see 8 GPUs but showing only GPUs. Kindly suggest on this

image

koomie commented 2 months ago

You likely need one more change to your local SLURM config to force the prolog to run at allocation time as opposed to the first srun invocation.

Add the following setting to your slurm.conf and restart the controller.

PrologFlags=Alloc
GowriShankarEAAS commented 2 months ago

Thank you Jorda. will update config file and check

GowriShankarEAAS commented 2 months ago

Thank you Karl. Will update the config file and check

GowriShankarEAAS commented 2 months ago

Issue resolved after updating the flag. PrologFlags=Alloc