Per README.md additions:
For some regions and VM sizes, some subscriptions may report an incorrect number of GPUs. This value is controlled in /opt/azure/slurm/autoscale.json
Note that here it is saying "For all VM sizes in all nodearrays, create a resource called slurm_gpus with the value of the gpu_count CycleCloud is reporting".
A common solution is to add a specific override for that VM size. In this case, 8 GPUs. Note the ordering here is critical - the blank select statement will set the default for all possible VM sizes and all other definitions will be ignored. For more information on how scalelib default_resources work, the underlying library used in all CycleCloud autoscalers, see the ScaleLib documentation
Simply run azslurm scale again for the changes to take effect. Note that if you need to iterate on this, you may also run azslurm partitions, which will write the partition definition out to stdout. This output will match what is in /etc/slurm/azure.conf after azslurm scale is run.
Fixes: #198
Per README.md additions: For some regions and VM sizes, some subscriptions may report an incorrect number of GPUs. This value is controlled in
/opt/azure/slurm/autoscale.json
The default definition looks like the following:
Note that here it is saying "For all VM sizes in all nodearrays, create a resource called
slurm_gpus
with the value of thegpu_count
CycleCloud is reporting".A common solution is to add a specific override for that VM size. In this case,
8
GPUs. Note the ordering here is critical - the blankselect
statement will set the default for all possible VM sizes and all other definitions will be ignored. For more information on how scalelibdefault_resources
work, the underlying library used in all CycleCloud autoscalers, see the ScaleLib documentationSimply run
azslurm scale
again for the changes to take effect. Note that if you need to iterate on this, you may also runazslurm partitions
, which will write the partition definition out to stdout. This output will match what is in/etc/slurm/azure.conf
afterazslurm scale
is run. Fixes: #198