aws-samples / aws-eda-slurm-cluster

AWS Slurm Cluster for EDA Workloads
MIT No Attribution
24 stars 7 forks source link

[FEATURE] Document how to update compute node AMIs #157

Open cartalla opened 10 months ago

cartalla commented 10 months ago

Is your feature request related to a problem? Please describe. When the compute node AMI needs to be updated, what affect does that have on running jobs? Can it be done on-the-fly?

Describe the solution you'd like At a minimum document how this should be done with ParallelCluster.

In the legacy cluster the AMI was stored in an SSM parameter that was read by SlurmPlugin.py. If the AMI changed then the plugin would drain all compute nodes using the old AMI. This would force Slurm to start idle nodes which would use the newly configured AMI. This was seamless and non-disruptive. If ParallelCluster can't do something similar, document the process and open a PC feature request with the above desired behavior. Ideally, I should just be able to update the compute AMI without stopping the cluster and have it take effect.