This takes work from #84 and expands on it a bit. Now something like this:
scontrol reboot ASAP nextstate=DOWN <nodelist>
Will reboot a node and when it comes up it won't get marked online until NHC deems the node healthy. This only works by NOT defining the reason parameter to this scontrol command and also ensuring the nextstate is DOWN. This is especially a useful feature when doing rolling reboots using Slurm to avoid nodes coming online too quickly before some services like GPFS have had a chance to start.
There is also an addition in here to better support unexpected reboots of nodes when using ReturnToService=1.
Also treat planned nodes like nodes that are essentially idle as planned nodes are idle nodes held back for future jobs.
These patches have been deployed to Ohio Supercomputer Center for quite a while now.
This takes work from #84 and expands on it a bit. Now something like this:
Will reboot a node and when it comes up it won't get marked online until NHC deems the node healthy. This only works by NOT defining the
reasonparameter to this scontrol command and also ensuring the nextstate is DOWN. This is especially a useful feature when doing rolling reboots using Slurm to avoid nodes coming online too quickly before some services like GPFS have had a chance to start.There is also an addition in here to better support unexpected reboots of nodes when using ReturnToService=1.
Also treat planned nodes like nodes that are essentially idle as planned nodes are idle nodes held back for future jobs.
These patches have been deployed to Ohio Supercomputer Center for quite a while now.