Fix issues with slurm daemon startup when adding nodes

[sjpb]

Fixes https://github.com/stackhpc/ansible-role-openhpc/issues/84 - please read comments there describing issue!

The fix here is to:

• Change the handler which reloads either slurm daemon into one which restarts first slurmctld first then slurmd, in that order. (the reload handler is still used for some cases, but it's moved after restart incase both happen)
• Flush handlers before the final task to set slurmctld/slurmd state, so that any slurmctld restart has definitely happened before ensuring slurm daemons are started.

This is a change in behaviour - changing the slurm.conf for any reason will now restart all the daemons.

NB: the check for this (check10) fails on idempotence - that is not a "real" failure and is due to the fact the converge part of the test has to modify the cluster to test adding a node ... needs fixing but that's complex.

[sjpb]

Ok I don't think this is good enough. It still doesn't restart the daemons on ALL nodes which is what the slurm docs state is required on adding/removing nodes. Currently restarts slurmctld and starts slurmd on the new node (only). It appears to work but I think it's probably fragile.

[sjpb]

NB: it's probably impossible to guarantee restarting all the slurmds as they might not be in the play!

[sjpb]

@JohnGarbutt , @jovial what do you think please? It works but a) is pretty complex in terms of ordering etc and b) requires the play adding the nodes to run on the entire cluster. I'm kind-of tempted to (as JG suggested) rip out all the slurmctld/slurmd handlers, just leaving the munge/slurmdb ones, and potentially add some plays for:

• scontrol reconfigure - only needs control node in play, works on most parameter changes
• all daemon reload - needs all nodes in play, handles any changes, will do it in the right order. (we had these as soft- and hard- reconfigure in the actions branch but we can structure it better than that actually)

[sjpb]

Note that this PR means that:

• slurmd now only has a restart, handler, not both reload and restart handlers. The use of a single handler makes ordering better and avoids a bounce on node creation. In configless mode the only thing triggering a reload was rewriting the unit files when entering this mode which should only really happen once (on node creation), so I can't see the use of restart instead here being an issue.
• slurmctld is now restarted on all config file changes. The previous release did a reload for all config changes, which was actually wrong for cluster size changes but got covered by the "ensure daemons are running" action at the end. The change to always restart is is not quite ideal as it interrupts the scheduler loop even for parameter changes. However fixing it would either a) need working out whether the config file changes to be applied require a reload or a restart, or else b) trying a reload and then trying a restart if that fails (noting this slurm bug means reload never appears to fail. Could be a future enhancement if this is considered an issue.
• All slurmd's (in the play) now get restarted for cluster size changes which is the correct behavior. Without this PR there is an (unidentified) bug when configless as running slurmd's won't get restarted.
• There is no need for a separate "restart daemons" step now when using the role to change cluster size.