Handling handleNodeFenceError

[bronhaim]

after reaching giveup retries we need to retry triggering all step jobs. giveup retries are raised each jobs polling if still not done successfully

[rgolangh]

btw, how do you keep jobs from colliding? i.e a job to fence a host, and a job to un-fence it? (or maybe cordon or uncordon is a better term)

[bronhaim]

the fence and un-fence scripts are different and not related to each-other. each script can do the opposite action - I can't prevent it. The admin sets the methods to run in each step, such declaration that can cause collision is less likely

[bronhaim]

once job failed we need to set anti-affinity to avoid running job on same node.

• check if jobs already contain such option (on fail to try on different node)