[5.9] Prevent further scale in/out process when a previous one failed, independently of the lock/unlock state of the config

[mathieucarbou]

Overview

• Introduced some validations to prevent scale in/out operation to be performed if one failed previously
• Introduced a repair mode for the user to allow the operation to proceed with config-tool repair -force allow_scaling
• Refactored the lock/unlock logic in attach and detach to correctly catch exceptions and add markers to deny any further scale in/out in case of errors
• Introduced some flags in the diagnostic output to show if scale in/out process is allowed or not
• Refactored the inheritance logic for attach and detach between OSS and EE to consolidate the lock/unlock/marker logic at one place and only let EE override the difference in behaviour

Scale in flow

detach command (CLI)

1. validation: fails if we find: a deny scale in marker
2. lock
3. trigger rebalancing
   • on failure:
     • try to place a marker deny scale in to prevent replaying the detach
     • try to unlock

on rebalancing success (server-side)

1. detach the stripe
2. unlock on nomad tx success

on rebalancing failure (server-side)

1. place a deny scale in marker
2. unlock

Scale out flow

attach command (CLI)

1. validations
   • fails if we find: a deny scale out marker
2. lock
3. attach
   • on failure:
     • try to place a marker deny scale out to prevent replaying the attach
     • try to unlock
     • in any case, either a marker is placed or the config is kept locked
4. trigger rebalancing on nomad success
   • on failure:
     • config is kept locked

on rebalancing success (server-side)

1. unlock

on rebalancing failure (server-side)

1. TRY detach
2. FINALLY place a deny scale out marker
3. FINALLY unlock

Why adding a marker if the attach or detach CLI fails ?

Because attach and detach are triggering 2 Nomad tx to lock and unlock (discovery/prepare/commit) and replaying would cause 2 problems:

• un-necessary append-log entries would fill the append-log
• un-necessary nomad transactions triggered would increase the chance to collide with a concurrent user transaction which aims are doing a valid config change or repair

Repairing

We can re-allow a scale op to be retried by running:

config-tool repair -force allow_scaling

[mathieucarbou]

@mobasherul @chrisdennis @jhouserizer : FYI, I've updated the description above to show the exact flow and error handling and how the markers work.

[mathieucarbou]

@mobasherul @chrisdennis : ready for review.