refactor(healthcontroller): generate only one add/remove operation per cycle

Currently, the health controller is performing both rolling updates and autoscaling. We can roughly divide these responsibilities among these functions executed by the operation:

performRollingUpdate: generates a pair of add and remove operations based on rolling update constraints (maxSurge and maxUnavailable = 0)
ensureDesiredAmountOfInstances: generates either an add or remove operation to move the current state equal to the desired

In a lot of cases, one cycle will generate conflicting operations: autoscale will try to remove X amount of rooms but rolling update only wants Y amount of rooms deleted. Moreover, autoscale deletion is based on its policy (currently only occupancy) but rolling update wants to delete based on the version of the scheduler. Thus, to streamline and simplify this process the refactor aims to split the responsibility and let only one piece of logic handle the creation or removal of rooms.

The work of this PR can be summarized in:

1. Split the logic, inside `healthcontroller/execute.go`

So that we have one logical abstraction to compute the desired number of rooms based on autoscaling and rolling update (if ongoing), and another logical abstraction responsible for fixing the drift between current and desired by enqueuing add or remove operations

All logic that is responsible for computing the desired amount can be refactored to a different operation, outside of execution worker, later on. All those methods are not receivers of the Execute and the dependency is injected by their arguments in the functions:

GetDesiredNumberOfRooms
IsRollingUpdating
ComputeMaxSurge

2. Refactor the Remove Rooms operation to sort by version when deleting a number of game rooms

Currently, the system will sort by game room status, however, we want to take the sorted rooms and apply a sort by version on Pending, Ready, and Occupied rooms.

Given this set of rooms with their status + versions: [Error1v1, Error2v2, Pending1v1, Pending2v2, Ready1v1, Ready2v2, Occupied1v1, Occupied2v2]

Currently, the sort by priority/status will return the above list, however, we want to ensure old scheduler versions (v1) are deleted first (Kubernetes deletes pods based on creation time, so it will be similar). Thus, the new sorting version will become:

Before: [Error1v1, Error2v2, Pending1v1, Pending2v2, Ready1v1, Ready2v2, Occupied1v1, Occupied2v2]
Now: [Error1v1, Error2v2, Pending1v1, Ready1v1, Occupied1v1, Pending2v2, Ready2v2, Occupied2v2]

If there's only one active version of the scheduler in the system, then we skip the additional sort.

3. Simplified calculations of desired when autoscaling + rolling updating

As a baseline, the system will always use the desired from autoscaling (or roomsReplica if no autoscale is defined). However, if we have a major version of the difference between schedulers of current rooms in the system, then we are in a rolling update and we should consider the maxSurge.

Maestro Rollout Strategy Design Doc

4. Add unit tests with ALL rolling update cycles

Create a matrix of test cases with table tests, validated in spreadsheets and with the desired behavior that each health_controller operation should take during a rolling update. There are 4 base cases: stable occupancy, occupancy increased, occupancy decreased, and stable occupancy but only 75% of new rooms becoming active.

Spreadsheet reference with the desired cases: https://docs.google.com/spreadsheets/d/13grNuk3os7mOesnpROqLOVTSWiGpgvi5JUgWUB3Z428/edit?usp=sharing

topfreegames / maestro