Simple signaling system for pool health control

[amv]

Currently we expect it to be outside of pollbal's scope to manage that back ends providing a certain service do not all exit used pools simultaneously. Normally the updates affecting this managed using remote orchestration tools like Capistrano but they pretty much depend on a very well defined system which is functioning in a normal state.

The main strength of pollbal is that the backends behind each HTTP service manage their own state completely autonomously. They communicate about their state using a simple pull-style interface which allows them to know absolutely nothing about the infrastructure that manages the routing and availability. This allows us to install several pollbal instances around the world without changing the backends at all.

Having several pollbal instances around the world with varying network connections and access policies makes defining, verifying and maintaining a "normal" state a non-trivial operation. Instead of aiming to have this well defined normal state before all updates, it might be more reasonable to use the state which is gathered as part of pollbal's normal operations to provide adequate information to backends on when it would be ok to go down for an update and when it would surely lead to a catastrophic failure.

To make this possible I suggest three simple alterations to the normal operation of pollbal:

1. Redefine the pull interface to convey information about future state
2. Allow configuring a pool to detect a critical condition
3. Push warnings to connected backends when critical conditions are detected

The first thing would be to change the /pool endpoint data structure from

[pool] [version]?

to

[pool] [serving | leaving]? [version]?

So when a backend is getting ready for an update, it would signal this to all the pollbals by returning "servicename leaving v_3" from /pool. This would signal to pollbal that this backend is still in the pool but it is planning to go down in the near future. Normally the response would just be "servicename serving v_3".

The second thing would be to allow configuring a pollbal pool to contain a "warning treshold". Each active service which is needed to maintain the service level defined as having "at least the warning treshold of online backends" would be sent a warning message indicating that this server should not go down for updates for a while. The warning could be implemented simply by sending a simple http request to the backend after each health check carried out by pollbal:

POST /pool_warning

For example if the warning_treshold would be set to 1, pollbal would after each availability check make sure that there are more than 1 backends connected to each of the pools advertised versions. If only one backend is connected to the pool for a certain version, that backend is immediately notified with a PUSH request to /pool_warning after each check (usually once every second).

If some backend responds to a pool request by indicating that it is leaving a queue soon, pollbal would check if the warning treshold is still met without this backend and if not, all remaining backends would be immediately warned. The service leaving the pool would not receive warnings and could continue with it's update as long as there still are at least warning_treshold of services left at the pool.

If however several backends would simultaneously indicate that they are leaving the pool, meaning that the number of "serving" backends would drop below the warning treshold, then also the backends indicating a "leaving" status should be immediately warned. This should then hint the backends to cancel the pending update and postpone it to a later time.

The implementation of this on the backend machines could be as easy as hooking a CGI script to "touch" an "update lock file" each time a POST request to /pool_warning is received. The update scripts for backends would work in the following way:

1. sleep for rand(30) seconds
2. check for recent (<10s) mtime on update lock file. if exists, start again from step 0.
3. write "servicename leaving vx" to the /pool file
4. sleep for 10 seconds
5. check for recent mtime (<10s) on update lock file. if exists, write "servicename serving vx" back to the /pool file, and start again from step 0.
6. clear the /pool file, sleep for 5 seconds and take service down for update.

This way all the backends could receive the same signal to start updating but the updates would start on different times, maintaining the pool size at or above warning treshold during the update cycle.

[amv]

Current implementation has

[serving | leaving]? [pool] [version]?

instead of

[pool] [serving | leaving]? [version]?

This makes the first verb mandatory even if versions are not used. I would like it to be possible to have pool responses that contain only the pool name and have "serving" be the default (that was why I had the question mark there).

If one wants to use versions, then "serving" should be present.

Renaming "serving" to "ok" is ok. I would like to leave "leaving" as it is.

[tstarck]

Note to self: pool [ status [ version ] ]

[amv]

I would like to leave a note that we should first implement only the "leaving" functionality (which is crucial for controlled updates) and implement the "/pool_warning" functionality later.

[amv]

In fact scratch that..

If there is no "/pool_warning", the "leaving" is almost the same as just returning an empty result for the /poll. It just leaves the system online for a couple more seconds in the unfortunate case of too many nodes trying to go down at the same time.

So this is not a priority for now. Let's just try to make it possible to do automatic updates that are controlled elsewhere so that the updates do not clash (like for example with an github hook autoupdate delay).