Support health checks

[hjr3]

When adding a server to the pool, it should kick off a periodic health check. If the health check fails n times, then the server should not receive active connections.

Most load balancers will keep checking an inactive server and if it starts passing health checks in the future, it can be activated again. One concern I have with that behavior is that this proxy is for a dynamic environment it might not be feasible for servers to be removed from the pool. In that case, we have a servers that exist in the pool and that will never come back.

A few options:

1. Stick with normal load balancer behavior and deal with a growing list of inactive servers.
2. Remove a server after n health checks. This would put the onus on the backend servers to re-add themselves to the pool. I expect people will periodically re-add servers via cron or something. That seems annoying for people to do.
3. Nuance between a server that is unable to be connected to and a server that is able to be connected to but fails a health check. In the case where the proxy cannot connect to the server after n attempts, it would be removed permanently.

There might be other ideas too. I am wary of having a default and letting it be user configured, but maybe that is another piece to the puzzle.

[yanns]

I think we should let the user defines how to check the backends, with a good default. Ideas:

• having a global health checks activated: yes/no
• possible rules:
  • just open a connection
  • http call
  • user must define an url
  • user can define expectations (http status...)
  • based on statistics. Ex: if the last requests return a 5xx with a rate > 50 % -> down
  • each rule can be activated / deactivated

Default: http call on / must return a 2xx status

[hjr3]

First implementation landed in https://github.com/hjr3/alacrity/pull/62 . Moving this future enhancements to 0.2 milestone as it will cause large changes to the Pool.

[hjr3]

The code is in a much better spot now that it is easier to enable/disable servers from a Pool. We can now proceed.