Implement naive health checks

[hjr3]

This my first pass at a health check. I thought about re-using an existing tokio Core, but if we use tokio-pool (or something like it) then we have a Core per thread. I don't see how we can get away with not having a dedicated thread to do health checks.

We may need to introduce some basic configuration at this point so the uri path for the health check is not hard-coded. I have to think about that a bit too.

[hjr3]

I am looking at storing a Server object in the pool. This will allow us to keep track of states, such as Active, Down, etc. This will allow the health check to keep checking a down server. It should also allow me to record some stats on the server too. I am working out the borrow details. For example: how does a normal request borrow a Server from the pool, make a request and increment some stat counters while other threads may be asking for the same Server. Especially when the health check thread wants to mark it Down.

[yanns]

Yes, I had the same idea. I was thinking if we should store the information from the user (Server config) and the runtime information at different places or in a mutable structure.

I was working on secure connection to backends, and I also need more information (https vs http). And we will need more information in the future (health check config, routing?)

[hjr3]

The health checks are working, but naive. They always check / every 1 seconds and will remove the server if a check fails. I want to do more, but I realized that Pool is the wrong abstraction. Currently, we get a Server from the Pool. This causes a lot of problems with Rust's ownership model. Instead, we should be passing the request into the Pool. This would hide the logic of how a Server is chosen, make sure the same Server is used for pipelined requests, centralize the place where stats will be counted and enforce the health checks.

As that is a pretty big change, I do not want to hold this up. This one of the last item for a 0.1 release.