rhs
commented
9 years ago One problem to be aware of that people have run into with this strategy is that because haproxy is deployed with every client in this architecture, if you have say 100 clients, then you will get a storm of healthchecks coming from all the haproxies deployed in front of those 100 clients. I know of some smartstack deployments where they have had to go to significant trouble to deal with the problem and consequently are inclined to move away from healthchecks.
Because Watson is a sibling process to the service process it is incapable of properly checking that connectivity can be established between a remote client and the service. For example, Watson will dutifully report that a service is working but in a situation where an inbound port is closed due to misconfiguration then client traffic will be rejected regardless. Therefore, we need connectivity checking which is checking that will occur from a client to the remote service. In a situation where the remote service is unavailable from the client then traffic will not be routed to that service.
One promising solution which we should explore is to use HAProxy and it's built in health checking facilities from the client side. This is how SmartStack solves this problem.
https://github.com/airbnb/smartstack-cookbook