search

airbnb / synapse

A transparent service discovery framework for connecting an SOA
MIT License
2.07k stars 251 forks source link

Be smarter about initial reconfigure #78

Open dcosson opened 10 years ago

dcosson commented 10 years ago

Synapse is designed to re-write the haproxy config & reload haproxy when it starts (set here).

This means that any time synapse is restarted there is a period between when it starts and when the watcher first registers where the defaults are being used (and if there are no defaults you will return 503's from haproxy).

I can see why you would want an initial reconfigure so that any time you restart synapse you know any changes unrelated to backends will get picked up, but it seems like it should be smarter and wait until all watchers have checked in once before doing its initial reconfiguration. I'm happy to submit a PR if this sounds reasonable.

jolynch commented 9 years ago

@dcosson Yea this has been somewhat of an annoying bug to us as well. We mitigated it slightly at Yelp by reducing the amount of zookeeper connecting we do and using the state_file so that at the very least as soon as the watcher gets information we can quickly enable the backend over the stat socket, but this just shortens the window of sad.

How are you thinking about implementing this? The watchers all live in separate threads, but perhaps we can communicate back to the main thread a la reconfigure?

jolynch commented 9 years ago

Now that I'm looking at it, the zookeeper watchers do setup before returning because they don't use a Thread.new in the start method (they rely on the zk-ruby gem's async callbacks to multithread themselves).

@dcosson I imagine you're using the ec2 tag watcher or some such?

jolynch commented 9 years ago

Oops, didn't mean to close, sorry about that.