flightstats / hub

fault tolerant, highly available service for data storage and distribution
http://www.flightstats.com
MIT License
103 stars 35 forks source link

Prioritize known-healthy nodes over others when starting a webhook #1241

Closed lkemmerer closed 4 years ago

lkemmerer commented 4 years ago

Scenario: A new webhook is picked up and should be started on the node with the fewest webhooks on it. Existing behavior: When figuring out which node has the fewest running webhooks, nodes that returned non-200 status codes were reporting as running 0 webhooks. This caused them to be bumped to the top of the list of nodes to try. New behavior: Nodes that return non-200 status codes when webhook counts are requested will behave as if they have max integer webhooks, which should cause them to drop to the bottom of the list for potential nodes to try.

Backstory: While I was fixing some other webhook code, I started writing tests against the internal webhook client. This caused me to realize that some of the mocks that were being used were overwriting themselves, which caused tests to pass when they shouldn't. I localized the mocks and fixed the tests and the code.