fabiolb / fabio

Consul Load-Balancing made simple
https://fabiolb.net
MIT License
7.25k stars 621 forks source link

[nasty bug] Using a static route rule to change the weight of a missing dynamic route causes Fabio to incorrectly startup #838

Open far-blue opened 3 years ago

far-blue commented 3 years ago

If a static route exists that changes the weight of a dynamic route but that dynamic route doesn't exist in the initial data fetched from consul when fabio first starts up, fabio ends up in a half-configured state and won't route anything.

Here's an example:

Add the following as a static rule - it doesn't matter what it's called or what it targets as long as the target dynamic rule doesn't exist. In case you wondered, I use it to make sure all web ui traffic for vault goes to the leader:

route weight vault vault.example.com/* weight 1 tags "active"

If added once fabio is already running, everything should be fine and fabio will keep working.

Restart fabio

When fabio restarts, I found:

If at this point you then remove the above static rule, everything starts working again. You can even put the rule back and everything is fine - until you restart fabio again.

This caused me serious problems as it took down our entire website and a number of internal services and took ages to track down because in our case the dynamic route in question had been changed weeks ago - but we'd just not restarted fabio!