search

fabiolb / fabio

Consul Load-Balancing made simple
https://fabiolb.net
MIT License
7.25k stars 620 forks source link

Fabio immediately drop routes when consul agent unavailable for a while #861

Closed andrekzn closed 2 years ago

andrekzn commented 2 years ago

There is a problem, when consul agent on the host have been restarted, for in-place version update. At this time fabio detect consul unavailability, and delete routes. At this moment we see lots of 5xx errors, which is expected. All this happen about to 2-5 seconds. When agent is started, all routes has back again. There in no big deal about it, but would be great if we can set some timeout on the consul agent availability. Mb there is some settings, but I can't find it.

tristanmorgan commented 2 years ago

I'm taking a guess but does registry.timeout = 10s do anything for this?

andrekzn commented 2 years ago

Unfortunately increasing the value to registry.timeout = 20s did not solve the problem. Imo, default value 10s is enough. Consul starts much faster. Docs says that this option affect during startup, apparently this is true.

nathanejohnson commented 2 years ago

@andrekzn do you have the ability to test against master?

https://github.com/fabiolb/fabio/pull/811

Setting these new options might possibly alleviate the issue.

andrekzn commented 2 years ago

@nathanejohnson hi, sorry for delay. I build a test docker image from master branch, added options in the fabio startup config from the above PR. But the behavior remains the same, routes are still deleted, when consul agent unavailable for a short period. Maybe I'm doing something wrong...

andrekzn commented 2 years ago

In general, I think I found another reason of this behavior. Nevertheless, the consul sends a request to remove the service from the catalog when restarting. And from here the specific route is removed. Not all are deleted, but only those apps that are running on the same host as fabio. In our infra it is ok to run load-balancer and apps on the same nomad agent.

I will test it again and post the results. Thanks anyway.

andrekzn commented 2 years ago

I close this issue, because it is not directly related to fabio. In my case this is the default behavior of the consul agent.