search

fabiolb / fabio

Consul Load-Balancing made simple
https://fabiolb.net
MIT License
7.25k stars 619 forks source link

Fabio 1.5.13 - no more "[INFO] Config updates" message in the logs #751

Closed samm-git closed 4 years ago

samm-git commented 4 years ago

Hi,

After updating from Fabio 1.5.11 to the Fabio 1.5.13 we found that [INFO] Config updates is completely gone. We are using this message to validate that Fabio configuration is correctly updated, so it is critical for us to have it. Also i did not found anything in the logs telling that it is removed.

I am using INFO error log, also tried with DEBUG one, with the same result. Fabio itself is updating routes, so looks like that issue affects only messaging.

This is how update was looking in 1.5.11:

 2020/02/04 14:58:03 [INFO] Config updates
 - fabio-heartbeat-nomad-5 fabio-heartbeat-nomad-5/ http://127.0.0.1:7/ tags "public,public:fabio-heartbeat-nomad-5/,private"
 - route add fabio-heartbeat-nomad-4 fabio-heartbeat-nomad-4/ http://127.0.0.1:7/ tags "public,public:fabio-heartbeat-nomad-4/,private"
 - route add

In 1.5.13 it is just empty. fabio-heartbeat-nomad-* is a special service which goes up and down every 5 minutes to validate that all routes are correctly distributed on all fabio nodes. However, it looks like that all route messages are missing entirely.

Please let me know if any other information is needed.

aaronhurt commented 4 years ago

This will be fixed in the next release. See #737 and #749 ... this was an unintended side affect of using a rolling buffer to build the route table.

aaronhurt commented 4 years ago

Closing this issue, feel free to re-open if this is not a duplicate of the above issues.

aaronhurt commented 4 years ago

Side note, would be curious about your nomad test. We're using Fabio with Nomad to deploy our applications as well.

samm-git commented 4 years ago

@leprechau thank you for the quick feedback! Test is very simple - in very rare case we have config updates stalled. It could be due to few issues:

  1. Routing table is bad due to application bugs (should never be the case, but better to check)
  2. Fabio is not reading updates from consul properly (i reported that already, no root cause found yet, happens not often).
  3. Consul is not working

So using this heartbeat services we are checking if there are updates of the config in the last n minutes and if not - generating alert. We can also try to self-heal it by restarting fabio, but not doing it now as still hope to catch the root cause of the issue.

P.S. will just wait for the new release, thanks again.

aaronhurt commented 4 years ago

@samm-git Thank you. I'm hoping to roll a 1.5.14-beta at least this week. With all the changes lately I think a beta release may be the best to get some real-world testing done before moving forward with an actual release.

codyja commented 4 years ago

@leprechau thank you for the quick feedback! Test is very simple - in very rare case we have config updates stalled. It could be due to few issues:

  1. Routing table is bad due to application bugs (should never be the case, but better to check)
  2. Fabio is not reading updates from consul properly (i reported that already, no root cause found yet, happens not often).
  3. Consul is not working

So using this heartbeat services we are checking if there are updates of the config in the last n minutes and if not - generating alert. We can also try to self-heal it by restarting fabio, but not doing it now as still hope to catch the root cause of the issue.

P.S. will just wait for the new release, thanks again.

Hi @samm-git , I was curious if you could share that hearbeat service you mentioned?