search

Graylog2 / collector-sidecar

Manage log collectors through Graylog
https://www.graylog.org/
Other
268 stars 56 forks source link

Re-resolve Graylog server hostname on interval #479

Closed damianharouff closed 10 months ago

damianharouff commented 11 months ago

Problem description

Sidecar does not re-resolve the Graylog server hostname after the connection is made, so if the endpoint IP changes at some point in the future, the only action is to restart Sidecar to re-resolve the hostname.

This was noted with several Graylog Cloud customers during V1 -> NG migrations, where we continued to note Sidecars still hitting the old endpoint IP several days later, and had to reach out to the affected customers to have them just restart Sidecar.

It's important to note that both the old and new endpoint IPs were live, but Sidecar never started contacting Graylog on the new IP, because it did not seem to ever re-resolve Graylog server hostname.

Environment

mpfz0r commented 10 months ago

It's important to note that both the old and new endpoint IPs were live

Yes, that's because we are using http keep alive. This is a known and yet unresolved problem.

https://github.com/golang/go/issues/23427

https://abhishekvrshny.medium.com/dns-aware-persistent-connections-abda9921db34

I think a simple fix would be to just re-create a new http connection every X periodical loops.