search

ovh / beamium

Prometheus to Warp10 metrics forwarder
Other
84 stars 21 forks source link

Too many logs on network error #78

Closed babolivier closed 6 years ago

babolivier commented 6 years ago

One of our hosts experienced a failure during a reboot recently, causing it to lose its public IP, resulting in an inability to communicate over the Internet. This caused beamium to constantly check for network availability, logging at each try. Here's an excerpt:

Aug 16 18:50:52.073 ERRO HTTP error: Network is unreachable (os error 101), sink: ovh-metrics-plan-m
Aug 16 18:50:52.075 ERRO HTTP error: Network is unreachable (os error 101), sink: ovh-metrics-plan-m
Aug 16 18:50:52.077 ERRO HTTP error: Network is unreachable (os error 101), sink: ovh-metrics-plan-m
Aug 16 18:50:52.079 ERRO HTTP error: Network is unreachable (os error 101), sink: ovh-metrics-plan-m

As shown above, beamium tries and logs every 2ms, which caused it to fill up the entirety of the host's available disk space. There should be a delay between two tries, ideally (imho) matching the parameters:scan-period config parameter (since that's the one determining the minimum delay between two pushes to Warp10). On that host, beamium has been installed using the Debian/Ubuntu package, and is at version 1.8.0-stretch.

babolivier commented 6 years ago

I just noticed that this issue was likely a duplicate of #36, which has been fixed with #83, so I'm closing it.