influxdata / telegraf

Agent for collecting, processing, aggregating, and writing metrics, logs, and other arbitrary data.
https://influxdata.com/telegraf
MIT License
14.52k stars 5.56k forks source link

Allow poll interval for --watch-config to be configurable #15793

Open simonwhybrow opened 3 weeks ago

simonwhybrow commented 3 weeks ago

Use Case

The current poll interval is fixed at 250ms. It would be useful for this polling interval to be configurable to any value the user requires. If no value is supplied, then it will default to the 250ms.

We have come across an "issue" where we hosted our telegraf config files on an Azure storage account. We configured the telegraf agents installed in the estate to reference this storage account and turned on the "--watch-config poll" option so that all agents would pick up any updates to the config files.

The issue was that each time a server polls for an update to the config files it actually performs a write action. (I believe this is just the way the golang polling process checks for differences by writing a checksum of the file?). Each write action on an Azure storage account incurs a cost. When we had over 500 agents polling multiple files every 250ms the writes quickly escalated ( > 50k transactions / minute) and so did the associated costs.

We've moved away from using an Azure storage account to avoid these costs but would like to get back to using at some point. To do so it would be great if we can adjust the polling interval via the --watch-config option as we'd only need to poll once a day or a few times a day. We definitely don't need to poll every 250ms.

e.g.

telegraf.exe --watch-config poll --watch-interval 6h

Expected behavior

If an interval parameter can be specified on the command line (or when installing the agent as a service in windows) to change how often the files are wachted/polled for changed.

Actual behavior

N/A

Additional info

No response

DStrand1 commented 3 weeks ago

Next steps: deprecate --config-url-watch-interval option in favor of a general --watch-interval option to be used by url config watching as well as non-url config watching