search

ricoberger / script_exporter

Prometheus exporter to execute scripts and collect metrics from the output or the exit status.
MIT License
354 stars 82 forks source link

Caching metrics #84

Closed torbenaa closed 1 year ago

torbenaa commented 1 year ago

Hi Rico.

Great work - I love your little exporter here.

Is it possible to add a caching feature to the exporter ?

I have some scripts that I would prefer to run on a hourly basis ( for performance reasons ) - but to avoid prometheus staleness, I cannot set the scrape_interval higher than 5m.

Can the exporter perhaps cache the script output and return the last values for a given set of parameters ?

Or is there a better way to accomplish this ?

Best regards Torben

ricoberger commented 1 year ago

Hi @torbenaa, I think as long as we do not have to care about restarts of the exporter or other cases where a persistence is required, we could add an optional cache / cacheTime parameter to the script configuration to achieve this.

If it is not enough to cache the results in memory, I wouldn't really see this feature in the scope of the exporter.

Please let me know what you think about this.

torbenaa commented 1 year ago

Caching these values in memory would be sufficient, I think. I totally agree that we don't need persistence, since that would probably add unwanted complexity.

Would love to see this functionality at some point. Keep up the good work :)

ricoberger commented 1 year ago

HI @torbenaa this feature should be available in the v2.14.0 release via the newly added cacheDuration option.

If you have any problems with it please let me know.

torbenaa commented 1 year ago

Hi Rico !

Wow - that was quick :)

I have tested the caching feature with various scripts and values. Everything looks good - except one thing :

I seems like the caching feature ignores the script parameters. Once a script has been executed with a given set of parameters, the output is cached and returned for any set of parameter values. I would have expected the parameters to be part of the caching selectors ( just like the script name ).

Best regards Torben

ricoberger commented 1 year ago

Hi @torbenaa, thanks for the fast feedback. The parameters should be included in the cache key in v2.14.1

torbenaa commented 1 year ago

And thank YOU for the prompt response. It works flawlessly !

Very very nice work - thanks again :)

torbenaa commented 1 year ago

What happens to the cached result if the script fails ? I noticed a strange behaviour at around 06:00 this morning, probably related to a transient error on the network. But ever since that, I get script_exit_code = 0. Restarting the docker container fixed it...

I dont know if this is related to caching - let me keep an eye on it....

ricoberger commented 1 year ago

Hi @torbenaa, at the moment the result is always cached, independent from the exit code. The only exception is when ignoreOutputOnFail is set and the script fails, then the result isn't cached.

I'm also ok with adding an additional check to only cache the result when the exit code is 0.

Please let me know, if this should be changed.