search

ribbybibby / s3_exporter

Exports Prometheus metrics about S3 buckets and objects
Apache License 2.0
105 stars 42 forks source link

AWS API caching #37

Open ivanovv opened 2 years ago

ivanovv commented 2 years ago

Given that it is adviced to keep scrapping interval equal to or less than 2 minutes (more details here https://stackoverflow.com/questions/70124042/prometheus-scrape-interval-1d/70129957) is there a way to add caching to the results of API call?

I don't want to send API requests every two monutes if I know that my backup script runs every 4 hours.

ribbybibby commented 2 years ago

I'm not going to implement caching in the exporter.

You could maybe experiment with putting a caching proxy (something like nginx) in front of the exporter to cache the response?

ivanovv commented 2 years ago

Ah, that's really unfortunate.

I really like how it is implemented fort example here https://github.com/free/sql_exporter

Given that AWS API is paid it only makes sense to reduce charges as much as possible. Though I'm not sure if there is a really high number of requests and maybe I should run it first to see the monetary impact.