search

ricoberger / script_exporter

Prometheus exporter to execute scripts and collect metrics from the output or the exit status.
MIT License
354 stars 82 forks source link

gaps between the metrics #104

Closed bobykus31 closed 7 months ago

bobykus31 commented 11 months ago

Since we started using script_exporter, we found a weird issue that makes us uncomfortable. We use script_exporter to execute some long-running scripts like smartctl metrics. Even though we have a reasonable cache duration set up, we still have gaps in between the metrics, that correspond to the time of script execution or close. The script-exporter.yml looks like 

scripts:
...
- name: node_smartmon
  command: "/usr/lib/prometheus/custom_metric_node_smartmon.sh"
  cacheDuration: 1200s
  timeout:
    max_timeout: 600
    enforced: false

The result is

image

Is there any good way to get rid of this gaps?

ricoberger commented 11 months ago

Hi @bobykus31, the problem might be the max_timeout of 600 seconds, which is larger as the default staleness of 5 minutes. So when the script is executed and the scrape takes more the 5 minutes, it would result in a small gap until the execution of the scrape is finished. Can you check if the gaps are corresponding with the script_duration_seconds metric when it is larger than 5 minutes, please?

ricoberger commented 7 months ago

Hi @bobykus31, since there is no activity for some time, I will close the issue for now. If this is still a problem please let me know and we can reopen the issue.