search

ncabatoff / script-exporter

Prometheus exporter to invoke scripts and parse their output as metrics.
MIT License
37 stars 20 forks source link

Error: Get http://localhost:9661/metrics/main: EOF #5

Open kevinekw opened 6 years ago

kevinekw commented 6 years ago

Hi ncabatoff,

Thank you for providing such an awesome exporter. I'm fairly new to Prometheus, so please bear with me :)

I have been using your script-exporter against a python executable file that runs some oracle sql query and output in the Prometheus text exposision format.

Out of the 3 queries, 2 of them are working fine (script duration is less than 5sec) and prometheus did not display any errors. The third one sometimes works fine, and sometimes gives me and EOF error as shown: eof_error eof_error2

This script duration runs between 5-60sec. This depends on how busy the database is, the script duration will vary.

To be honest, I dont think this is a script duration issue. Looking at the error, would you be able to tell me what could be the problem and how I could fix it?

Thank you in advance :)

Regards, Kevin

ncabatoff commented 6 years ago

Not sure, but I fixed a serious bug (race condition) a few days ago, so you might want to try updating. Otherwise, are you seeing anything in the exporter's stdout/stderr? Are any of the script_exporter internal error metrics nonzero?

zstoth commented 5 years ago

I see the exact same issue randomly appearing for one of my targets. The script logs that it was invoked, but somehow Prometheus does not receive the response. Can't see anything in Prometheus' or the script exporter's logs. Using the 0.1.3 release.

nikosmeds commented 5 years ago

@zstoth I'm having similar issue. What happens when you curl the exporter directly?

$ time curl localhost:9661/metrics/servers.py
curl: (52) Empty reply from server

real    0m21.358s
user    0m0.008s
sys 0m0.000s

Seeing the same EOF error in Prometheus targets, and curl returns with the empty reply message shown above. Couple things of note.

  1. Running the script directly with ./servers.py is successful and takes ~30 seconds.
  2. Running the script via curl or as Prometheus returns the above mentioned errors, however the script is successfully running in both cases (confirmed with log files). Have scrape interval set to 60 seconds and timeout 50 seconds for the specific Prometheus job.
nikosmeds commented 5 years ago

Possibly related to https://github.com/ncabatoff/script-exporter/pull/6 ?

EDIT: Yep. I recompiled the binary with change in https://github.com/ncabatoff/script-exporter/pull/6 and it has resolved the issue.

Go-Pomegranate commented 4 years ago

For me setting scrape_timeout to 5 min (before 1 min) solved a similar issue in another project.

jialanli commented 3 years ago

I also had a similar error, but it happened in the process of prometheus notifying alertmanager, please move: https://github.com/prometheus/prometheus/issues/9176 Do you have a solution to the error, thank you very much!