search

prometheus-community / postgres_exporter

A PostgreSQL metric exporter for Prometheus
Apache License 2.0
2.82k stars 743 forks source link

http: multiple response.WriteHeader calls #427

Open anandgia opened 4 years ago

anandgia commented 4 years ago

We are running version 0.8.0 of postgres_exporter and notice these errors sporadically:

time="2020-08-25T20:32:43Z" level=info msg="Established new database connection to \"configdb.local.:5432\"." source="postgres_exporter.go:878" time="2020-08-25T20:32:43Z" level=info msg="Semantic Version Changed on \"configdb.local.:5432\": 0.0.0 -> 10.11.0" source="postgres_exporter.go:1405" time="2020-08-25T20:32:47Z" level=info msg="Starting Server: :9187" source="postgres_exporter.go:1672" 2020/08/25 22:04:35 http: multiple response.WriteHeader calls 2020/08/25 23:01:35 http: multiple response.WriteHeader calls 2020/08/25 23:02:34 http: multiple response.WriteHeader calls 2020/08/25 23:03:34 http: multiple response.WriteHeader calls 2020/08/25 23:04:34 http: multiple response.WriteHeader calls 2020/08/25 23:07:34 http: multiple response.WriteHeader calls 2020/08/25 23:09:34 http: multiple response.WriteHeader calls 2020/08/26 00:02:34 http: multiple response.WriteHeader calls 2020/08/26 00:03:36 http: multiple response.WriteHeader calls 2020/08/26 00:04:34 http: multiple response.WriteHeader calls 2020/08/26 00:05:35 http: multiple response.WriteHeader calls 2020/08/26 00:06:36 http: multiple response.WriteHeader calls 2020/08/26 00:07:44 http: multiple response.WriteHeader calls 2020/08/26 00:11:08 http: multiple response.WriteHeader calls 2020/08/26 00:11:14 http: multiple response.WriteHeader calls 2020/08/26 00:11:35 http: multiple response.WriteHeader calls 2020/08/26 00:11:44 http: multiple response.WriteHeader calls 2020/08/26 01:01:34 http: multiple response.WriteHeader calls 2020/08/26 01:08:34 http: multiple response.WriteHeader calls 2020/08/26 01:09:34 http: multiple response.WriteHeader calls 2020/08/26 03:02:34 http: multiple response.WriteHeader calls

It appears the scrape timeout expires and multiple calls at same time causes this race condition. This issue seems to be have been addressed in later version of prometheus client https://github.com/prometheus/graphite_exporter/issues/123 but we notice postgres_exporter is using an older client library.

AlexDCraig commented 4 years ago

We see the same issue all the time. I don't think the exporter is maintained much. We always increase the scrape interval and timeout to address it (not a fantastic solution).

suveshmathur commented 4 years ago

After increasing the timeout for prom, the message didn't appear in the log.