pg_exporter: fail scrapping server [meta]: fail fetching server version: context deadline exceeded

[oukanghua]

There are something wrong with pg_exporter

OS: EL7 pigsty version: 2.5 pg_exporter version: 0.6

$ systemctl status pg_exporter -l
● pg_exporter.service - Prometheus exporter for PostgreSQL/Pgbouncer server metrics
   Loaded: loaded (/usr/lib/systemd/system/pg_exporter.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2023-11-27 22:42:02 CST; 21h ago
     Docs: https://github.com/Vonng/pg_exporter
 Main PID: 61695 (pg_exporter)
   Memory: 13.4M (limit: 200.0M)
   CGroup: /system.slice/pg_exporter.service
           └─61695 /usr/bin/pg_exporter --web.listen-address=:9630 --log.level=info --log.format=logfmt

Nov 28 19:51:56 pg-meta-1 pg_exporter[61695]: level=error timestamp=2023-11-28T11:51:56.707788694Z caller=utils.go:66 msg="fail scrapping server [meta]: fail fetching server version: context deadline exceeded"
Nov 28 19:53:46 pg-meta-1 pg_exporter[61695]: level=error timestamp=2023-11-28T11:53:46.707353806Z caller=utils.go:66 msg="fail scrapping server [postgres]: fail fetching server version: context deadline exceeded"
Nov 28 19:57:06 pg-meta-1 pg_exporter[61695]: level=warn timestamp=2023-11-28T11:57:06.708422855Z caller=utils.go:61 msg="query [pg_clustering] error skipped: query [pg_clustering] failed: pq: canceling statement due to user request"
Nov 28 20:02:26 pg-meta-1 pg_exporter[61695]: level=warn timestamp=2023-11-28T12:02:26.712001561Z caller=utils.go:61 msg="query [pg_subrel] error skipped: query [pg_subrel] failed: pq: canceling statement due to user request"
Nov 28 20:02:36 pg-meta-1 pg_exporter[61695]: level=warn timestamp=2023-11-28T12:02:36.509779189Z caller=utils.go:61 msg="query [pg_subrel] error skipped: query [pg_subrel] failed: pq: canceling statement due to user request"
Nov 28 20:04:26 pg-meta-1 pg_exporter[61695]: level=error timestamp=2023-11-28T12:04:26.808761832Z caller=utils.go:66 msg="query [pg] error: query [pg] failed: pq: canceling statement due to user request"
Nov 28 20:04:26 pg-meta-1 pg_exporter[61695]: level=error timestamp=2023-11-28T12:04:26.808823831Z caller=utils.go:66 msg="fail scrapping server [postgres]: query [pg] failed: pq: canceling statement due to user request"
Nov 28 20:05:36 pg-meta-1 pg_exporter[61695]: level=error timestamp=2023-11-28T12:05:36.911010023Z caller=utils.go:66 msg="fail scrapping server [meta]: fail fetching server version: context deadline exceeded"
Nov 28 20:08:56 pg-meta-1 pg_exporter[61695]: level=error timestamp=2023-11-28T12:08:56.808512993Z caller=utils.go:66 msg="fail scrapping server [meta]: fail fetching server version: context deadline exceeded"
Nov 28 20:12:06 pg-meta-1 pg_exporter[61695]: level=warn timestamp=2023-11-28T12:12:06.810556502Z caller=utils.go:61 msg="query [pg_func] error skipped: query [pg_func] failed: pq: canceling statement due to user request"

[Vonng]

It seems a timeout (100ms by default) is happend between your pg_exporter and postgres instance. Are you monitoring a remote Postgres instance? if it is a local instance, maybe you can check local firewalls, selinux, and if any other security restrictions may prevent the exporter from connecting to database.

[oukanghua]

I check the vim /etc/default/pg_exporter profile and thought the password contained the character '@', that caused error.So I change the password and synchronize all server times.And then the problem was solved.

[Vonng]

Maybe I can add this notice to the documentation, and thanks for the feedback.