`PG_EXPORTER_DISABLE_DEFAULT_METRICS` is not working properly

[atsu0127]

What did you do?

I ran postgres-exporter with environment variables set like below.

$ cat ./queries-test.yaml
pg_postmaster:
  query: "SELECT pg_postmaster_start_time as start_time_seconds from pg_postmaster_start_time()"
  master: true
  metrics:
    - start_time_seconds:
        usage: "GAUGE"
        description: "Time at which postmaster started"
$ export PG_EXPORTER_DISABLE_DEFAULT_METRICS="true"
$ export PG_EXPORTER_DISABLE_SETTINGS_METRICS="true"
$ export DATA_SOURCE_URI="<URI>"
$ export DATA_SOURCE_PASS="<PASS>"
$ export DATA_SOURCE_USER="postgres"
$ ./postgres_exporter --extend.query-path="./queries-test.yaml"

And I ran call.

curl -q localhost:9187/metrics | grep pg_

What did you expect to see? Only the following can be confirmed.

# TYPE pg_postmaster_start_time_seconds gauge
pg_postmaster_start_time_seconds{server="<server>"} 1.675141895e+09

I figured it would be as above because the README said the following.

PG_EXPORTER_DISABLE_DEFAULT_METRICS…Use only metrics supplied from queries.yaml

What did you see instead? Under which circumstances? Other metrics are shown.

# HELP pg_database_size_bytes Disk space used by the database
# TYPE pg_database_size_bytes gauge
pg_database_size_bytes{datname="<db1>",server="<server>"} 2.12219171375e+11
pg_database_size_bytes{datname="postgres",server="<server>"} 8.344111e+06
pg_database_size_bytes{datname="template0",server="<server>"} 8.311343e+06
pg_database_size_bytes{datname="template1",server="<server>"} 8.311343e+06
pg_database_size_bytes{datname="<db2>",server="<server>"} 8.311343e+06
# HELP pg_exporter_last_scrape_duration_seconds Duration of the last scrape of metrics from PostgreSQL.
# TYPE pg_exporter_last_scrape_duration_seconds gauge
pg_exporter_last_scrape_duration_seconds 0.001650698
# HELP pg_exporter_last_scrape_error Whether the last scrape of metrics from PostgreSQL resulted in an error (1 for error, 0 for success).
# TYPE pg_exporter_last_scrape_error gauge
pg_exporter_last_scrape_error 0
# HELP pg_exporter_scrapes_total Total number of times PostgreSQL was scraped for metrics.
# TYPE pg_exporter_scrapes_total counter
pg_exporter_scrapes_total 5
# HELP pg_exporter_user_queries_load_error Whether the user queries file was loaded and parsed successfully (1 for error, 0 for success).
# TYPE pg_exporter_user_queries_load_error gauge
pg_exporter_user_queries_load_error{filename="./queries-test.yaml",hashsum="800e12a78243d812f323a64223773fabcd9e3ae0c038f710d508e9c93322ec70"} 0
# HELP pg_postmaster_start_time_seconds Time at which postmaster started
# TYPE pg_postmaster_start_time_seconds gauge
pg_postmaster_start_time_seconds{server="<server>"} 1.675141895e+09
# HELP pg_scrape_collector_duration_seconds postgres_exporter: Duration of a collector scrape.
# TYPE pg_scrape_collector_duration_seconds gauge
pg_scrape_collector_duration_seconds{collector="bgwriter"} 0.001010159
pg_scrape_collector_duration_seconds{collector="database"} 0.005522488
# HELP pg_scrape_collector_success postgres_exporter: Whether a collector succeeded.
# TYPE pg_scrape_collector_success gauge
pg_scrape_collector_success{collector="bgwriter"} 1
pg_scrape_collector_success{collector="database"} 1
# HELP pg_stat_bgwriter_buffers_alloc_total Number of buffers allocated
# TYPE pg_stat_bgwriter_buffers_alloc_total counter
pg_stat_bgwriter_buffers_alloc_total{server="<server>"} 2.73716985e+08
# HELP pg_stat_bgwriter_buffers_backend_fsync_total Number of times a backend had to execute its own fsync call (normally the background writer handles those even when the backend does its own write)
# TYPE pg_stat_bgwriter_buffers_backend_fsync_total counter
pg_stat_bgwriter_buffers_backend_fsync_total{server="<server>"} 0
# HELP pg_stat_bgwriter_buffers_backend_total Number of buffers written directly by a backend
# TYPE pg_stat_bgwriter_buffers_backend_total counter
pg_stat_bgwriter_buffers_backend_total{server="<server>"} 2.06971e+06
# HELP pg_stat_bgwriter_buffers_checkpoint_total Number of buffers written during checkpoints
# TYPE pg_stat_bgwriter_buffers_checkpoint_total counter
pg_stat_bgwriter_buffers_checkpoint_total{server="<server>"} 1.69325119e+08
# HELP pg_stat_bgwriter_buffers_clean_total Number of buffers written by the background writer
# TYPE pg_stat_bgwriter_buffers_clean_total counter
pg_stat_bgwriter_buffers_clean_total{server="<server>"} 304054
# HELP pg_stat_bgwriter_checkpoint_sync_time_total Total amount of time that has been spent in the portion of checkpoint processing where files are synchronized to disk, in milliseconds
# TYPE pg_stat_bgwriter_checkpoint_sync_time_total counter
pg_stat_bgwriter_checkpoint_sync_time_total{server="<server>"} 114886
# HELP pg_stat_bgwriter_checkpoint_write_time_total Total amount of time that has been spent in the portion of checkpoint processing where files are written to disk, in milliseconds
# TYPE pg_stat_bgwriter_checkpoint_write_time_total counter
pg_stat_bgwriter_checkpoint_write_time_total{server="<server>"} 2.963633e+07
# HELP pg_stat_bgwriter_checkpoints_req_total Number of requested checkpoints that have been performed
# TYPE pg_stat_bgwriter_checkpoints_req_total counter
pg_stat_bgwriter_checkpoints_req_total{server="<server>"} 2273
# HELP pg_stat_bgwriter_checkpoints_timed_total Number of scheduled checkpoints that have been performed
# TYPE pg_stat_bgwriter_checkpoints_timed_total counter
pg_stat_bgwriter_checkpoints_timed_total{server="<server>"} 693
# HELP pg_stat_bgwriter_maxwritten_clean_total Number of times the background writer stopped a cleaning scan because it had written too many buffers
# TYPE pg_stat_bgwriter_maxwritten_clean_total counter
pg_stat_bgwriter_maxwritten_clean_total{server="<server>"} 42
# HELP pg_stat_bgwriter_stats_reset_total Time at which these statistics were last reset
# TYPE pg_stat_bgwriter_stats_reset_total counter
pg_stat_bgwriter_stats_reset_total{server="<server>"} 1.675141896e+09
# HELP pg_up Whether the last scrape of metrics from PostgreSQL was able to connect to the server (<server>

Environment

• System information:

Linux 3.10.0-1160.59.1.el7.x86_64 x86_64

• postgres_exporter version:

  postgres_exporter, version 0.11.1 (branch: HEAD, revision: b5fd2465646a09bb1c8dbd5c9c00b852da93927a)
  build user:       root@a2c981f603ff
  build date:       20220818-20:58:17
  go version:       go1.18.5
  platform:         linux/amd64

• postgres_exporter flags:

--extend.query-path="./queries-test.yaml"

• PostgreSQL version:

  13.5

• Logs:

  ts=2023-02-15T03:09:22.336Z caller=main.go:135 level=info msg="Listening on address" address=:9187
  ts=2023-02-15T03:09:22.336Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false
  ts=2023-02-15T03:10:03.874Z caller=server.go:74 level=info msg="Established new database connection" fingerprint=...
  ts=2023-02-15T03:10:04.023Z caller=postgres_exporter.go:645 level=info msg="Semantic version changed" server=... from=0.0.0 to=13.5.0

[hcastrofactored]

@atsu0127 Did you figure it out?

[sysadmind]

There are now collectors enabled/disabled individually via their flags (i.e. --collector.database or the option to turn it off --no-collector.database). Some of the collectors are enabled by default while others are disabled by default. The option PG_EXPORTER_DISABLE_DEFAULT_METRICS will probably be removed at some point in the future. The intention is for this exporter to provide good out of the box metrics about postgres itself. For custom queries around business metrics, we would advise using an generic sql exporter like https://github.com/burningalchemist/sql_exporter