collected metric was collected before with the same name and label values

[eriktelepovsky]

What did you do?

I started postgres-exporter using following docker-compose file:

  postgres-exporter:
    container_name: postgres-exporter
    image: prometheuscommunity/postgres-exporter
    environment:
      - DATA_SOURCE_NAME=postgresql://user:password@host:port/my_database  # I changed this for issue purpose 
    command:
      # - "--disable-default-metrics"  # doesn't matter if I run it with or without this argument 
      - "--collector.stat_statements"  # error: collected before with the same name and label values
    ports:
      - "9187:9187"

What did you expect to see?

Opening http://127.0.0.1:9187/metrics I expected to see all metrics including statements.

What did you see instead? Under which circumstances?

I see 120 errors instead, but only when using --collector.stat_statements:

An error has occurred while serving metrics:

120 error(s) occurred:
* collected metric "pg_stat_statements_calls_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:145094}} was collected before with the same name and label values
* collected metric "pg_stat_statements_seconds_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:1382.6511825559967}} was collected before with the same name and label values
* collected metric "pg_stat_statements_rows_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:435282}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_read_seconds_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_write_seconds_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_calls_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:145185}} was collected before with the same name and label values
* collected metric "pg_stat_statements_seconds_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:55.34379315600012}} was collected before with the same name and label values
* collected metric "pg_stat_statements_rows_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:871110}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_read_seconds_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_write_seconds_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_calls_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"_updb"}  counter:{value:2.173387e+06}} was collected before with the same name and label values
* collected metric "pg_stat_statements_seconds_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"_updb"}  counter:{value:47.36949219000851}} was collected before with the same name and label values
* collected metric "pg_stat_statements_rows_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"_updb"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_read_seconds_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"_updb"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_write_seconds_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"_updb"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_calls_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:145185}} was collected before with the same name and label values
* collected metric "pg_stat_statements_seconds_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:33.97874588199985}} was collected before with the same name and label values
* collected metric "pg_stat_statements_rows_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:290354}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_read_seconds_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_write_seconds_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_calls_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:431657}} was collected before with the same name and label values
* collected metric "pg_stat_statements_seconds_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:32.00555152200009}} was collected before with the same name and label values
* collected metric "pg_stat_statements_rows_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:1.294971e+06}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_read_seconds_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_write_seconds_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_calls_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:145185}} was collected before with the same name and label values
* collected metric "pg_stat_statements_seconds_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:22.935943721999884}} was collected before with the same name and label values
* collected metric "pg_stat_statements_rows_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:145185}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_read_seconds_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_write_seconds_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_calls_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"_updb"}  counter:{value:2.173386e+06}} was collected before with the same name and label values
* collected metric "pg_stat_statements_seconds_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"_updb"}  counter:{value:19.923873926997413}} was collected before with the same name and label values
* collected metric "pg_stat_statements_rows_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"_updb"}  counter:{value:2.173386e+06}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_read_seconds_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"_updb"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_write_seconds_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"_updb"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_calls_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:145185}} was collected before with the same name and label values
* collected metric "pg_stat_statements_seconds_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:16.86972648199979}} was collected before with the same name and label values
* collected metric "pg_stat_statements_rows_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:145185}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_read_seconds_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_write_seconds_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_calls_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:430448}} was collected before with the same name and label values
* collected metric "pg_stat_statements_seconds_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:11.949401952000116}} was collected before with the same name and label values
* collected metric "pg_stat_statements_rows_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:430448}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_read_seconds_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_write_seconds_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_calls_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"_updb"}  counter:{value:2.173387e+06}} was collected before with the same name and label values
* collected metric "pg_stat_statements_seconds_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"_updb"}  counter:{value:9.745451962989884}} was collected before with the same name and label values
* collected metric "pg_stat_statements_rows_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"_updb"}  counter:{value:2.173387e+06}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_read_seconds_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"_updb"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_write_seconds_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"_updb"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_calls_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:145185}} was collected before with the same name and label values
* collected metric "pg_stat_statements_seconds_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:8.085575071000099}} was collected before with the same name and label values
* collected metric "pg_stat_statements_rows_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:1.016295e+06}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_read_seconds_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_write_seconds_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_calls_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:430450}} was collected before with the same name and label values
* collected metric "pg_stat_statements_seconds_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:7.9014078029997465}} was collected before with the same name and label values
* collected metric "pg_stat_statements_rows_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:1.297141e+06}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_read_seconds_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_write_seconds_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_calls_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:290370}} was collected before with the same name and label values
* collected metric "pg_stat_statements_seconds_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:5.888327989999761}} was collected before with the same name and label values
* collected metric "pg_stat_statements_rows_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_read_seconds_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_write_seconds_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_calls_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"_updb"}  counter:{value:51}} was collected before with the same name and label values
* collected metric "pg_stat_statements_seconds_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"_updb"}  counter:{value:5.648648003}} was collected before with the same name and label values
* collected metric "pg_stat_statements_rows_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"_updb"}  counter:{value:51}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_read_seconds_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"_updb"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_write_seconds_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"_updb"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_calls_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:585396}} was collected before with the same name and label values
* collected metric "pg_stat_statements_seconds_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:5.033520371000066}} was collected before with the same name and label values
* collected metric "pg_stat_statements_rows_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:585396}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_read_seconds_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_write_seconds_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_calls_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:290370}} was collected before with the same name and label values
* collected metric "pg_stat_statements_seconds_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:4.884215216999991}} was collected before with the same name and label values
* collected metric "pg_stat_statements_rows_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_read_seconds_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_write_seconds_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_calls_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:142092}} was collected before with the same name and label values
* collected metric "pg_stat_statements_seconds_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:4.015534736999865}} was collected before with the same name and label values
* collected metric "pg_stat_statements_rows_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_read_seconds_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_write_seconds_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_calls_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:145185}} was collected before with the same name and label values
* collected metric "pg_stat_statements_seconds_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:3.905134878999939}} was collected before with the same name and label values
* collected metric "pg_stat_statements_rows_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:725925}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_read_seconds_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_write_seconds_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_calls_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:72344}} was collected before with the same name and label values
* collected metric "pg_stat_statements_seconds_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:3.298348626000012}} was collected before with the same name and label values
* collected metric "pg_stat_statements_rows_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:145663}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_read_seconds_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_write_seconds_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_calls_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"_updb"}  counter:{value:51}} was collected before with the same name and label values
* collected metric "pg_stat_statements_seconds_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"_updb"}  counter:{value:3.278106381000001}} was collected before with the same name and label values
* collected metric "pg_stat_statements_rows_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"_updb"}  counter:{value:51}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_read_seconds_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"_updb"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_write_seconds_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"_updb"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_calls_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:145185}} was collected before with the same name and label values
* collected metric "pg_stat_statements_seconds_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:3.211697052999987}} was collected before with the same name and label values
* collected metric "pg_stat_statements_rows_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:145185}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_read_seconds_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_write_seconds_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_calls_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:52}} was collected before with the same name and label values
* collected metric "pg_stat_statements_seconds_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:2.730091644}} was collected before with the same name and label values
* collected metric "pg_stat_statements_rows_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:52}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_read_seconds_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_write_seconds_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_calls_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"_updb"}  counter:{value:51}} was collected before with the same name and label values
* collected metric "pg_stat_statements_seconds_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"_updb"}  counter:{value:2.6115781110000005}} was collected before with the same name and label values
* collected metric "pg_stat_statements_rows_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"_updb"}  counter:{value:51}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_read_seconds_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"_updb"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_write_seconds_total" { label:{name:"datname"  value:"_updb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"_updb"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_calls_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:145185}} was collected before with the same name and label values
* collected metric "pg_stat_statements_seconds_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:1.297608032999997}} was collected before with the same name and label values
* collected metric "pg_stat_statements_rows_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:145185}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_read_seconds_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_write_seconds_total" { label:{name:"datname"  value:"defaultdb"}  label:{name:"queryid"  value:"unknown"}  label:{name:"user"  value:"postgres"}  counter:{value:0}} was collected before with the same name and label values

Environment

• System information:

I tried on 2 systems:

1. my MacBooks: Darwin 22.5.0 arm64
2. my Linux server: Linux 5.10.0-18-amd64 x86_64

• postgres_exporter version:

postgres_exporter, version 0.15.0 (branch: HEAD, revision: 68c176b8833b7580bf847cecf60f8e0ad5923f9a) build user: root@7c5a76bc737a build date: 20231027-14:38:04 go version: go1.21.3 platform: linux/arm64 tags: unknown

• postgres_exporter flags:

collector.stat_statements

• PostgreSQL version:

14.10.0

• Logs:

  ts=2024-01-22T15:07:02.463Z caller=main.go:86 level=warn msg="Error loading config" err="Error opening config file \"postgres_exporter.yml\": open postgres_exporter.yml: no such file or directory"
  ts=2024-01-22T15:07:02.464Z caller=proc.go:267 msg="Excluded databases" databases="[\"template0 template1 _updb defaultdb\"]"
  ts=2024-01-22T15:07:02.464Z caller=main.go:103 level=warn msg="Scraping additional databases via auto discovery is DEPRECATED"
  ts=2024-01-22T15:07:02.472Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9187
  ts=2024-01-22T15:07:02.472Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9187
  ts=2024-01-22T15:07:02.946Z caller=server.go:74 level=info msg="Established new database connection" fingerprint=host:port
  ts=2024-01-22T15:07:03.243Z caller=postgres_exporter.go:613 level=info msg="Semantic version changed" server=host:port from=0.0.0 to=14.10.0

(I changed host and port for issue purpose)

[hajali-amine]

I am having almost the same issue.

An error has occurred while serving metrics:

5 error(s) occurred:
* collected metric "pg_stat_statements_calls_total" { label:{name:"datname"  value:"mydb"}  label:{name:"queryid"  value:"996906246641985132"}  label:{name:"user"  value:"mydbuser"}  counter:{value:1}} was collected before with the same name and label values
* collected metric "pg_stat_statements_seconds_total" { label:{name:"datname"  value:"mydb"}  label:{name:"queryid"  value:"996906246641985132"}  label:{name:"user"  value:"mydbuser"}  counter:{value:0.00079746}} was collected before with the same name and label values
* collected metric "pg_stat_statements_rows_total" { label:{name:"datname"  value:"mydb"}  label:{name:"queryid"  value:"996906246641985132"}  label:{name:"user"  value:"mydbuser"}  counter:{value:74}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_read_seconds_total" { label:{name:"datname"  value:"mydb"}  label:{name:"queryid"  value:"996906246641985132"}  label:{name:"user"  value:"mydbuser"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_write_seconds_total" { label:{name:"datname"  value:"mydb"}  label:{name:"queryid"  value:"996906246641985132"}  label:{name:"user"  value:"mydbuser"}  counter:{value:0}} was collected before with the same name and label values

Mine has a value in the label queryid.

I am running the exporter with Docker Compose;

version: '3'
services:
  postgres-exporter:
    image: quay.io/prometheuscommunity/postgres-exporter
    ports:
      - "9187:9187"
    environment:
      DATA_SOURCE_NAME: "postgresql://postgres_exporter:mypass@postgres:5432/mydb?sslmode=disable"
    command: --web.listen-address=:9187 --web.telemetry-path=/metrics --collector.database_wraparound --collector.long_running_transactions --collector.postmaster --collector.process_idle --collector.stat_activity_autovacuum --collector.stat_statements --collector.stat_wal_receiver --collector.statio_user_indexes

[acharbha]

same issue here -

curl localhost:26058/metrics

* collected metric "pg_stat_statements_calls_total" { label:{name:"datname"  value:"mydb"}  label:{name:"queryid"  value:"-2355316157827874919"}  label:{name:"user"  value:"mydb_so"}  counter:{value:2}} was collected before with the same name and label values
* collected metric "pg_stat_statements_seconds_total" { label:{name:"datname"  value:"mydb"}  label:{name:"queryid"  value:"-2355316157827874919"}  label:{name:"user"  value:"mydb_so"}  counter:{value:5.981075956000001}} was collected before with the same name and label values
* collected metric "pg_stat_statements_rows_total" { label:{name:"datname"  value:"mydb"}  label:{name:"queryid"  value:"-2355316157827874919"}  label:{name:"user"  value:"mydb_so"}  counter:{value:8}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_read_seconds_total" { label:{name:"datname"  value:"mydb"}  label:{name:"queryid"  value:"-2355316157827874919"}  label:{name:"user"  value:"mydb_so"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_write_seconds_total" { label:{name:"datname"  value:"mydb"}  label:{name:"queryid"  value:"-2355316157827874919"}  label:{name:"user"  value:"mydb_so"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_calls_total" { label:{name:"datname"  value:"mydb"}  label:{name:"queryid"  value:"-3686023444609730445"}  label:{name:"user"  value:"mydb_so"}  counter:{value:2}} was collected before with the same name and label values
* collected metric "pg_stat_statements_seconds_total" { label:{name:"datname"  value:"mydb"}  label:{name:"queryid"  value:"-3686023444609730445"}  label:{name:"user"  value:"mydb_so"}  counter:{value:5.831786688}} was collected before with the same name and label values
* collected metric "pg_stat_statements_rows_total" { label:{name:"datname"  value:"mydb"}  label:{name:"queryid"  value:"-3686023444609730445"}  label:{name:"user"  value:"mydb_so"}  counter:{value:8}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_read_seconds_total" { label:{name:"datname"  value:"mydb"}  label:{name:"queryid"  value:"-3686023444609730445"}  label:{name:"user"  value:"mydb_so"}  counter:{value:0}} was collected before with the same name and label values
* collected metric "pg_stat_statements_block_write_seconds_total" { label:{name:"datname"  value:"mydb"}  label:{name:"queryid"  value:"-3686023444609730445"}  label:{name:"user"  value:"mydb_so"}  counter:{value:0}} was collected before with the same name and label values

[vigneshkumar2016]

Let me explain the potential root cause for this problem.

• pg_stat_statements: doesnt hold the unique key constraint on the queryid column. however the pg_stat_statements is designed in such a way that the query and the underlying subquery on a nested query will have the same query id (or) transactions that are nested on same block will have the same query id. This is basically a computed hash identifier generated by postgres pg_stat_statements that are fetched from the query sent to planner and executor.

• postgres exporter: So now the exporter is collecting the metrics. say when you simulate running a nested queries which has joins, cross joins, lateral joins and so on.

• Labels: Since the metrics are scrapped with labels ( user, databasename, queryid ). It is expected to have this error.

• How, ever there is a fix which can be implemented. i have submitted a PR which holds the change CLICK HERE FOR PR.

Hope this explains the situation dealing with the problem.

Note: if you are using custom prometheus code with customized labels.. then you need to change the labels accordingly in the PSQL queries.

[volcano1111]

I had the same issue when I defined 2 connection strings in DATA_SOURCE_NAME. Helm chart values states that I can do that. I removed 1 string and everything became fine.