multiple targets, creates duplicate metrics

[cscottthomas]

Hey, we're using the exporter as described in the readme, and getting odd results.

If we have more than one target to check against, we see results for both targets against each ilo_host:

Metrics page queried directly against ilo1_fqdn.domain:

# HELP hpilo_storage HP iLO storage status
# TYPE hpilo_storage gauge
hpilo_storage{product_name="ProLiant DL560 Gen9",server_name="ilo1_fqdn.domain"} 0.0
hpilo_storage{product_name="ProLiant DL560 Gen9",server_name="ilo2_fqdn.domain"} 0.0

Prometheus metrics query:

hpilo_battery{ilo_host="ilo1_fqdn.domain",instance="hpilo_crsm:9416",job="crsm_hpilo",monitor="Cerberus-CRSM",product_name="ProLiant DL560 Gen9",server_name="ilo1_fqdn.domain"}    0
hpilo_battery{ilo_host="ilo1_fqdn.domain",instance="hpilo_crsm:9416",job="crsm_hpilo",monitor="Cerberus-CRSM",product_name="ProLiant DL560 Gen9",server_name="ilo2_fqdn.domain"}    0
hpilo_battery{ilo_host="ilo2_fqdn.domain",instance="hpilo_crsm:9416",job="crsm_hpilo",monitor="Cerberus-CRSM",product_name="ProLiant DL560 Gen9",server_name="ilo1_fqdn.domain"}    0
hpilo_battery{ilo_host="ilo2_fqdn.domain",instance="hpilo_crsm:9416",job="crsm_hpilo",monitor="Cerberus-CRSM",product_name="ProLiant DL560 Gen9",server_name="ilo2_fqdn.domain"}    0

As you can see we have one job but results for both targets, are coming back on each.

Our docker container is deployed as:

  hpilo_crsm:
    image: idnt/hpilo-exporter
    ports:
      - 9416:9416
    command:
      - '--port=9416'
    networks:
      - monitor-net
    deploy:
      placement:
        constraints:
          - node.labels.datacenter==crsm

our prometheus.yml:

  - job_name: 'crsm_hpilo'
    scrape_interval: 1m
    params:
      ilo_port: ['443']
      ilo_user: ['username']
      ilo_password: ['password']
    file_sd_configs:
    - files:
      - 'crsm_hpilo.yml'
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_ilo_host
      - source_labels: [__param_ilo_host]
        target_label: ilo_host
      - target_label: __address__
        replacement: hpilo_crsm:9416 # hpilo exporter.

Our crsm_hpilo.yml (target file):

- targets:
    - ilo1_fqdn.domain
    - ilo2_fqdn.domain
  labels:
    env: coreptl_crsm
    job: crsm_ilo

Are we doing something wrong? It feels like the only way to get accurate metrics is to have a separate exporter per iLO. When we're talking 100s of iLO's to query, this seems ...ugh.

Having the ability to add user/pass for all iLOs feels like this is not intended however?

I've tried separate jobs in prometheus which still produced the same result.

Any help would be appreciated. Cheers, Chris.

[4id3y]

I have rebuilt the container from the src and the dupe goes away. I assume the latest image on the repo must be broken.

[cscottthomas]

Grand, can close then. hopefully this will be helpful to anyone else bumping into this. 👍