dora-metrics / pelorus

Automate the measurement of organizational behavior
https://pelorus.readthedocs.io/
Apache License 2.0
245 stars 83 forks source link

Incorrect scrape config secret when adding external prometheus host #1142

Closed SendToNull closed 4 months ago

SendToNull commented 4 months ago

OpenShift version

Not related to OpenShift

Problem description

I have a pelorus CRD that we are deploying where we are trying to add an external Prometheus host (show below)

kind: Pelorus
apiVersion: charts.pelorus.dora-metrics.io/v1alpha1
metadata:
  name: pelorus-devspaces
spec:
  federate_openshift_monitoring:
    enabled: true
    metrics_filter:
      # pull devspaces metrics we're interested in
      - '{namespace=~".*-devspaces",pod=~"workspace.+"}'
  external_prometheus_hosts:
    - id: 'prometheus-ldap'
      hostname: 'custom-prometheus-service.pelorus-operator.svc.cluster.local:9091'
  exporters:
    instances: []
  prometheus_storage: true
  prometheus_storage_pvc_capacity: 3Gi
  prometheus_storage_pvc_storageclass: gp2
  prometheus_retention_size: 2GB

When we add the external_prometheus_hosts, the additional scrape config picks up the change and adds the new host as an additional scrape source. However, it appears that the formatting of the secret is incorrect, and the external Prometheus host is not showing up in the targets on the Pelorus Prometheus instance.

Steps to reproduce

  1. Create a Pelorus deployment that adds an external prometheus host
  2. Observe the additional scrape config secret has been updated
  3. Check the targets on the prometheus-pelorus instance (via curl -s 'http://prometheus-pelorus.pelorus-operator.svc.cluster.local:9090/api/v1/targets' | jq)

Current behavior

Currently, the additional scrape config is not being picked up properly and is not being added to the prometheus pelorus instance.

Expected behavior

When we add an external Prometheus host, we should see the additional scrape config being shown in the targets on the pelorus Prometheus instance.

Code of Conduct