Two prometheus receivers cannot be defined.

[kahirokunn]

Is your feature request related to a problem? Please describe.

Two prometheus receivers cannot be defined. The effect of this is that when I use targetAllocator, all static_configs are overwritten and I cannot get cadvisor metrics in EKS as shown below. When using the targetAllocator, static_configs are automatically rewritten.

receivers:
  prometheus:
    config:
      scrape_configs:
      - honor_labels: true
        job_name: otel-collector
        metric_relabel_configs:
        - action: labeldrop
          regex: (id|name)
          replacement: $$1
        - action: labelmap
          regex: label_(.+)
          replacement: $$1
        scrape_interval: 10s
        static_configs:
        - targets:
          - 0.0.0.0:8888
    target_allocator:
      collector_id: ${POD_NAME}
      endpoint: http://otel-targetallocator
      interval: 10s

However, there are cases where I want to set static_configs directly.

receivers:
  prometheus:
    config:
      scrape_configs:
      - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        honor_labels: true
        job_name: kubernetes-nodes-cadvisor
        kubernetes_sd_configs:
        - role: node
        scheme: https
        scrape_interval: 30s
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecure_skip_verify: true

If I write it like this, the settings for cadvisor will also be changed.

receivers:
  prometheus:
    config:
      scrape_configs:
      - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        honor_labels: true
        job_name: kubernetes-nodes-cadvisor
        kubernetes_sd_configs:
        - role: node
        scheme: https
        scrape_interval: 30s
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecure_skip_verify: true
      - honor_labels: true
        job_name: otel-collector
        metric_relabel_configs:
        - action: labeldrop
          regex: (id|name)
          replacement: $$1
        - action: labelmap
          regex: label_(.+)
          replacement: $$1
        scrape_interval: 10s
        static_configs:
        - targets:
          - 0.0.0.0:8888
    target_allocator:
      collector_id: ${POD_NAME}
      endpoint: http://otel-targetallocator
      interval: 10s

However, defining two receivers like this returns an error.

receivers:
  prometheus/ta:
    config:
      scrape_configs:
      - honor_labels: true
        job_name: otel-collector
        metric_relabel_configs:
        - action: labeldrop
          regex: (id|name)
          replacement: $$1
        - action: labelmap
          regex: label_(.+)
          replacement: $$1
        scrape_interval: 10s
        static_configs:
        - targets:
          - 0.0.0.0:8888
    target_allocator:
      collector_id: ${POD_NAME}
      endpoint: http://otel-targetallocator
      interval: 10s
  prometheus/static:
    config:
      scrape_configs:
      - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        honor_labels: true
        job_name: kubernetes-nodes-cadvisor
        kubernetes_sd_configs:
        - role: node
        scheme: https
        scrape_interval: 30s
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecure_skip_verify: true

Describe the solution you'd like

I would like to be able to define multiple prometheus receivers.

receivers:
  prometheus/ta:
    config:
      scrape_configs:
      - honor_labels: true
        job_name: otel-collector
        metric_relabel_configs:
        - action: labeldrop
          regex: (id|name)
          replacement: $$1
        - action: labelmap
          regex: label_(.+)
          replacement: $$1
        scrape_interval: 10s
        static_configs:
        - targets:
          - 0.0.0.0:8888
    target_allocator:
      collector_id: ${POD_NAME}
      endpoint: http://otel-targetallocator
      interval: 10s
  prometheus/static:
    config:
      scrape_configs:
      - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        honor_labels: true
        job_name: kubernetes-nodes-cadvisor
        kubernetes_sd_configs:
        - role: node
        scheme: https
        scrape_interval: 30s
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecure_skip_verify: true

Describe alternatives you've considered

None

Additional context Add any other context or screenshots about the feature request here.

Please delete paragraphs that you did not use before submitting.

[github-actions[bot]]

Pinging code owners for receiver/prometheus: @Aneurysm9 @dashpole. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[bryan-aguilar]

This issue seems more appropriate for https://github.com/open-telemetry/opentelemetry-operator. Could you please open an issue there instead?

[github-actions[bot]]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/prometheus: @Aneurysm9 @dashpole

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions[bot]]

This issue has been closed as inactive because it has been stale for 120 days with no activity.

[kahirokunn]

/reopen

[github-actions[bot]]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/prometheus: @Aneurysm9 @dashpole

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[kahirokunn]

Keep

[github-actions[bot]]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/prometheus: @Aneurysm9 @dashpole

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[kahirokunn]

done. https://github.com/open-telemetry/opentelemetry-operator/issues/3034