Prometheus Receiver: kubernetes_sd_configs tag cannot be Unmarshalled

ns-czc commented 3 weeks ago

Describe the bug

When using the Prometheus Receiver with the kubernetes_sd_configs tag, the OpenTelemetry Collector cannot unmarshal the Prometheus config yaml file.

This is the same as https://github.com/open-telemetry/opentelemetry-collector/issues/1867. Maybe upgrading is needed?

Steps to reproduce

Install opentelemetry-operator via helm install opentelemetry-operator opentelemetry/opentelemetry-operator --namespace opentelemetry-operator --create-namespace --version v0.68.1 --set manager.collectorImage.repository=otel/opentelemetry-collector-k8s
Config OpenTelemetryCollector with the following config:

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: collector
  namespace: opentelemetry-operator
spec:
  config:
    receivers:
      prometheus:
        config:
          scrape_configs:
          - job_name: 'kubernetes-pods'
            scrape_interval: 60s
            kubernetes_sd_config:
            - role: pod
            relabel_configs:
            - action: labelmap
              regex: __meta_kubernetes_pod_label_(.+)
            - source_labels: [__meta_kubernetes_namespace]
              action: replace
              target_label: namespace
            - source_labels: [__meta_kubernetes_pod_name]
              action: replace
              target_label: pod
    processors:
      memory_limiter:
        check_interval: 1s
        limit_percentage: 75
        spike_limit_percentage: 15
      batch:
        send_batch_size: 10000
        timeout: 10s

    exporters:
      debug: {}

    service:
      pipelines:
        traces:
          receivers: [prometheus]
          processors: [memory_limiter, batch]
          exporters: [debug]

What did you expect to see?

opentelemetry collector runs and fetch Prometheus metrics according to kubernetes_sd_config.

What did you see instead?

Found the following error in collector pod.

Error: failed to get config: cannot unmarshal the configuration: decoding failed due to the following error(s):

error decoding 'receivers': error reading configuration for "prometheus": decoding failed due to the following error(s):

error decoding 'config': prometheus receiver: failed to unmarshal yaml to prometheus config object: yaml: unmarshal errors:
  line 3: field kubernetes_sd_config not found in type config.ScrapeConfig
2024/08/22 07:35:25 collector server run finished with error: failed to get config: cannot unmarshal the configuration: decoding failed due to the following error(s):

error decoding 'receivers': error reading configuration for "prometheus": decoding failed due to the following error(s):

error decoding 'config': prometheus receiver: failed to unmarshal yaml to prometheus config object: yaml: unmarshal errors:
  line 3: field kubernetes_sd_config not found in type config.ScrapeConfig

What version did you use?

otel/opentelemetry-collector-k8s:0.107.0

What config did you use?

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: collector
  namespace: opentelemetry-operator
spec:
  config:
    receivers:
      prometheus:
        config:
          scrape_configs:
          - job_name: 'kubernetes-pods'
            scrape_interval: 60s
            kubernetes_sd_config:
            - role: pod
            relabel_configs:
            - action: labelmap
              regex: __meta_kubernetes_pod_label_(.+)
            - source_labels: [__meta_kubernetes_namespace]
              action: replace
              target_label: namespace
            - source_labels: [__meta_kubernetes_pod_name]
              action: replace
              target_label: pod
    processors:
      memory_limiter:
        check_interval: 1s
        limit_percentage: 75
        spike_limit_percentage: 15
      batch:
        send_batch_size: 10000
        timeout: 10s

    exporters:
      debug: {}

    service:
      pipelines:
        traces:
          receivers: [prometheus]
          processors: [memory_limiter, batch]
          exporters: [debug]

Environment

GKE v1.29.6-gke.1254000

Additional context

iblancasa commented 3 weeks ago

I think this should be open in https://github.com/open-telemetry/opentelemetry-collector-contrib/issues

iblancasa commented 3 weeks ago

Can it be a typo? kubernetes_sd_config: -> kubernetes_sd_configs:

czchen commented 3 weeks ago

It is indeed typo, thanks for the help.

iblancasa commented 2 weeks ago

Great! Can we close the issue in that case? @czchen

czchen commented 2 weeks ago

this can be closed, thanks for the help.

open-telemetry / opentelemetry-collector-contrib

Prometheus Receiver: kubernetes_sd_configs tag cannot be Unmarshalled #34822