otelcol.exporter.loadbalancing fails with no configuration has been provided, try setting KUBERNETES_MASTER environment variable

bvsvas commented 2 days ago

Component(s)

exporter/loadbalancing

What happened?

Description

Loadbalancing exporter fails to start.

Error message:

2024-07-01T10:20:51.981Z        debug   extension@v0.103.0/extension.go:170     Beta component. May change in the future.       {"kind": "extension", "name": "health_check"}
2024-07-01T10:20:51.982Z        debug   exporter@v0.103.0/exporter.go:278       Beta component. May change in the future.       {"kind": "exporter", "data_type": "traces", "name": "loadbalancing"}
Error: failed to build pipelines: failed to create "loadbalancing" exporter for data type "traces": invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
2024/07/01 10:20:51 collector server run finished with error: failed to build pipelines: failed to create "loadbalancing" exporter for data type "traces": invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable

Note: It works if I change k8s resolver to dns resolver. This error comes only with k8s resolver (configuration in the below).

Steps to Reproduce

Create OpenTelemetry Collector with loadbalance exporter

Image: otel/opentelemetry-collector-contrib:0.103.0

Loadbalance exporter configuration

    extensions:
      health_check:
        endpoint: 0.0.0.0:13133
        path: "/health"
    exporters:
      loadbalancing:
        protocol:
          otlp:
            timeout: 1s
            tls:
              insecure: true
        resolver:
          k8s:
            service: dx-otelcol-ingester-service.dxotel
            ports:
            - 4318
            - 4317
    processors:
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
    service:
      telemetry:
        logs:
          level: debug
      extensions: [health_check]
      pipelines:
        metrics:
          exporters:
            - loadbalancing
          processors: []
          receivers:
            - otlp
        traces:
          exporters:
            - loadbalancing
          processors: []
          receivers:
            - otlp

Deployment: (only container section pasted here)

   containers:
      - command:
        - /otelcol-contrib
        - --config=/etc/otel/conf/dx-collector-lb.yaml
        env:
        - name: OTEL_POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: OTEL_POD_IP
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.podIP
        - name: KUBERNETES_MASTER
          value: https://kubernetes.default.svc
        - name: KUBERNETES_SERVICE_HOST
          value: kubernetes.default.svc
        - name: KUBERNETES_SERVICE_PORT
          value: "443"
        image: otel/opentelemetry-collector-contrib:0.103.0
        imagePullPolicy: Always
        name: otel-lb
        ports:
        - containerPort: 4317
          name: otlp-grpc
          protocol: TCP
        - containerPort: 4318
          name: otlp-http
          protocol: TCP
        resources: {}
        securityContext: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/otel/conf
          name: otel-collector-configmap-vol

Note: Created role and role binding with endpoint, services, pods as well.

Expected Result

otelcol pod up and running with loadbalancing exporter.

Actual Result

pod is in CrashLoopBackOff with error.

Collector version

0.103.0

Environment information

Environment

OS: CentOS v8 Kubernetes Server: v1.28.3

OpenTelemetry Collector configuration

extensions:
      health_check:
        endpoint: 0.0.0.0:13133
        path: "/health"
    exporters:
      loadbalancing:
        protocol:
          otlp:
            timeout: 1s
            tls:
              insecure: true
        resolver:
          k8s:
            service: dx-otelcol-ingester-service.dxotel
            ports:
            - 4318
            - 4317
    processors:
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
    service:
      telemetry:
        logs:
          level: debug
      extensions: [health_check]
      pipelines:
        metrics:
          exporters:
            - loadbalancing
          processors: []
          receivers:
            - otlp
        traces:
          exporters:
            - loadbalancing
          processors: []
          receivers:
            - otlp

Log output

2024-07-01T10:31:56.738Z        info    service@v0.103.0/service.go:115 Setting up own telemetry...
2024-07-01T10:31:56.738Z        info    service@v0.103.0/telemetry.go:96        Serving metrics {"address": ":8888", "level": "Normal"}
2024-07-01T10:31:56.738Z        debug   extension@v0.103.0/extension.go:170     Beta component. May change in the future.       {"kind": "extension", "name": "health_check"}
2024-07-01T10:31:56.738Z        info    exporter@v0.103.0/exporter.go:280       Development component. May change in the future.        {"kind": "exporter", "data_type": "metrics", "name": "loadbalancing"}
Error: failed to build pipelines: failed to create "loadbalancing" exporter for data type "metrics": invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
2024/07/01 10:31:56 collector server run finished with error: failed to build pipelines: failed to create "loadbalancing" exporter for data type "metrics": invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable

Additional context

No response

github-actions[bot] commented 2 days ago

Pinging code owners:

exporter/loadbalancing: @jpkrohling

See Adding Labels via Comments if you do not have permissions to add labels yourself.

codefromthecrypt commented 2 days ago

This error is from the kubernetes client when it has cannot find a valid config https://github.com/kubernetes/client-go/blob/ab86e03da476bd58948f318b4f73d67d0801fcbd/tools/clientcmd/validation.go#L33

Maybe double-check your serviceAccount similar to the example README?

https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/loadbalancingexporter/example/k8s-resolver/README.md

bvsvas commented 2 days ago

Yes, it seems the issue was related to the Kubernetes cluster setup on our local environment. We reinstalled the Kubernetes cluster, and it appears to be working fine now.

open-telemetry / opentelemetry-collector-contrib