OpenObserve Collector scraping errors

[jhuynh08]

Hello,

I'm receiving errors specifically against scraping kubeAPIserver from the openobserve collector pod logs. openobserve-collector-gateway-collector.log

Errors:

2024-02-20T23:35:07.538Z    warn    internal/transaction.go:123 Failed to scrape Prometheus endpoint    {"kind": "receiver", "name": "prometheus", "data_type": "metrics", "scrape_timestamp": 1708472103095, "target_labels": "{__name__=\"up\", instance=\"10.244.12.199:8080\", job=\"kubeApiServer\"}"}

I have currently running the latest release(v0.8.1) on a self hosted setup with AKS. Below is my helm values configuration. everything was left at default except the values that were mandatory.

values:
    exporters:
      otlphttp/openobserve:
        endpoint: [redacted]
        headers:
          Authorization: Basic [redacted]
      otlphttp/openobserve_k8s_events:
        endpoint: [redacted]
        headers:
          Authorization: Basic [redacted]
          stream-name: k8s_events
    # opentelemetry-operator:
    #   enabled: true
    gateway:
      enabled: true
      affinity: {}
      nodeSelector: 
        kubernetes.io/os: linux
      receivers:
        httpcheck:
          targets:
            - endpoint: [redacted]
              method: GET
            - endpoint: [redacted]
              method: GET
            - endpoint: [redacted]
              method: GET
            - endpoint: [redacted]
              method: GET
            - endpoint: [redacted]
              method: GET
            - endpoint: [redacted]
              method: GET
          collection_interval: 10s
      service:
        extensions: [zpages]
        pipelines:
          logs/k8s_events:
            receivers: [k8s_events]
            processors: [batch, k8sattributes, resourcedetection]
            exporters: [otlphttp/openobserve_k8s_events]
          metrics:
            receivers: [httpcheck, k8s_cluster, prometheus, spanmetrics, servicegraph]
            processors: [batch, k8sattributes, resourcedetection]
            exporters: [otlphttp/openobserve]
          traces:
            receivers: [otlp]
            processors: [batch, k8sattributes, resourcedetection]
            exporters: [otlphttp/openobserve, spanmetrics, servicegraph]

[prabhatsharma]

fixed with https://github.com/openobserve/openobserve-helm-chart/pull/49

[hengfeiyang]

@prabhatsharma The user should run helm repo update first, then install otel-collector again, right?

[prabhatsharma]

Yes, need to update helm repo update and helm upgrade after that.

e.g.

helm -n openobserve-collector -f values.yaml upgrade --install o2c openobserve/openobserve-collector