search

grafana / alloy

OpenTelemetry Collector distribution with programmable pipelines
https://grafana.com/oss/alloy
Apache License 2.0
1.48k stars 224 forks source link

k8sattributes processor doesnt work with loki.source.kubernetes #780

Open lieberlois opened 7 months ago

lieberlois commented 7 months ago

What's wrong?

Im trying to setup the otelcol.processor.k8sattributes with Grafana Loki. Unfortunately, the k8sattributes processor seems to not be capable of extracting the k8s metadata from the loki.source.kubernetes module. The module outputs no new labels. My assumption is that it doesnt work because loki.source.kubernetes does not expose k8s.pod.ip, which the k8sattributes processor uses for pod matching

This is my config:

        logging {
          level  = "info"
          format = "logfmt"
        }

        discovery.kubernetes "pods" {
          role = "pod"

          namespaces {
            names = ["default"]
          }
        }

        loki.source.kubernetes "poc" {
          targets    = discovery.kubernetes.pods.targets
          forward_to = [otelcol.receiver.loki.default.receiver]
        }

        otelcol.receiver.loki "default" {
          output {
            logs = [otelcol.processor.k8sattributes.default.input]
          }
        }

        otelcol.receiver.otlp "default" {
          http {}
          grpc {}

          output {
            logs    = [otelcol.processor.k8sattributes.default.input]
          }
        }

        otelcol.processor.k8sattributes "default" {
          extract {
            metadata = [
              "k8s.namespace.name",
              "k8s.deployment.name",
              "k8s.statefulset.name",
              "k8s.daemonset.name",
              "k8s.cronjob.name",
              "k8s.job.name",
              "k8s.node.name",
              "k8s.pod.name",
              "k8s.pod.uid",
              "k8s.pod.start_time",
            ]
          }

          output {
            logs    = [otelcol.processor.batch.default.input]
          }
        }

        otelcol.processor.batch "default" {
          output {
            logs    = [otelcol.exporter.loki.default.input]
          }
        }

        otelcol.exporter.loki "default" {
          forward_to = [loki.echo.debug.receiver]
        }

        loki.echo "debug" {}

The pipeline thus looks like this:

image

The debug component however only writes this to the console (there is a ubuntu container in the default namespace that writes hello world to stdout):

ts=2024-05-07T13:30:29.836589274Z level=info component_path=/ component_id=loki.echo.debug receiver=loki.echo.debug entry="{\"body\":\"hello world\\n\"}" labels="{exporter=\"OTLP\", instance=\"default/ubuntu:ubuntu\", job=\"loki.source.kubernetes.poc\"}"

Steps to reproduce

System information

Linux, Azure AKS

Software version

Grafana Alloy v1.0.0

Configuration

logging {
  level  = "info"
  format = "logfmt"
}

discovery.kubernetes "pods" {
  role = "pod"

  namespaces {
    names = ["default"]
  }
}

loki.source.kubernetes "poc" {
  targets    = discovery.kubernetes.pods.targets
  forward_to = [otelcol.receiver.loki.default.receiver]
}

otelcol.receiver.loki "default" {
  output {
    logs = [otelcol.processor.k8sattributes.default.input]
  }
}

otelcol.receiver.otlp "default" {
  http {}
  grpc {}

  output {
    logs    = [otelcol.processor.k8sattributes.default.input]
  }
}

otelcol.processor.k8sattributes "default" {
  extract {
    metadata = [
      "k8s.namespace.name",
      "k8s.deployment.name",
      "k8s.statefulset.name",
      "k8s.daemonset.name",
      "k8s.cronjob.name",
      "k8s.job.name",
      "k8s.node.name",
      "k8s.pod.name",
      "k8s.pod.uid",
      "k8s.pod.start_time",
    ]
  }

  output {
    logs    = [otelcol.processor.batch.default.input]
  }
}

otelcol.processor.batch "default" {
  output {
    logs    = [otelcol.exporter.loki.default.input]
  }
}

otelcol.exporter.loki "default" {
  forward_to = [loki.echo.debug.receiver]
}

loki.echo "debug" {}

Logs

ts=2024-05-07T13:36:17.89708731Z level=info "boringcrypto enabled"=false
ts=2024-05-07T13:36:17.89712761Z level=info msg="running usage stats reporter"
ts=2024-05-07T13:36:17.89713361Z level=info msg="starting complete graph evaluation" controller_path=/ controller_id="" trace_id=e32481e9e3bb8187e9ea80bfdc70f68b
ts=2024-05-07T13:36:17.89714471Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=e32481e9e3bb8187e9ea80bfdc70f68b node_id=loki.echo.debug duration=34.7µs
ts=2024-05-07T13:36:17.89715211Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=e32481e9e3bb8187e9ea80bfdc70f68b node_id=otelcol.exporter.loki.default duration=374.898µs
ts=2024-05-07T13:36:17.89715801Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=e32481e9e3bb8187e9ea80bfdc70f68b node_id=otelcol.processor.batch.default duration=335.499µs
ts=2024-05-07T13:36:17.89716371Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=e32481e9e3bb8187e9ea80bfdc70f68b node_id=otelcol.processor.k8sattributes.default duration=175.199µs
ts=2024-05-07T13:36:17.89716911Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=e32481e9e3bb8187e9ea80bfdc70f68b node_id=otelcol.receiver.loki.default duration=28.3µs
ts=2024-05-07T13:36:17.89717491Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=e32481e9e3bb8187e9ea80bfdc70f68b node_id=otelcol.receiver.otlp.default duration=161.699µs
ts=2024-05-07T13:36:17.89718051Z level=info msg="Using pod service account via in-cluster config" component_path=/ component_id=discovery.kubernetes.pods
ts=2024-05-07T13:36:17.89718521Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=e32481e9e3bb8187e9ea80bfdc70f68b node_id=discovery.kubernetes.pods duration=785.896µs
ts=2024-05-07T13:36:17.897191009Z level=info msg="Using pod service account via in-cluster config" component_path=/ component_id=loki.source.kubernetes.poc
ts=2024-05-07T13:36:17.897196309Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=e32481e9e3bb8187e9ea80bfdc70f68b node_id=loki.source.kubernetes.poc duration=575.498µs
ts=2024-05-07T13:36:17.897206509Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=e32481e9e3bb8187e9ea80bfdc70f68b node_id=remotecfg duration=97.499µs
ts=2024-05-07T13:36:17.897216709Z level=info msg="applying non-TLS config to HTTP server" service=http
ts=2024-05-07T13:36:17.897223609Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=e32481e9e3bb8187e9ea80bfdc70f68b node_id=http duration=17.7µs
ts=2024-05-07T13:36:17.897234109Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=e32481e9e3bb8187e9ea80bfdc70f68b node_id=ui duration=10.4µs
ts=2024-05-07T13:36:17.897248009Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=e32481e9e3bb8187e9ea80bfdc70f68b node_id=cluster duration=9.5µs
ts=2024-05-07T13:36:17.897258909Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=e32481e9e3bb8187e9ea80bfdc70f68b node_id=otel duration=15.3µs
ts=2024-05-07T13:36:17.897267609Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=e32481e9e3bb8187e9ea80bfdc70f68b node_id=tracing duration=15.6µs
ts=2024-05-07T13:36:17.897279509Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=e32481e9e3bb8187e9ea80bfdc70f68b node_id=logging duration=227.899µs
ts=2024-05-07T13:36:17.897329409Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=e32481e9e3bb8187e9ea80bfdc70f68b node_id=labelstore duration=27.8µs
ts=2024-05-07T13:36:17.897359609Z level=info msg="finished complete graph evaluation" controller_path=/ controller_id="" trace_id=e32481e9e3bb8187e9ea80bfdc70f68b duration=3.333686ms
ts=2024-05-07T13:36:17.897866607Z level=info msg="scheduling loaded components and services"
ts=2024-05-07T13:36:17.898087006Z level=info msg="finished node evaluation" controller_path=/ controller_id="" node_id=loki.source.kubernetes.poc duration=172.1µs
ts=2024-05-07T13:36:17.898423404Z level=warn msg="Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks. Enable the feature gate to change the default and remove this warning." component_path=/ component_id=otelcol.receiver.otlp.default documentation=https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks "feature gate ID"=component.UseLocalHostAsDefaultHost
ts=2024-05-07T13:36:17.898784703Z level=info msg="now listening for http traffic" service=http addr=0.0.0.0:12345
ts=2024-05-07T13:36:17.898901302Z level=info msg="k8s filtering" component_path=/ component_id=otelcol.processor.k8sattributes.default labelSelector="" fieldSelector=""
ts=2024-05-07T13:36:17.899223101Z level=info msg="Starting GRPC server" component_path=/ component_id=otelcol.receiver.otlp.default endpoint=0.0.0.0:4317
ts=2024-05-07T13:36:17.899540999Z level=warn msg="Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks. Enable the feature gate to change the default and remove this warning." component_path=/ component_id=otelcol.receiver.otlp.default documentation=https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks "feature gate ID"=component.UseLocalHostAsDefaultHost
ts=2024-05-07T13:36:17.899753198Z level=info msg="Starting HTTP server" component_path=/ component_id=otelcol.receiver.otlp.default endpoint=0.0.0.0:4318
ts=2024-05-07T13:36:17.899984897Z level=info msg="starting cluster node" peers="" advertise_addr=127.0.0.1:12345
ts=2024-05-07T13:36:17.900304496Z level=info msg="peers changed" new_peers=alloy-infra-0
ts=2024-05-07T13:36:17.900501595Z level=info msg="finished node evaluation" controller_path=/ controller_id="" node_id=otelcol.processor.k8sattributes.default duration=2.37169ms
ts=2024-05-07T13:36:17.900473095Z level=info msg="finished node evaluation" controller_path=/ controller_id="" node_id=otelcol.processor.batch.default duration=2.532889ms
ts=2024-05-07T13:36:17.900646094Z level=info msg="finished node evaluation" controller_path=/ controller_id="" node_id=otelcol.receiver.loki.default duration=98.499µs
ts=2024-05-07T13:36:17.900743894Z level=info msg="finished node evaluation" controller_path=/ controller_id="" node_id=otelcol.receiver.otlp.default duration=2.829988ms
ts=2024-05-07T13:36:17.900768394Z level=info msg="finished node evaluation" controller_path=/ controller_id="" node_id=otelcol.exporter.loki.default duration=2.857088ms
ts=2024-05-07T13:36:22.89945045Z level=info msg="Using pod service account via in-cluster config" component_path=/ component_id=loki.source.kubernetes.poc
ts=2024-05-07T13:36:22.900047648Z level=info msg="tailer running" target=default/nginx-proxy-f95fdf4b-68mpx:nginx component_path=/ component_id=loki.source.kubernetes.poc
ts=2024-05-07T13:36:22.900148447Z level=info msg="tailer running" target=default/blackbox-exporter-prometheus-blackbox-exporter-6b8d77db6b-gjk9k:blackbox-exporter component_path=/ component_id=loki.source.kubernetes.poc
ts=2024-05-07T13:36:22.900283747Z level=info msg="tailer running" target=default/ubuntu:ubuntu component_path=/ component_id=loki.source.kubernetes.poc
ts=2024-05-07T13:36:22.900232747Z level=info msg="finished node evaluation" controller_path=/ controller_id="" node_id=loki.source.kubernetes.poc duration=1.189896ms
ts=2024-05-07T13:36:22.91724249Z level=info msg="opened log stream" target=default/blackbox-exporter-prometheus-blackbox-exporter-6b8d77db6b-gjk9k:blackbox-exporter component_path=/ component_id=loki.source.kubernetes.poc "start time"=1970-01-01T00:00:00.000Z
ts=2024-05-07T13:36:22.917795288Z level=info msg="opened log stream" target=default/ubuntu:ubuntu component_path=/ component_id=loki.source.kubernetes.poc "start time"=1970-01-01T00:00:00.000Z
ts=2024-05-07T13:36:22.919356983Z level=info msg="opened log stream" target=default/nginx-proxy-f95fdf4b-68mpx:nginx component_path=/ component_id=loki.source.kubernetes.poc "start time"=1970-01-01T00:00:00.000Z
ts=2024-05-07T13:36:23.118116118Z level=info component_path=/ component_id=loki.echo.debug receiver=loki.echo.debug entry="{\"body\":\"ts=2024-05-03T12:33:09.758Z caller=main.go:87 level=info msg=\\\"Starting blackbox_exporter\\\" version=\\\"(version=0.25.0, branch=HEAD, revision=ef3ff4fef195333fb8ee0039fb487b2f5007908f)\\\"\\n\"}" labels="{exporter=\"OTLP\", instance=\"default/blackbox-exporter-prometheus-blackbox-exporter-6b8d77db6b-gjk9k:blackbox-exporter\", job=\"loki.source.kubernetes.poc\"}"
ts=2024-05-07T13:36:23.118209518Z level=info component_path=/ component_id=loki.echo.debug receiver=loki.echo.debug entry="{\"body\":\"ts=2024-05-03T12:33:09.758Z caller=main.go:88 level=info build_context=\\\"(go=go1.22.2, platform=linux/amd64, user=root@47d5b0d99f18, date=20240409-12:58:39, tags=unknown)\\\"\\n\"}" labels="{exporter=\"OTLP\", instance=\"default/blackbox-exporter-prometheus-blackbox-exporter-6b8d77db6b-gjk9k:blackbox-exporter\", job=\"loki.source.kubernetes.poc\"}"
ts=2024-05-07T13:36:23.118242918Z level=info component_path=/ component_id=loki.echo.debug receiver=loki.echo.debug entry="{\"body\":\"ts=2024-05-03T12:33:09.759Z caller=main.go:100 level=info msg=\\\"Loaded config file\\\"\\n\"}" labels="{exporter=\"OTLP\", instance=\"default/blackbox-exporter-prometheus-blackbox-exporter-6b8d77db6b-gjk9k:blackbox-exporter\", job=\"loki.source.kubernetes.poc\"}"
ts=2024-05-07T13:36:23.118258318Z level=info component_path=/ component_id=loki.echo.debug receiver=loki.echo.debug entry="{\"body\":\"ts=2024-05-03T12:33:09.760Z caller=tls_config.go:313 level=info msg=\\\"Listening on\\\" address=[::]:9115\\n\"}" labels="{exporter=\"OTLP\", instance=\"default/blackbox-exporter-prometheus-blackbox-exporter-6b8d77db6b-gjk9k:blackbox-exporter\", job=\"loki.source.kubernetes.poc\"}"
ts=2024-05-07T13:36:23.118277118Z level=info component_path=/ component_id=loki.echo.debug receiver=loki.echo.debug entry="{\"body\":\"ts=2024-05-03T12:33:09.760Z caller=tls_config.go:316 level=info msg=\\\"TLS is disabled.\\\" http2=false address=[::]:9115\\n\"}" labels="{exporter=\"OTLP\", instance=\"default/blackbox-exporter-prometheus-blackbox-exporter-6b8d77db6b-gjk9k:blackbox-exporter\", job=\"loki.source.kubernetes.poc\"}"
ts=2024-05-07T13:36:23.118311418Z level=info component_path=/ component_id=loki.echo.debug receiver=loki.echo.debug entry="{\"body\":\"hello world\\n\"}" labels="{exporter=\"OTLP\", instance=\"default/ubuntu:ubuntu\", job=\"loki.source.kubernetes.poc\"}"
github-actions[bot] commented 5 months ago

This issue has not had any activity in the past 30 days, so the needs-attention label has been added to it. If the opened issue is a bug, check to see if a newer release fixed your issue. If it is no longer relevant, please feel free to close this issue. The needs-attention label signals to maintainers that something has fallen through the cracks. No action is needed by you; your issue will be kept open and you do not have to respond to this comment. The label will be removed the next time this job runs if there is new activity. Thank you for your contributions!

algo7 commented 1 month ago

any news?

lieberlois commented 1 month ago

@algo7 I think it was just a matter of bad docs. Apparently the k8sattributes modules only really work with the push based approach via OTLP, at least from what i figured.