Discovery drops on a non-existent label

[adamhackl]

This same rule is found across various modules as the first discovery.relabel rule. It does a keep based on labels that the pod has a port name defined or one named metrics, is running and ready. But it also checks that the container is not an init container. The problem I have is that I don't have the label for init containers on any of my scrapes. So I'm getting a lot of stuff dropped that shouldn't be. This came to light because I can't get the kube_dns scrape in the core module to work and removing this fourth label from the keep caused it to start working.

I'm happy to submit a PR, but I'm wondering if the right move is to remove this last keep requirement for __meta_kubernetes_pod_container_init ... we could add a drop rule after this rule for if __meta_kubernetes_pod_container_init=true to solve what I think this is trying to do. Can anyone confirm my solution is correct and I'll get it PRd to all the places this exists?

  // kube_dns relabelings (pre-scrape)
  discovery.relabel "kube_dns" {
    targets = discovery.kubernetes.kube_dns.targets

    // keep only the specified metrics port name, and pods that are Running and ready
    rule {
      source_labels = [
        "__meta_kubernetes_pod_container_port_name",
        "__meta_kubernetes_pod_phase",
        "__meta_kubernetes_pod_ready",
        "__meta_kubernetes_pod_container_init",
      ]
      separator = "@"
      regex = coalesce(argument.port_name.value, "metrics") + "@Running@true@false"
      action = "keep"
    }

  // kube_dns relabelings (pre-scrape)
  discovery.relabel "kube_dns" {
    targets = discovery.kubernetes.kube_dns.targets

    // keep only the specified metrics port name, and pods that are Running and ready
    rule {
      source_labels = [
        "__meta_kubernetes_pod_container_port_name",
        "__meta_kubernetes_pod_phase",
        "__meta_kubernetes_pod_ready",
      ]
      separator = "@"
      regex = coalesce(argument.port_name.value, "metrics") + "@Running@true"
      action = "keep"
    }

    // drop any init containers
    rule {
      source_labels = ["__meta_kubernetes_pod_container_init"]
      regex = "true"
      action = "drop"
    }

[adamhackl]

40

[adamhackl]

PR merged