loki.relabel not creating labels with regex

emilrowland commented 3 months ago

What's wrong?

I have configured a Loki pipeline to set some labels from the scraped pod logs. Unfortunately, no labels are set after the relabeler

Steps to reproduce

Deploy Alloy with the below configurations as a DeamonSet

System information

Amazon Linux 2 x86_64

Software version

docker.io/grafana/alloy:v1.3.0

Configuration

alloy:
  clustering:
    enabled: true
  alloy:
    stabilityLevel: experimental
    mounts:
      varlog: true
    configMap:
      content: |-
        local.file_match "pod_logs" {
            path_targets = [{"__path__" = "/var/log/pods/*/*/*.log"}]
            sync_period = "5s"
        }
        loki.source.file "log_scrape" {
          targets    = local.file_match.pod_logs.targets
          forward_to = [loki.relabel.labels.receiver]
          tail_from_end = true
        }
        loki.relabel "labels" {
          forward_to = [loki.write.default.receiver]
          rule {
            action = "replace"
            source_labels = ["filename"]
            regex = "/var/log/pods/([^/]+)_([^_/]+)_"
            target_label = "namespace"
            replacement = "$1"
          }
          rule {
            action = "replace"
            source_labels = ["filename"]
            regex = "/var/log/pods/([^/]+)_([^_/]+)_"
            target_label = "pod_name"
            replacement = "$2"
          }
        }
        loki.write "default" {
          endpoint {
            url = "http://observability-infra-loki-gateway/loki/api/v1/push"
          }
        }

        livedebugging {
          enabled = true
        }

Logs

entry: 2024-08-07T14:36:39.174435045Z stderr F level=info ts=2024-08-07T14:36:39.085943835Z caller=table_manager.go:136 index-store=tsdb-2024-04-01 msg="uploading tables", labels: {filename="/var/log/pods/observability_loki-write-1_679ad592-68cf-4fe7-a8c4-825dcf7dba53/loki/0.log"} => {filename="/var/log/pods/observability_loki-write-1_679ad592-68cf-4fe7-a8c4-825dcf7dba53/loki/0.log"}

github-actions[bot] commented 2 months ago

This issue has not had any activity in the past 30 days, so the needs-attention label has been added to it. If the opened issue is a bug, check to see if a newer release fixed your issue. If it is no longer relevant, please feel free to close this issue. The needs-attention label signals to maintainers that something has fallen through the cracks. No action is needed by you; your issue will be kept open and you do not have to respond to this comment. The label will be removed the next time this job runs if there is new activity. Thank you for your contributions!

geoffmore commented 1 month ago

Hey, @emilrowland, I had a similar issue when I was trying to get labels on pod logs. I just got my config working. Hopefully this helps!

System information

Kubelet: v1.30.5+k3s1 OS: Ubuntu 24.04 LTS

Software version

docker.io/grafana/alloy:v1.4.2

Configuration (config.alloy)

logging {
  level = "info"
  format = "json"
}

// https://grafana.com/docs/alloy/latest/reference/components/discovery/discovery.kubernetes/#pod-role
discovery.kubernetes "pods" {
  role = "pod"
}

// https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/
discovery.relabel "k8scommon" {
  targets = discovery.kubernetes.pods.targets
  rule {
    action = "labelmap"
    regex = "__meta_kubernetes_pod_label_app_kubernetes_io_(.+)"
    replacement = "app_kubernetes_io_${1}"
  }
  rule {
    action = "labelmap"
    regex = "__meta_(kubernetes_namespace|kubernetes_pod_name|kubernetes_pod_container_name)"
    replacement = "${1}"
  }
  rule {
    action = "labelmap"
    regex = "__meta_(kubernetes_pod_ip|kubernetes_pod_uid|kubernetes_pod_node_name)"
    replacement = "${1}"
  }
  // Insert other k8s rules here
}

loki.source.kubernetes "pods" {
  targets    = discovery.relabel.k8scommon.output
  forward_to = [loki.process.metadata.receiver]
}

loki.process "metadata" {
  // https://community.grafana.com/t/so-when-to-use-structured-metadata-and-when-to-use-labels/120337
  stage.structured_metadata {
    values = {
      kubernetes_pod_ip = "",
      kubernetes_pod_uid = "",
      kubernetes_pod_node_name = "",
    }
  }
  forward_to = [loki.write.grafanacloud.receiver]
}

loki.write "grafanacloud" {
  external_labels = {
    cluster = sys.env("CLUSTER"),
  }

  endpoint {
    // TODO - capture URL as var
    url = "https://logs-prod-us-central1.grafana.net/loki/api/v1/push"
    basic_auth {
      username = sys.env("GRAFANA_USERNAME")
      password = sys.env("GRAFANA_PASSWORD")
    }
  }
}

grafana / alloy