extraRelabelingRules configuration is not working

[LingamEswar]

Hi team, I want to push the logs of pod/containers whose status is other than success or successfully running. For that i have added extraRelabelingRules rules to the values. yaml. But logs are not pushing as per configuration. Below is my configuration.

logs:

pod_logs:
  enabled: true
  extraRelabelingRules: |
    rule {
      source_labels = ["__meta_kubernetes_pod_phase"]
      regex = "(Failed|Pending|CrashLoopBackOff|Unknown)"
      action = "keep"
    }
    rule {
      source_labels = ["__meta_kubernetes_pod_container_init_status_waiting_reason"]
      regex = ".*"
      action = "keep"
    }
    rule {
      source_labels = ["__meta_kubernetes_pod_container_init_status_terminated_reason"]
      regex = ".*"
      action = "keep"
    }
    rule {
      source_labels = ["__meta_kubernetes_pod_container_status_waiting_reason"]
      regex = ".*"
      action = "keep"
    }
    rule {
      source_labels = ["__meta_kubernetes_pod_container_status_terminated_reason"]
      regex = ".*"
      action = "keep"
    }
    rule {
      source_labels = ["__meta_kubernetes_pod_phase"]
      regex = "Succeeded"
      action = "drop"
    }

As per the configuration, it should drop logs for successful containers and push for failed containers. But i am not getting logs for both. Can some please help me, where i am doing wrong. Its quite urgent. Appreciate the help.

[petewall]

Try just:

pod_logs:
  enabled: true
  extraRelabelingRules: |
    rule {
      source_labels = ["__meta_kubernetes_pod_phase"]
      regex = "(Running|Succeeded)"
      action = "drop"
    }

That'll match the options for the pod role: https://grafana.com/docs/alloy/latest/reference/components/discovery/discovery.kubernetes/#pod-role

The regex .* should match everything, so it'll always evaluate to true. If you want to only keep non-empty values, you should use .+.

[LingamEswar]

@petewall My one of pod is with status "CrashLoopBackOff" but as per pod lifecycle it is running. In this case using above config, logs for the pod with status "CrashLoopBackOff" is not pushing. Any suggestions here. I want logs based pod statuses other than pod phase

[LingamEswar]

@petewall My one of pod is with status "CrashLoopBackOff" but as per pod lifecycle it is running. In this case using above config, logs for the pod with status "CrashLoopBackOff" is not pushing. Any suggestions here. I want logs based pod statuses other than pod phase

@petewall any update on this really helps

[petewall]

Is there anything in the logs for the alloy-logs pods that shows that it's opening tail's for the pods that failed? Could this be related to https://github.com/grafana/k8s-monitoring-helm/issues/632 where your log files are not where Alloy expects? Have you been able to get any logs before trying to filter to non-running pods?

[LingamEswar]

Is there anything in the logs for the alloy-logs pods that shows that it's opening tail's for the pods that failed? Could this be related to #632 where your log files are not where Alloy expects? Have you been able to get any logs before trying to filter to non-running pods?

@petewall no #632 approach is the new one as this is not working. The thing is pod_phase is comming as running even though the status is "CrashLoopBackOff" . So this approach is not helping us, so moved to #632