Push only failed container logs

[LingamEswar]

Hi team,

I was looking for configuration that helps me to push only logs for the failed containers. But i am not finding the best configuration fo this. Can someone help me on this. When i was checking with bot on grafana website . It showed me something like mentioned below. But that Promtail configuration i am not seeing in values.yaml of k8s-monitoring chart. So can some one help me on this.

---

To push logs only for failed containers in Grafana, you can use Promtail's label filtering feature. Promtail is the agent responsible for scraping logs and sending them to Loki.

You can configure Promtail to filter logs based on specific labels, such as the container status. Here's an example of how you can achieve this:

Edit the values.yaml file of your Grafana Helm chart. Locate the promtail section and add the following configuration under extraScrapeConfigs: extraScrapeConfigs:

• job_name: kubernetes-pods kubernetes_sd_configs:
  • role: pod relabel_configs:
  • source_labels: [__meta_kubernetes_pod_container_status] regex: Failed action: keep
  • source_labels: [__meta_kubernetes_pod_container_status] regex: .* action: drop This configuration will filter logs only for containers with a status of "Failed" and drop logs for containers with any other status.

Save the changes and redeploy the Grafana Helm chart. After applying this configuration, Promtail will only push logs for failed containers to Loki, providing you with the desired visibility into the failed containers' logs.

[petewall]

This should definitely be doable with a few rules in the discovery portion of the pod logs pipeline. We just need to translate them into Alloy rules.

logs:
  pod_logs:
    extraRelabelingRules: |
      rule {
        source_labels = ["__meta_kubernetes_pod_container_status"]
        regex = "Failed"
        action = "keep"
      }

I don't think we need the other rule because the first keep rule should implicitly drop anything that doesn't match the regex.

[LingamEswar]

@petewall thanks a lot for configuration, but it is stopping the logs from successful container and it is not even pushing for failed container as well. I mean container which is with status CrashLoopBackOff.

One more thing is this label __meta_kubernetes_pod_container_status will be default available for containers or do we need to add them

[petewall]

You may need to adjust as necessary. here are the list of phases: https://github.com/kubernetes/kubernetes/blob/v1.27.2/pkg/apis/core/types.go#L2514-L2532