Dropping logs

[tcborg]

Hello fellas,

I am trying to reduce the amount of logs I ingest, but I am unsure what is happening. I tried different ways, but none seemed to work. For instance, trying to drop logs from default and kube-system namespaces, I tried like this:

externalServices:
    logs:
         pod_logs:
              extraRelabelingRules: |-
                rule {
                   action        = "labeldrop"
                   regex         = "default|kube-system"
                }

and

externalServices:
       logs:
            pod_logs:
              extraRelabelingRules: |
                rule {
                   action        = "drop"
                   source_labels = ["__meta_kubernetes_namespace", "namespace"]
                   regex         = "default|kube-system"
                }

Am I doing something wrong?

[petewall]

You're very close! The drop action is what you need, but source_labels = ["__meta_kubernetes_namespace", "namespace"] will result in both labels being combined into something like: default;default

This rule should work:

                rule {
                   action        = "drop"
                   source_labels = ["namespace"]
                   regex         = "default|kube-system"
                }

This works because we already set namespace to the same as the meta label __meta_kubernetes_namespace: https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/templates/alloy_config/_pod_logs_discovery.alloy.txt#L15-L19

[tcborg]

You're very close! The drop action is what you need, but source_labels = ["__meta_kubernetes_namespace", "namespace"] will result in both labels being combined into something like: default;default

This rule should work:

                rule {
                   action        = "drop"
                   source_labels = ["namespace"]
                   regex         = "default|kube-system"
                }

This works because we already set namespace to the same as the meta label __meta_kubernetes_namespace: https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/templates/alloy_config/_pod_logs_discovery.alloy.txt#L15-L19

Thanks @petewall, but that didn't work :( I still see the logs coming

[petewall]

I just tested this and it's working for my deployment. Sometimes, updated the Helm deployment will update the ConfigMap, but it takes a while for the Alloy pod to get the latest. Restarting your alloy logs instances would pick it up faster.

[tcborg]

@petewall that gave me some insights. The templating is not working for some reason, the configmap is not getting updated. But when I edit the config map directly and restarts the pod, it is working.

That is enough for me to move forward. Thank you very much @petewall !!!