Promtail - Ability to filter for labels in clients config

[butschi84]

problem We run promtail to collect logs from our openshift kubernetes Plattform

• The openshift plattform itself is managed by our team openshift-mgmt.
• The business applications / workloads / pods that run on the plattform are managed by our team businessapp-mgmt.

This means I should be able to send the logs with label, say "team=openshift-mgmt" to loki-tenant openshift-mgmt and logs with "team=businessapp-mgmt" to loki tenant businessapp-mgmt.

I see no configuration option at the moment to achieve this.

possible solution I suggest a modification in the promtail config "clients" section to be able to filter the logs like:

clients:
   - url: http://loki
     headers:
        X-Scope-OrgID: openshift-mgmt
     label_filters: 
       - labels: [team]
         regex: openshift-mgmt
     action: keep
   - url: http://loki
     headers:
        X-Scope-OrgID: businessapp-mgmt
     label_filters: 
       - labels: [team]
         regex: businessapp-mgmt
     action: keep

Alternatives I found no way to achieve the desired split-sending with the available config options. Other tools like fluentD support this split-sending

[vlad-diachenko]

I believe you can achieve the same behavior using tenant pipeline stage. Please let me know if it works for you

[butschi84]

@vlad-diachenko Thank you a lot for this input. I just tested and this seems to work. Two problems though - might you have also an idea how to solve?

• I have now the label loki_customer_idin all logs, which I used for the tenant pipeline stage but otherwise I don't need
• We also plan to be using "basic authentication". Each tenant will have a different password. The team businessapp-mgmt cannot be allowed to read the cluster logs of openshift-mgmt

[vlad-diachenko]

I have now the label loki_customer_idin all logs, which I used for the tenant pipeline stage but otherwise I don't need

you can add labeldrop stage right after tenant stage

We also plan to be using "basic authentication". Each tenant will have a different password. The team businessapp-mgmt cannot be allowed to read the cluster logs of openshift-mgmt

can you allow to promtail to write the logs for both tenants? but the end users will use their own credentials to access a single tenant. btw, what are you going to use for basic auth? your own applications or something else?

[butschi84]

Alright I think that's fine, we can make the write path unprotected and all other paths protected. We can close this issue.

Regarding basic auth, I already did a pilot using a little modification in the nginx config of the gateway in the "loki-distributed" helm chart. That worked fine. But I believe for production we'll be using a whole custom nginx-auth-proxy in front of loki because I like to keep such modifications of the loki-distributed helm chart at a minimum and hope to make future version updates easier that way.

[vlad-diachenko]

ok, awesome ;) was happy to help