search

grafana / loki

Like Prometheus, but for logs.
https://grafana.com/loki
GNU Affero General Public License v3.0
23.16k stars 3.36k forks source link

[promtail] drop stage - negate RE2 match #5758

Open luispintomartins opened 2 years ago

luispintomartins commented 2 years ago

Is your feature request related to a problem? Please describe. In promtail, the drop stage allows to drop lines that match a specific pattern, so it's easy to implement a blacklist of bad expressions. However, if the user wants to drop all lines that do not match a specific expression, to implement a whitelist of good expressions, this is not so easily achievable.

Describe the solution you'd like It would be nice to have in the drop stage an option that negates the matching result (e.g. if the expression does not match => drop the line).

Describe alternatives you've considered Unfortunately it seems RE2 doesn't allow negative lookahead/lookbehind ? https://github.com/google/re2/issues/156

Additional context

steverweber commented 1 year ago

I also hit this issue when trying to drop logs. In my case i wanted to drop logs where the host is not FQDN. Not sure if this works but im trying to use the not selector to get it working https://grafana.com/docs/loki/latest/logql/log_queries/

    pipeline_stages:
      -
        match:
          selector: '{host!~".+\\..+"}'
          action: drop
cstyan commented 9 months ago

Hello, thanks for your feature request!

We're currently reevaluating promtails position as a project within Grafana Labs. Internally we're actually using the Agent for both metrics and logs collection at this point.

While we haven't made a formal decision yet, we expect in the near future that all new feature work will be done in the Agent's log collection pipelines rather than in Promtail.