search

open-telemetry / opentelemetry-rust

The Rust OpenTelemetry implementation
https://opentelemetry.io
Apache License 2.0
1.82k stars 424 forks source link

[Feature]: Severity filter support in BatchLogProcessor #1881

Open scb01 opened 3 months ago

scb01 commented 3 months ago

Related Problems?

No response

Describe the solution you'd like:

I would like to be able to configure an exporter to only exports logs at a specified severity or above. For example, in my app, I want to log at a default level of debug, but only export logs of level info and above to an open telemetry collector.

I looked at the logs_level_enabled feature, but based on what I saw in the implementation, it is just a place holder at this point as it is hardcoded to always return true.

A way to accomplish this would be to have a parameter, say export_severity added to the BatchLogConfig and have that be honored in the BatchLogProcessor. I am happy to send a PR if there is agreement with this approach, but wanted to check if this is in line with the plans for full support for the logs_level_enabled feature.

Considered Alternatives

No response

Additional Context

No response

cijothomas commented 3 months ago

You can do this upstream, even before OpenTelemetry itself. tracing offers filtering mechanism. An example within this repo can be found here : https://github.com/open-telemetry/opentelemetry-rust/blob/main/opentelemetry-otlp/examples/basic-otlp/src/main.rs#L108-L116

lalitb commented 3 months ago

Any reason not to create a tokio tracing filter to achieve it, and so drop the event before reaching the otel pipeline. In case the exporter needs to decide, as of now you would have to create a custom processor as done here. I don't remember why BatchLogProcessor was hardwired to be always enabled, probably because the feature was more relevant for the synchronous processors.

scb01 commented 3 months ago

Sorry about not being clearer in my request. The idea is to be able to do this on a per exporter basis. For example, my application logs at debug level and I have two exporters a) stdout exporter and b) otlp exporter. I want to send all logs to exporter a, but only send logs of severity info and above to exporter b.

cijothomas commented 3 months ago

mm.. 2 loggerproviders (1 with stdout, another with otlp), and create 2 layers with diff. filtering - this should be possible.

These are things typically done at Collector level. Also - are you using stdout exporter in production in anyway ?

scb01 commented 3 months ago

No, not planning to use stdout exporter in production, but just using it here as an example.

scb01 commented 3 months ago

Also, should clarify that I'm using the rust log: https://crates.io/crates/log and not tracing: https://crates.io/crates/tracing for application logging.

lalitb commented 3 months ago

mm.. 2 loggerproviders (1 with stdout, another with otlp), and create 2 layers with diff. filtering - this should be possible.

This is the right way to achieving it. Though it would more straightforward with tracing layers, should be also doable with log.