search

fzakaria / slf4j-timbre

SLF4J binding for Clojure's Timbre
Eclipse Public License 1.0
94 stars 24 forks source link

High log volume can cause serious performance degredation #40

Open CamdenClark opened 4 years ago

CamdenClark commented 4 years ago

When using this library in a project, an upstream library was using apache.http.*, which has significant trace logging. It looks like on every log message this library calls .getStackTrace, which causes a really significant spike in CPU usage, even if our log level isn't at trace.

rufoa commented 4 years ago

Thanks, will look into this

rufoa commented 4 years ago

Does putting apache.http.* in timbre's :ns-blacklist resolve the issue?

I realise this isn't a proper solution but the blacklist should be checked before the call to getStackTrace occurs.

CamdenClark commented 4 years ago

Yes, that was a successful temporary fix for us.

CamdenClark commented 4 years ago

We were scared that there was unknown impact--any time any new upstream dependency is added we're nervous about the potential for new spats of trace logging that causes a performance degredation that's opaque unless you're explicitly looking for it.

rufoa commented 4 years ago

Yes I understand your concern. Thanks for confirming the blacklist works.

Please could you now check timbre's :ns-log-level works?

Put :ns-log-level [["apache.http.*" :error]] in your config, and check the calls to getStackTrace are skipped

ieugen commented 1 year ago

Is this still an issue? Should we check if we have "apache.http.*" logging in our setup? Is there a better/more reliable way to check without "watching the CPU usage" ?

greglook commented 1 year ago

We (Amperity) eventually wound up writing a new logging library that combines some of the concepts in timbre with a direct SLF4J implementation, so it doesn't suffer from some of the handoff pain this ticket is an example of.

ieugen commented 1 year ago

@greglook : thanks for the suggestion. Will check it out in depth.

We have a big app that uses libraries with all sorts of logging. As of right now, we can't filter logging from other logging frameworks. This is kind of a deal breaker.

ptaoussanis commented 1 month ago

Hi folks - in case this is still relevant for you, you might want to try Timbre v6.6.0-RC1 which now includes built-in SLF4J(v2) support that may be more performant.

Alternatively, if general performance is a concern - I'd consider Telemere which also has built-in SLF4J interop but can be dramatically faster (~350 nanosecs / call with full filtering). It includes a shim for easily migrating from Timbre.