search

grafana / tempo

Grafana Tempo is a high volume, minimal dependency distributed tracing backend.
https://grafana.com/oss/tempo/
GNU Affero General Public License v3.0
4.02k stars 521 forks source link

"searching for trace in block" log flood #4105

Closed fede843 closed 3 weeks ago

fede843 commented 1 month ago

We are logging Tempo with INFO level. Every time we do a simple query, logs get bombarded by hundreds of instances saying "searching for trace in block" and some other non relevant info. As we keep querying this becomes quickly in a issue, getting thousands of these unwanted records. It would be nice these can be dropped down to debug or something like that? Not sure what is the value on having all these under INFO level.

Here a screen shot for a single and simple query causing almost 1k entries:

Image

joe-elliott commented 1 month ago

Thanks for filing this issue. Agree this would make more sense to be logged at a debug level. This is a great first PR for anyone who wants to take it on.

Aki0x137 commented 1 month ago

hi @joe-elliott. So all that is required to be done here is to change log level for this single line in tempodb?

joe-elliott commented 4 weeks ago

yup, i believe that's it.