search

grafana / loki

Like Prometheus, but for logs.
https://grafana.com/loki
GNU Affero General Public License v3.0
23.85k stars 3.44k forks source link

Add deduplicating logger client #6077

Open dannykopping opened 2 years ago

dannykopping commented 2 years ago

Is your feature request related to a problem? Please describe. Currently if a log message is logged repeatedly, it will spam the log file and provide little operational value.

Describe the solution you'd like A logger which keeps a small buffer to determine if a log message has been seen within a configurable time-window, and only logs if not.

Prometheus has this functionality already: https://github.com/prometheus/prometheus/blob/main/util/logging/dedupe.go#L45

Describe alternatives you've considered N/A

Additional context The creation of this issue was provoked by a minor incident in our SaaS environment where a debug log, repeated on every call to memcached, produced a large volume of largely useless repeated logs.

alexis-wefight commented 2 years ago

I have the same issue, is there any workaround? I tried to play with the timestamp stage with Promtail, but timestamp is never updated (I would like to change it from nanoseconds to milliseconds)