search

loki4j / loki-logback-appender

Fast and lightweight implementation of Logback appender for Grafana Loki
https://loki4j.github.io/loki-logback-appender/
BSD 2-Clause "Simplified" License
300 stars 26 forks source link

nanoCounter leads 'out of order' errors on overflow #90

Closed nehaev closed 3 years ago

nehaev commented 3 years ago

Logback provides log event timestamp with millisecond precision. In order to preserve the message order as in other Logback outputs (such as CONSOLE or FILE), Loki4j used nanoCounter to emulate this order for Loki (otherwise events with the same millisecond are randomly mixed). nanoCounter was implemented as AtomicInteger with the following update procedure:

nanoCounter.updateAndGet(i -> i < 999_999 ? i + 1 : 0);

So nanos was a simple global count of the event that is being reset each 1M events. This reset could cause 'out of order' for all subsequent events with the same millisecond.

nehaev commented 3 years ago

In #87 nanoCounter was removed completely with nanos part of the timestamp. However, this is not a solution for a problem. nanoCounter needs to be re-implemented.