There is a gap on the Grafana Logs Volume chart when run a query

grafana / loki

Like Prometheus, but for logs.

https://grafana.com/loki

GNU Affero General Public License v3.0

23.28k stars 3.37k forks source link

There is a gap on the Grafana Logs Volume chart when run a query #8767

Open ercansayici opened 1 year ago

ercansayici commented 1 year ago

When I run a query on Grafana to see some logs from my app, even though the logs are gathered from Loki and can be seen in the logs section, some of the logs are not present on the Logs Volume Chart (there is a gap). and after some time passed this gap disappears. (please see the screenshots below, I took screenshots at 1-hour intervals)

8-15 9-15 10-15 1-15

Notes:

Actually, logs are still in there.
The gap always appears near the query time.
The gap only appears when the number of logs is huge ( > 5000 logs per second).
if the size of the logs is small, the gap does not appear.

Environment:

Grafana version: 9.4.0 Data source type & version: Loki 2.7.1 Distributed deployment (use Memcached) OS Grafana is installed on: Linux User OS & Browser: Windows, Mac

liguozhong commented 1 year ago

me too. Mine is also like this, but I currently think that it is caused by the delay in sending logs from my promtail agent, and it is not a problem with the loki server.

So the data will level out after 5 minutes.

Looking forward to more discussions, it is also important to be able to identify this issue. Currently ourrecording rule will be affected by this, resulting in inaccurate data.

ercansayici commented 1 year ago

hi @liguozhong My Loki is Distributed mode that deployed using bitnami helmchart, and as a agent I deployed Fluent-bit. what is your environment? Regards

liguozhong commented 1 year ago

Distributed mode + promtail / vector / go http sdk / java http sdk .

liguozhong commented 1 year ago

This issue is hard to track down, I noticed it 9 months ago, but I can't figure it out

joemiller commented 1 year ago

I think I'm seeing a very similar issue. Only noticed it over switching out promtail for vector when shipping logs to loki, but it's possible the issue was there before and I just didn't notice it since I wasn't looking as closely.

And like others said it seems to happen to pods/apps generating high (100s/1000s of lines/sec) of logs, but does not impact other apps.