Open wkshare opened 1 year ago
this is a trend graph
Loki version: 2.8.0
Hi there,
We are also seeing similar issues when trying to gather Loki ingest volume metrics.
We are currently in a transition period from cutting logs completely over from another Log collector (we've been using this for years and has never had any complains of missing logs) to Loki, and we are now sending logs to both our existing log collector and Loki.
We have a good idea of the total log volume ingested in our old collector but when trying to compare with Loki's ingest volume from loki_distributor_bytes_received_total
, the metrics seems inconsistent.
For example:
Our log ingest volume metric from our existing log collector stays consistent of 7/31 - 8/6 6.50 TB 8/7 - 8/16 7.28 TB 8/17 - 8/26 7.07 TB
Where as the numbers I gathered from loki_distributor_bytes_received_total
varies quite a lot.
I've tried a few various of aggregation methods
For 7/31 - 8/6
running_sum(loki_distributor_bytes_received_total{job="loki-prod"}[_interval])
552968862335 bytes = 0.553 TB
sum(max_over_time(loki_distributor_bytes_received_total{job="loki-prod"}[$__range]))
895442033191 = 0.895TB
increase(loki_distributor_bytes_received_total{job="loki-prod"}[$__range])
760847954266571 = 760 TB
avg(rate(loki_distributor_bytes_received_total[$__range])) * 7 * 24 * 60 * 60
380423977149437 = 380 TB
Note that we do have pod restarts from time to time and is reflected on the metric itself.
We are really puzzled by this, any pointers on how to get the accurate data is much appreciated!
bump, this is still an issue
Can anyone please provide guidance on how to get the accurate data on Loki ingestion volume?
Describe the bug I have a three node Loki cluster, but there is always one Metric whose stats look inaccurate, as you can see in the screenshot below. The total amount of logs received keeps increasing, while in reality there is not such a large amount of data. With three nodes, only one node has this, the other 2 are normal. How should I check where the data traffic is coming from? wich ip?
To Reproduce loki, version 2.6.1 (branch: HEAD, revision: https://github.com/grafana/loki/commit/6bd05c9a4399805b7951f8a3fb59f4aebf60e6cd) build user: root@af90ed01061f build date: 2022-07-18T08:41:09Z go version: go1.17.6 platform: linux/amd64
Query: rate(loki_distributor_bytes_received_total[3m])
Expected behavior Unable to correctly determine Loki's data traffic
Environment: Oracle Linux 7
Screenshots, Promtail config, or terminal output my configs: