Open nlowe-sx opened 1 year ago
dimitarvdimitrov
commented
1 year ago Thank you for reporting this.
Can you send a screenshot of the raw series' samples from this query? Normally in a range query promQL groups data points into steps and you might lose some resolution. You can get the raw samples by running this instant query in grafana
node_network_transmit_bytes_total{device="..."}[$__range]
nlowe-sx
commented
1 year ago Huh. That looks weird to me. There seems to be two samples for each timestamp and if mimir happens to treat them as sequential samples and the second value for the timestamp happens to be lower, then I could see how that could cause it to think the counter was reset. I've omitted most of the labels here to make the table easier to read but this is indeed all one series.
node_network_transmit_bytes_total{device="...",node="..."}[4m] Here's another interesting one that happened earlier today so I can compare against prometheus. This one seems to have a single data point with two values for the same time stamp:
cortex_request_duration_seconds_count{cluster="...",container="ingester",route="/cortex.Ingester/Push",status_code="success",pod="ingester-zone-f-14"}[4m] And here's what prometheus sees:
cortex_request_duration_seconds_count{container="ingester",route="/cortex.Ingester/Push",status_code="success",pod="ingester-zone-f-14"}[4m] Does this indicate a problem with the HA Tracker?
nlowe-sx
commented
1 year ago We tried turning off OOO sample ingestion and are still seeing these weird samples show up every so often. At this point we're unable to trust mimir to handle our data and cannot complete our migration. Is there any additional data I can provide that would make it easier to troubleshoot this?
nlowe-sx
commented
1 year ago We recently changed our HA Tracker store from etcd to consul and haven't experienced this bug in 9+ days. We will keep monitoring but for now this seems to be related to etcd perhaps? Looking at the cortex_request_duration_seconds_count series that seems to reproduce this bug the most, we haven't seen any spikes since cutting over to consul (the blue annotation in the below screenshot). Our nominal rate for this series is around 120k / sec.
jmichalek132
commented
1 year ago Hi, we are affected by this too, we are using the opentelemetry-collector to send metrics into mimir.
We see the exact same issue happening, particularly on a metric coming from otel with high value k8s_pod_network_io. We are also running 2.8.0 version of mimir. However we are running otel as a daemonset so we don't use the HA tracker feature for deduplication because we send the metric only once. I can provide the same level of details if requested. I will also upgrade to 2.9.0 soon to see if that helps.
jmichalek132
commented
1 year ago Finally, I found time to upgraded mimir to 2.9.0 can confirm it's still happening.
We can see a reset is detected.
The raw query:
Data from the query:
"data": {
"values": [
[
1691403780000,
1691403840000,
1691403900000,
1691403960000,
1691404020000,
1691404080000,
1691404140000,
1691404200000,
1691404260000,
1691404320000,
1691404380000,
1691404440000,
1691404500000,
1691404560000
],
[
616327985923,
616410676205,
616479558915,
616562515237,
616644059848,
616697027902,
616768046955,
616850655521,
616913982572,
616995731837,
617071140070,
617126787917,
617222044703,
617303448582
]
]
}
a517363523
commented
1 year ago
I also encountered such a problem
bboreham
commented
9 months ago Does this indicate a problem with the HA Tracker?
HA-Tracker picks one of two sources of data for the same series, intended for situations where one of the sources stops sending.
If, at the point of fail-over, the other source is sending a lower value for the same counter, this will look like a reset with a big jump. I.e. it matches the reported symptom.
The expectation is that both "HA" sources are scraping the same data source, so they should send the same values. I can't immediately think of a straightforward way in which this can go wrong. Also we have not had many reports of this symptom.
We recently changed our HA Tracker store from etcd to consul and haven't experienced this bug in 9+ days.
Again, I do not know any reason why that would be the case.
At this time I can only suggest inspecting the various components for overload. Anything else you can add may be useful.
ChrisSteinbach
commented
9 months ago We are seeing exactly the same thing: a false reset that gives spurious results for rate queries. Oddly this somehow fixes itself if we add resolution to the query so that instead of:
rate(container_cpu_usage_seconds_total{container!=""}[10m])
we give something like,
rate(container_cpu_usage_seconds_total{container!=""}[10m:3m])
Here's the reset i.e. resets(container_cpu_usage_seconds_total{}[10m]):
And the actual counter showing no resets for the same period:
We are running Mimir 2.9.0
dimitarvdimitrov
commented
9 months ago And the actual counter showing no resets for the same period:
How did you run this query? You can try running this as an instant query and then inspecting the samples around 15:45
container_cpu_usage_seconds_total{}[$__range]@dimitarvdimitrov We ran the query like this, except with one pod selected:
container_cpu_usage_seconds_total{}
If I run the query you suggest, it looks like this:
So there does indeed appear to be a small "reset", but the same artefact is not present in the original data scraped from a Prometheus instance when I zoom in to the same spot using the Prometheus GUI.
dimitarvdimitrov
commented
9 months ago @ChrisSteinbach does this mean that there is only one Prometheus remote-writing to Mimir? So you're not using the distributor HA tracker? (If the answer is "no," then can you check the exact sample value and timestamps on both HA prometheis at ~15:45:30)
ChrisSteinbach
commented
9 months ago @dimitarvdimitrov we're scraping from Prometheus using a grafana agent, but you may be on to something. I'm not the administrator of the Prometheus we scrape, but I can see that there are two instances at work which suggests an HA setup. Unfortunately I can't address them individually. It's something for us to look into anyhow.
Describe the bug
While attempting to migrate from Cortex 1.10.0 to Mimir 2.8.0 we observed some strange behavior with
rate()andirate()queries. Sometimes, for very large counters,rate()/irate()seem to insert ghost / synthetic counter resets, jumping the reported rate by 10s of orders of magnitude for a few data points. Comparing the query against the prometheus instance that remote wrote the data we do not see this behavior so we do not believe it is a problem on the ingest side. Additionally, Cortex 1.10.0, which our clusters are configured to still remote_write to, does not exhibit this problem.Initially, we saw this behavior in the p99
cortex_request_duration_seconds_buckethistogram for distributors and ingesters where the p99 would jump to several minutes from a nominal level of 25ms and thought it might be a data reporting problem with mimir itself. However, we recently discovered and extended period of time (several days) where this happened for thenode_network_transmit_bytes_totalmetric for a few interfaces on one of our nodes.In Grafana, the problem looks like this:
If we query the
irate(), we can see a very large spike from a value that is nominally single-digit Gbit/sec to 100s of millions of Gbit/sec:Looking at the raw counter, there are no counter resets and this jump does not match the slope:
If we take the slope directly with
derivinstead it looks normal:Here's another example we can compare with data in prometheus. Again, the counter on request rate spikes many orders of magnitude:
Prometheus disagrees:
If we ask mimir about counter resets, it thinks it sees one:
Again, prometheus disagrees:
The only commonality I can find between these two series is they are large (values in excess of 1 Billion).
We have observed this problem in both compacted data in historical queries and recent data served from ingesters.
We have tried disabling query sharding (even though these queries should not be shard-able) and did not notice any change.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
The rate roughly matches the slope of the counter
Environment
Additional Context
We're trying to reproduce this synthetically to make it easier to troubleshoot. Please let me know if there's any additional data that would be useful to provide and I can try to work through getting that cleared to share.