Describe the bug a clear and concise description of what the bug is.
Prometheus logs warnings about conflicting samples with the same timestamp, and some metric samples are dropped, leading to incomplete data.
Error on Ingesting Samples with Different Values but Same Timestamp
Client Version: v1.29.3 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.27.12 WARNING: version difference between client (1.29) and server (1.27) exceeds the supported minor version skew of +/-1
Which chart?
kube-prometheus-stack
What's the chart version?
v65.0.0
What happened?
After upgrading from v51.0.0 to v65.0.0 ( prometheus upgraded fro 2.47 to 2.54 ) . We are encountering an issue where Prometheus is logging warnings about dropping samples due to conflicting values with the same timestamp .
ts=2024-10-14T07:23:44.552Z caller=scrape.go:1744 level=warn component="scrape manager" scrape_pool=consul-appli target=http://x.x.x.x:8081/metrics msg="Error on ingesting samples with different value but same timestamp" num_dropped=864ts=2024-10-14T07:15:27.184Z caller=scrape.go:1744 level=warn component="scrape manager" scrape_pool=serviceMonitor/monitoring/monitoring-k8s-kube-state-metrics/0 target=http://x.x.x.x::8080/metrics msg="Error on ingesting samples with different value but same timestamp" num_dropped=25Following the bug fix14685 , the issue should normally be resolved in version 2.54.1. However, the warnings logs are still appearing
What you expected to happen?
Prometheus should be able to scrape metrics without encountering conflicting values for the same timestamp, or the application/service should emit metrics with unique timestamps.
How to reproduce it?
No response
Enter the changed values of values.yaml?
No particular changes .
CRDs are updated
Enter the command that you execute and failing/misfunctioning.
No particular changes ,
Anything else we need to know?
I downgraded from v65.0.0 to v51.0.0 ( prom-v2.47) and there is no scrapping warnings logs .
Describe the bug a clear and concise description of what the bug is.
Prometheus logs warnings about conflicting samples with the same timestamp, and some metric samples are dropped, leading to incomplete data. Error on Ingesting Samples with Different Values but Same Timestamp
What's your helm version?
version.BuildInfo{Version:"v3.14.3", GitCommit:"f03cc04caaa8f6d7c3e67cf918929150cf6f3f12", GitTreeState:"clean", GoVersion:"go1.21.7"}
What's your kubectl version?
Client Version: v1.29.3 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.27.12 WARNING: version difference between client (1.29) and server (1.27) exceeds the supported minor version skew of +/-1
Which chart?
kube-prometheus-stack
What's the chart version?
v65.0.0
What happened?
After upgrading from v51.0.0 to v65.0.0 ( prometheus upgraded fro 2.47 to 2.54 ) . We are encountering an issue where Prometheus is logging warnings about dropping samples due to conflicting values with the same timestamp .
ts=2024-10-14T07:23:44.552Z caller=scrape.go:1744 level=warn component="scrape manager" scrape_pool=consul-appli target=http://x.x.x.x:8081/metrics msg="Error on ingesting samples with different value but same timestamp" num_dropped=864
ts=2024-10-14T07:15:27.184Z caller=scrape.go:1744 level=warn component="scrape manager" scrape_pool=serviceMonitor/monitoring/monitoring-k8s-kube-state-metrics/0 target=http://x.x.x.x::8080/metrics msg="Error on ingesting samples with different value but same timestamp" num_dropped=25
Following the bug fix 14685 , the issue should normally be resolved in version 2.54.1. However, the warnings logs are still appearingWhat you expected to happen?
Prometheus should be able to scrape metrics without encountering conflicting values for the same timestamp, or the application/service should emit metrics with unique timestamps.
How to reproduce it?
No response
Enter the changed values of values.yaml?
No particular changes . CRDs are updated
Enter the command that you execute and failing/misfunctioning.
No particular changes ,
Anything else we need to know?
I downgraded from v65.0.0 to v51.0.0 ( prom-v2.47) and there is no scrapping warnings logs .