Describe the bug a clear and concise description of what the bug is.
After upgrade kube-prometheus-stack from 56.21.4 to 57.2.1we alerts about PrometheusDuplicateTimestamps start firing.
What's your helm version?
v3.15.2
What's your kubectl version?
1.30.1
Which chart?
kube-prometheus-stack
What's the chart version?
57.2.1
What happened?
After upgrading from kube-prometheus-stack chart from 56.21.4 to 57.2.1 we start receiving alerts about PrometheusDuplicateTimestamps.
I the logs of prometheus pods i see:
stern prometheus-kube-prometheus-stack-prometheus-0 --tail 10 -i "scrape manager"
+ prometheus-kube-prometheus-stack-prometheus-0 › prometheus
+ prometheus-kube-prometheus-stack-prometheus-0 › thanos-sidecar
+ prometheus-kube-prometheus-stack-prometheus-0 › config-reloader
prometheus-kube-prometheus-stack-prometheus-0 prometheus ts=2024-06-25T08:08:56.277Z caller=scrape.go:1738 level=warn component="scrape manager" scrape_pool=serviceMonitor/kube-system/kube-prometheus-stack-kube-state-metrics/0 target=http://100.81.77.2:8080/metrics msg="Error on ingesting samples with different value but same timestamp" num_dropped=76
prometheus-kube-prometheus-stack-prometheus-0 prometheus ts=2024-06-25T08:09:26.289Z caller=scrape.go:1738 level=warn component="scrape manager" scrape_pool=serviceMonitor/kube-system/kube-prometheus-stack-kube-state-metrics/0 target=http://100.81.77.2:8080/metrics msg="Error on ingesting samples with different value but same timestamp" num_dropped=76
prometheus-kube-prometheus-stack-prometheus-0 prometheus ts=2024-06-25T08:09:56.250Z caller=scrape.go:1738 level=warn component="scrape manager" scrape_pool=serviceMonitor/kube-system/kube-prometheus-stack-kube-state-metrics/0 target=http://100.81.77.2:8080/metrics msg="Error on ingesting samples with different value but same timestamp" num_dropped=76
prometheus-kube-prometheus-stack-prometheus-0 prometheus ts=2024-06-25T08:10:26.256Z caller=scrape.go:1738 level=warn component="scrape manager" scrape_pool=serviceMonitor/kube-system/kube-prometheus-stack-kube-state-metrics/0 target=http://100.81.77.2:8080/metrics msg="Error on ingesting samples with different value but same timestamp" num_dropped=76
prometheus-kube-prometheus-stack-prometheus-0 prometheus ts=2024-06-25T08:10:56.313Z caller=scrape.go:1738 level=warn component="scrape manager" scrape_pool=serviceMonitor/kube-system/kube-prometheus-stack-kube-state-metrics/0 target=http://100.81.77.2:8080/metrics msg="Error on ingesting samples with different value but same timestamp" num_dropped=76
prometheus-kube-prometheus-stack-prometheus-0 prometheus ts=2024-06-25T08:11:26.255Z caller=scrape.go:1738 level=warn component="scrape manager" scrape_pool=serviceMonitor/kube-system/kube-prometheus-stack-kube-state-metrics/0 target=http://100.81.77.2:8080/metrics msg="Error on ingesting samples with different value but same timestamp" num_dropped=76
prometheus-kube-prometheus-stack-prometheus-0 prometheus ts=2024-06-25T08:11:56.339Z caller=scrape.go:1738 level=warn component="scrape manager" scrape_pool=serviceMonitor/kube-system/kube-prometheus-stack-kube-state-metrics/0 target=http://100.81.77.2:8080/metrics msg="Error on ingesting samples with different value but same timestamp" num_dropped=76
prometheus-kube-prometheus-stack-prometheus-0 prometheus ts=2024-06-25T08:12:26.204Z caller=scrape.go:1738 level=warn component="scrape manager" scrape_pool=serviceMonitor/kube-system/kube-prometheus-stack-kube-state-metrics/0 target=http://100.81.77.2:8080/metrics msg="Error on ingesting samples with different value but same timestamp" num_dropped=76
prometheus-kube-prometheus-stack-prometheus-0 prometheus ts=2024-06-25T08:12:56.232Z caller=scrape.go:1738 level=warn component="scrape manager" scrape_pool=serviceMonitor/kube-system/kube-prometheus-stack-kube-state-metrics/0 target=http://100.81.77.2:8080/metrics msg="Error on ingesting samples with different value but same timestamp" num_dropped=76
prometheus-kube-prometheus-stack-prometheus-0 prometheus ts=2024-06-25T08:13:26.334Z caller=scrape.go:1738 level=warn component="scrape manager" scrape_pool=serviceMonitor/kube-system/kube-prometheus-stack-kube-state-metrics/0 target=http://100.81.77.2:8080/metrics msg="Error on ingesting samples with different value but same timestamp" num_dropped=76
What you expected to happen?
No receive PrometheusDuplicateTimestamps alerts
How to reproduce it?
Install kube-prometheus-stack with the provided values.yaml
Describe the bug a clear and concise description of what the bug is.
After upgrade![image](https://github.com/prometheus-community/helm-charts/assets/24940221/dcfbb39e-95a8-43dc-9677-abc75159f527)
kube-prometheus-stack
from56.21.4
to57.2.1
we alerts aboutPrometheusDuplicateTimestamps
start firing.What's your helm version?
v3.15.2
What's your kubectl version?
1.30.1
Which chart?
kube-prometheus-stack
What's the chart version?
57.2.1
What happened?
After upgrading from kube-prometheus-stack chart from
56.21.4
to57.2.1
we start receiving alerts aboutPrometheusDuplicateTimestamps
. I the logs of prometheus pods i see:What you expected to happen?
No receive
PrometheusDuplicateTimestamps
alertsHow to reproduce it?
Install
kube-prometheus-stack
with the providedvalues.yaml
Enter the changed values of values.yaml?
Enter the command that you execute and failing/misfunctioning.
we use terraform:
Anything else we need to know?
no