search

openebs / monitoring

OpenEBS Monitoring add-on. A set of Grafana, Prometheus, and alert manager plugins.
Apache License 2.0
26 stars 20 forks source link

`StalePersistentVolumeClaim` duplicate time series in Victoria Metrics #105

Open dhess opened 7 months ago

dhess commented 7 months ago

Hi,

The StalePersistentVolumeClaim query here breaks Victoria Metrics in some configurations:

https://github.com/openebs/monitoring/blob/161b5af7f7525c223b9165a13db2d6b667d08aad/deploy/charts/rules/volume/volume-rules.json#L12

422: error when executing query="kube_persistentvolumeclaim_info unless (kube_persistentvolumeclaim_info * on(persistentvolumeclaim) group_right kube_pod_spec_volumes_persistentvolumeclaims_info) == 1" on the time range (start=1700931915000, end=1700932215000, step=15000): cannot execute "kube_persistentvolumeclaim_info unless ((kube_persistentvolumeclaim_info * on(persistentvolumeclaim) group_right() kube_pod_spec_volumes_persistentvolumeclaims_info) == 1)": cannot execute "(kube_persistentvolumeclaim_info{persistentvolumeclaim=~\"audit-vault-0|audit-vault-1|audit-vault-2|... duplicate time series on the left side of `* on(persistentvolumeclaim) group_right()`: ...

In our particular case, this happens on any ReadWriteMany PVC that occurs in more than one pod in the same namespace.