search

jmcgrath207 / k8s-ephemeral-storage-metrics

Prometheus ephemeral storage metrics exporter
https://jmcgrath207.github.io/k8s-ephemeral-storage-metrics/
MIT License
75 stars 29 forks source link

Prometheus log errors "observedGeneration" field not declared in schema #100

Closed aitimashnov closed 5 days ago

aitimashnov commented 3 weeks ago

Hello!

Description

I started seeing the errors in the prometheus-operator pod when I began to use k8s-ephemeral-storage-metrics exporter. Logs:

level=info ts=2024-07-03T10:43:56.939201168Z caller=operator.go:1185 component=prometheus-controller msg="failed to apply prometheus status subresource, trying again without scale fields" err="failed to create typed patch object (monitoring/kube-prometheus-stack-prometheus; monitoring.coreos.com/v1, Kind=Prometheus): errors:\n  .status.conditions[0].observedGeneration: field not declared in schema\n  .status.conditions[1].observedGeneration: field not declared in schema\n  .status.shards: field not declared in schema"
level=error ts=2024-07-03T10:43:56.952912424Z caller=klog.go:126 component=k8s_client_runtime func=ErrorDepth msg="status \"monitoring/kube-prometheus-stack-prometheus\" failed: failed to apply prometheus status subresource: failed to create typed patch object (monitoring/kube-prometheus-stack-prometheus; monitoring.coreos.com/v1, Kind=Prometheus): errors:\n  .status.conditions[0].observedGeneration: field not declared in schema\n  .status.conditions[1].observedGeneration: field not declared in schema"
level=info ts=2024-07-03T10:43:57.18298472Z caller=operator.go:1185 component=prometheus-controller msg="failed to apply prometheus status subresource, trying again without scale fields" err="failed to create typed patch object (monitoring/kube-prometheus-stack-prometheus; monitoring.coreos.com/v1, Kind=Prometheus): .status.selector: field not declared in schema"

To Reproduce

ephemeral-storage-exporter helm release values:

deploy_type: DaemonSet
serviceMonitor:
    additionalLabels:
        release: kube-prometheus-stack

Help to solve the issue, please.

jmcgrath207 commented 2 weeks ago

Hi @aitimashnov, Thank you for raising this issue and going into detail.

I attempted to recreate your environment in this PR. I ran make deploy_observability to do this.

https://github.com/jmcgrath207/k8s-ephemeral-storage-metrics/pull/101/files

Here is my test chart tests/charts/observability

The CI test did pass. https://github.com/jmcgrath207/k8s-ephemeral-storage-metrics/actions/runs/9778835511/job/26997543635

Could you confirm you are running this CRD in your cluster? Also do you get the same error when deploying 1.10.1 of this chart?

kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.71.0/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml

https://artifacthub.io/packages/helm/prometheus-community/kube-prometheus-stack/56.13.1#from-55-x-to-56-x

Thanks!

aitimashnov commented 5 days ago

Hi! Thank you for the help. I've restored all v0.71.0 CRDs for kube-prometheus-stack and the issue has been resolved.