search

VictoriaMetrics / operator

Kubernetes operator for Victoria Metrics
Apache License 2.0
410 stars 141 forks source link

Reconciler error vmagent, vmalertmanager #903

Closed gasgurda closed 4 months ago

gasgurda commented 4 months ago

Hello. Helm chart version: victoria-metrics-k8s-stack - 0.19.4 victoria-metrics-operator - 0.29.0

images: vmagent version - v1.99.0 vmalertmanager version - v0.25.0 operator version - 0.42.0, 0.42.2

The operator receives endless errors in the logs::

{"level":"error","ts":"2024-03-12T16:15:32Z","logger":"manager","msg":"Reconciler error","controller":"vmagent","controllerGroup":"operator.victoriametrics.com","controllerKind":"VMAgent","VMAgent":{"name":"victoria-metrics-k8s-stack","namespace":"monitoring"},"namespace":"monitoring","name":"victoria-metrics-k8s-stack","reconcileID":"0724086f-4a99-4da9-b013-ee8b51aa7d0e","error":"failed to update object status: failed to update object status to=\"expanding\": VMAgent.operator.victoriametrics.com \"victoria-metrics-k8s-stack\" is invalid: [status.availableReplicas: Required value, status.selector: Required value, status.shards: Required value, status.unavailableReplicas: Required value, status.updatedReplicas: Required value]","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).proce
ssNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/
controller.go:227"}

{"level":"error","ts":"2024-03-12T16:16:17Z","logger":"manager","msg":"Reconciler error","controller":"vmalertmanager","controllerGroup":"operator.victoriametrics.com","controllerKind":"VMAlertmanager","VMAlertmanager":{"name":"victoria-metrics-k8s-stack","namespace":"monitoring"},"namespace":"monitoring","name":"victoria-metrics-k8s-stack","reconcileID":"dfe5ba4f-76f1-4c61-9abf-92dfd653bd6b","error":"failed to update object status: failed to update object status to=\"expanding\": vmalertmanagers.operator.victoriametrics.com \"victoria-metrics-k8s-stack\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Control
ler).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227"}

Also operator can't control vmagent and vmalertmanager

f41gh7 commented 4 months ago

Hello, current workaround - perform a manual CRDs update with command:

kubectl apply -f https://raw.githubusercontent.com/VictoriaMetrics/helm-charts/master/charts/victoria-metrics-operator/crd.yaml

Unfortunately helm based k8s-stack installation doesn't perform any CRD updates due to architecture of helm.

And for now, k8s-stack wasn't updated with a new version of CRDs, we're going to fix it soon.

gasgurda commented 4 months ago

Thank you, it solved my problem. This new CRD different that in helm chart CRD. Please add new CRD to helm chart https://github.com/VictoriaMetrics/helm-charts/tree/master/charts/victoria-metrics-k8s-stack/charts/crds/crds