search

VictoriaMetrics / helm-charts

Helm charts for VictoriaMetrics, VictoriaLogs and ecosystem
https://victoriametrics.github.io/helm-charts/
Apache License 2.0
341 stars 331 forks source link

K8S VM Stack: Resources are not deleted after deleting the namespace. #1091

Closed SalvadorOlivar closed 4 months ago

SalvadorOlivar commented 4 months ago

Hello!

Version: victoria-metrics-k8s-stack 0.19.4

What I was trying to do was update the release but I was getting a TLS error for two IPs that do not exist in the cluster. So what I try to do is delete the namespace and start from scratch.

I see that deleting the namespace where I have the k8s stack VM installed does not delete the following resources:

How can I clean up the helm chart in the cluster?

This prevents me from reinstalling the helm chart.

VMAlertmanager yaml.

apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlertmanager
metadata:
  annotations:
    meta.helm.sh/release-name: vm
    meta.helm.sh/release-namespace: monitoring
  creationTimestamp: "2024-05-29T19:53:29Z"
  deletionGracePeriodSeconds: 0
  deletionTimestamp: "2024-06-28T14:19:09Z"
  finalizers:
  - apps.victoriametrics.com/finalizer
  generation: 3
  labels:
    app.kubernetes.io/component: victoria-metrics-k8s-stack-alertmanager
    app.kubernetes.io/instance: vm
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: victoria-metrics-k8s-stack
    app.kubernetes.io/version: v1.99.0
    helm.sh/chart: victoria-metrics-k8s-stack-0.19.4
  name: vm-victoria-metrics-k8s-stack
  namespace: monitoring
spec:
  configSecret: vm-victoria-metrics-k8s-stack-alertmanager
  image:
    tag: v0.26.0
  nodeSelector:
    upm.environment: main
    upm.nodeType: monitoring
  resources: {}
  routePrefix: /
  selectAllByDefault: true
  templates:
  - key: monzo.tmpl
    name: vm-victoria-metrics-k8s-stack-alertmanager-monzo-tpl

VMCluster

apiVersion: operator.victoriametrics.com/v1beta1
kind: VMCluster
metadata:
  annotations:
    meta.helm.sh/release-name: vm
    meta.helm.sh/release-namespace: monitoring
  creationTimestamp: "2024-04-02T19:50:26Z"
  deletionGracePeriodSeconds: 0
  deletionTimestamp: "2024-06-28T14:19:09Z"
  finalizers:
  - apps.victoriametrics.com/finalizer
  generation: 19
  labels:
    app.kubernetes.io/instance: vm
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: victoria-metrics-k8s-stack
    app.kubernetes.io/version: v1.99.0
    helm.sh/chart: victoria-metrics-k8s-stack-0.19.4
  name: vm-victoria-metrics-k8s-stack
  namespace: monitoring
spec:
  replicationFactor: 2
  retentionPeriod: "6"
  vminsert:
    extraArgs:
      maxLabelsPerTimeseries: "50"
    image:
      tag: v1.97.1-cluster
    nodeSelector:
      upm.environment: main
      upm.nodeType: monitoring
    replicaCount: 2
    resources:
      limits:
        cpu: "1"
        memory: 1000Mi
      requests:
        cpu: 500m
        memory: 500Mi
  vmselect:
    cacheMountPath: /select-cache
    extraArgs:
      vmalert.proxyURL: ""
    image:
      tag: v1.97.1-cluster
    nodeSelector:
      upm.environment: main
      upm.nodeType: monitoring
    replicaCount: 2
    resources: {}
    storage:
      volumeClaimTemplate:
        metadata: {}
        spec:
          resources:
            requests:
              storage: 2Gi
          storageClassName: managed-csi-premium
        status: {}
  vmstorage:
    image:
      tag: v1.97.1-cluster
    nodeSelector:
      upm.environment: main
      upm.nodeType: monitoring
    replicaCount: 3
    resources:
      limits:
        cpu: "1"
        memory: 5000Mi
      requests:
        cpu: 250m
        memory: 3000Mi
    storage:
      volumeClaimTemplate:
        metadata: {}
        spec:
          resources:
            requests:
              storage: 100Gi
          storageClassName: managed-csi-premium
        status: {}
    storageDataPath: /vm-data
status:
  clusterStatus: operational
  updateFailCount: 0

VMAlert Example

apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlert
metadata:
  creationTimestamp: "2024-05-31T11:54:08Z"
  deletionGracePeriodSeconds: 0
  deletionTimestamp: "2024-06-28T14:19:10Z"
  finalizers:
  - apps.victoriametrics.com/finalizer
  generation: 2
  name: k3s-vmalert
  namespace: monitoring
spec:
  datasource:
    url: http://vmselect-vm-victoria-metrics-k8s-stack.monitoring.svc:8481/select/10/prometheus
  evaluationInterval: 15s
  extraArgs:
    remoteWrite.disablePathAppend: "true"
  image:
    tag: v1.97.1
  nodeSelector:
    upm.environment: main
    upm.nodeType: monitoring
  notifiers:
  - url: http://vmalertmanager-vm-victoria-metrics-k8s-stack-0.vmalertmanager-vm-victoria-metrics-k8s-stack.monitoring.svc:9093
  remoteRead:
    url: http://vmselect-vm-victoria-metrics-k8s-stack.monitoring.svc:8481/select/10/prometheus
  remoteWrite:
    url: http://vminsert-vm-victoria-metrics-k8s-stack:8480/insert/10/prometheus/
  resources: {}
  selectAllByDefault: true

Operator logs:

{"level":"error","ts":1719600846.9822176,"msg":"Reconciler error","controller":"vmalert","controllerGroup":"operator.victoriametrics.com","controllerKind":"VMAlert","VMAlert":{"name":"k3s-frb","namespace":"monitoring"},"namespace":"monitoring","name":"k3s-frb","reconcileID":"37ddff4c-c43a-4adf-ac5c-9fa67f3628a7","error":"Internal error occurred: failed calling webhook \"vmalert.victoriametrics.com\": failed to call webhook: Post \"https://vm-2-victoria-metrics-operator.monitoring.svc:443/validate-operator-victoriametrics-com-v1beta1-vmalert?timeout=10s\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-06-28T18:54:06Z is after 2024-06-10T15:01:52Z","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234"}
2024/06/28 18:54:07 http: TLS handshake error from 10.233.1.69:35192: EOF
{"level":"error","ts":1719600847.109276,"msg":"Reconciler error","controller":"vmalert","controllerGroup":"operator.victoriametrics.com","controllerKind":"VMAlert","VMAlert":{"name":"k3s-mvd-port","namespace":"monitoring"},"namespace":"monitoring","name":"k3s-mvd-port","reconcileID":"d7cd13b2-e241-4eb7-bab9-2c5be9441ec7","error":"Internal error occurred: failed calling webhook \"vmalert.victoriametrics.com\": failed to call webhook: Post \"https://vm-victoria-metrics-operator.monitoring.svc:443/validate-operator-victoriametrics-com-v1beta1-vmalert?timeout=10s\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-06-28T18:54:07Z is after 2024-06-10T15:01:52Z","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234"}
2024/06/28 18:54:07 http: TLS handshake error from 10.233.7.228:50602: remote error: tls: bad certificate
{"level":"error","ts":1719600847.1978478,"msg":"Reconciler error","controller":"vmalert","controllerGroup":"operator.victoriametrics.com","controllerKind":"VMAlert","VMAlert":{"name":"k3s-pdt","namespace":"monitoring"},"namespace":"monitoring","name":"k3s-pdt","reconcileID":"dc6acbb2-6af1-416e-afa7-1a9393d26aea","error":"Internal error occurred: failed calling webhook \"vmalert.victoriametrics.com\": failed to call webhook: Post \"https://vm-victoria-metrics-operator.monitoring.svc:443/validate-operator-victoriametrics-com-v1beta1-vmalert?timeout=10s\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-06-28T18:54:07Z is after 2024-06-10T15:01:52Z","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234"}
2024/06/28 18:54:07 http: TLS handshake error from 10.233.7.228:50612: EOF
{"level":"error","ts":1719600847.288661,"msg":"Reconciler error","controller":"vmalert","controllerGroup":"operator.victoriametrics.com","controllerKind":"VMAlert","VMAlert":{"name":"k3s-pdu-norte","namespace":"monitoring"},"namespace":"monitoring","name":"k3s-pdu-norte","reconcileID":"e9f968a3-e6d8-4110-9e1a-b5cbe8307ee9","error":"Internal error occurred: failed calling webhook \"vmalert.victoriametrics.com\": failed to call webhook: Post \"https://vm-2-victoria-metrics-operator.monitoring.svc:443/validate-operator-victoriametrics-com-v1beta1-vmalert?timeout=10s\": http: server gave HTTP response to HTTPS client","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234"}
2024/06/28 18:54:07 http: TLS handshake error from 10.233.1.69:35194: EOF
{"level":"error","ts":1719600847.396496,"msg":"Reconciler error","controller":"vmalert","controllerGroup":"operator.victoriametrics.com","controllerKind":"VMAlert","VMAlert":{"name":"k3s-tacuarembo","namespace":"monitoring"},"namespace":"monitoring","name":"k3s-tacuarembo","reconcileID":"6023414e-6785-470c-8105-5e060d0579c1","error":"Internal error occurred: failed calling webhook \"vmalert.victoriametrics.com\": failed to call webhook: Post \"https://vm-victoria-metrics-operator.monitoring.svc:443/validate-operator-victoriametrics-com-v1beta1-vmalert?timeout=10s\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-06-28T18:54:07Z is after 2024-06-10T15:01:52Z","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234"}
2024/06/28 18:54:07 http: TLS handshake error from 10.233.7.228:50626: EOF
{"level":"error","ts":1719600847.4774985,"msg":"Reconciler error","controller":"vmalert","controllerGroup":"operator.victoriametrics.com","controllerKind":"VMAlert","VMAlert":{"name":"k3s-capricho","namespace":"monitoring"},"namespace":"monitoring","name":"k3s-capricho","reconcileID":"f98ec7d8-8bce-4950-80bd-93924d8a1f17","error":"Internal error occurred: failed calling webhook \"vmalert.victoriametrics.com\": failed to call webhook: Post \"https://vm-2-victoria-metrics-operator.monitoring.svc:443/validate-operator-victoriametrics-com-v1beta1-vmalert?timeout=10s\": http: server gave HTTP response to HTTPS client","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234"}
2024/06/28 18:54:07 http: TLS handshake error from 10.233.7.228:50634: EOF
{"level":"error","ts":1719600847.5586135,"msg":"Reconciler error","controller":"vmalert","controllerGroup":"operator.victoriametrics.com","controllerKind":"VMAlert","VMAlert":{"name":"k3s-cerrezuelo","namespace":"monitoring"},"namespace":"monitoring","name":"k3s-cerrezuelo","reconcileID":"542c96ca-c114-416a-be12-5913eafc2532","error":"Internal error occurred: failed calling webhook \"vmalert.victoriametrics.com\": failed to call webhook: Post \"https://vm-victoria-metrics-operator.monitoring.svc:443/validate-operator-victoriametrics-com-v1beta1-vmalert?timeout=10s\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-06-28T18:54:07Z is after 2024-06-10T15:01:52Z","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234"}
2024/06/28 18:54:07 http: TLS handshake error from 10.233.7.228:50644: remote error: tls: bad certificate
{"level":"error","ts":1719600847.6348548,"msg":"Reconciler error","controller":"vmalert","controllerGroup":"operator.victoriametrics.com","controllerKind":"VMAlert","VMAlert":{"name":"k3s-el-criollo","namespace":"monitoring"},"namespace":"monitoring","name":"k3s-el-criollo","reconcileID":"03ba2993-b2b0-44aa-8d39-853d3302e6b9","error":"Internal error occurred: failed calling webhook \"vmalert.victoriametrics.com\": failed to call webhook: Post \"https://vm-victoria-metrics-operator.monitoring.svc:443/validate-operator-victoriametrics-com-v1beta1-vmalert?timeout=10s\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-06-28T18:54:07Z is after 2024-06-10T15:01:52Z","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234"}
2024/06/28 18:54:07 http: TLS handshake error from 10.233.1.69:35210: remote error: tls: bad certificate
{"level":"error","ts":1719600847.773791,"msg":"Reconciler error","controller":"vmalert","controllerGroup":"operator.victoriametrics.com","controllerKind":"VMAlert","VMAlert":{"name":"k3s-sarandi","namespace":"monitoring"},"namespace":"monitoring","name":"k3s-sarandi","reconcileID":"aff367c2-31fa-41e7-bbc7-a08551838e99","error":"Internal error occurred: failed calling webhook \"vmalert.victoriametrics.com\": failed to call webhook: Post \"https://vm-victoria-metrics-operator.monitoring.svc:443/validate-operator-victoriametrics-com-v1beta1-vmalert?timeout=10s\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-06-28T18:54:07Z is after 2024-06-10T15:01:52Z","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234"}
2024/06/28 18:54:07 http: TLS handshake error from 10.233.7.228:60524: EOF
{"level":"error","ts":1719600847.8796675,"msg":"Reconciler error","controller":"vmalert","controllerGroup":"operator.victoriametrics.com","controllerKind":"VMAlert","VMAlert":{"name":"application-01-aks-main","namespace":"monitoring"},"namespace":"monitoring","name":"application-01-aks-main","reconcileID":"c95189b2-30e1-444e-b074-6147d42af8c7","error":"Internal error occurred: failed calling webhook \"vmalert.victoriametrics.com\": failed to call webhook: Post \"https://vm-victoria-metrics-operator.monitoring.svc:443/validate-operator-victoriametrics-com-v1beta1-vmalert?timeout=10s\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-06-28T18:54:07Z is after 2024-06-10T15:01:52Z","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234"}
2024/06/28 18:54:07 http: TLS handshake error from 10.233.1.69:35214: EOF
{"level":"error","ts":1719600847.9610658,"msg":"Reconciler error","controller":"vmalert","controllerGroup":"operator.victoriametrics.com","controllerKind":"VMAlert","VMAlert":{"name":"application-01-aks-stream","namespace":"monitoring"},"namespace":"monitoring","name":"application-01-aks-stream","reconcileID":"820949e5-3e66-49bc-9ee8-2788fd35f167","error":"Internal error occurred: failed calling webhook \"vmalert.victoriametrics.com\": failed to call webhook: Post \"https://vm-2-victoria-metrics-operator.monitoring.svc:443/validate-operator-victoriametrics-com-v1beta1-vmalert?timeout=10s\": http: server gave HTTP response to HTTPS client","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234"}
2024/06/28 18:54:42 http: TLS handshake error from 10.233.1.69:45346: EOF
{"level":"error","ts":1719600882.6870382,"msg":"Reconciler error","controller":"vmalertmanager","controllerGroup":"operator.victoriametrics.com","controllerKind":"VMAlertmanager","VMAlertmanager":{"name":"vm-victoria-metrics-k8s-stack","namespace":"monitoring"},"namespace":"monitoring","name":"vm-victoria-metrics-k8s-stack","reconcileID":"92d439bb-2b45-4746-bac9-0d00f380952d","error":"Internal error occurred: failed calling webhook \"vmalertmanager.victoriametrics.com\": failed to call webhook: Post \"https://vm-2-victoria-metrics-operator.monitoring.svc:443/validate-operator-victoriametrics-com-v1beta1-vmalertmanager?timeout=10s\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-06-28T18:54:42Z is after 2024-06-10T15:01:52Z","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234"}
{"level":"info","ts":1719600924.8038883,"logger":"controller","msg":"Reconciling VMCluster","Request.Namespace":"monitoring","Request.Name":"vm-victoria-metrics-k8s-stack"}
2024/06/28 18:55:24 http: TLS handshake error from 10.233.7.228:40150: EOF
{"level":"error","ts":1719600924.8840938,"msg":"Reconciler error","controller":"vmcluster","controllerGroup":"operator.victoriametrics.com","controllerKind":"VMCluster","VMCluster":{"name":"vm-victoria-metrics-k8s-stack","namespace":"monitoring"},"namespace":"monitoring","name":"vm-victoria-metrics-k8s-stack","reconcileID":"c2bc98e8-699a-490a-b605-08c4f66ad34b","error":"Internal error occurred: failed calling webhook \"vmcluster.victoriametrics.com\": failed to call webhook: Post \"https://vm-victoria-metrics-operator.monitoring.svc:443/validate-operator-victoriametrics-com-v1beta1-vmcluster?timeout=10s\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-06-28T18:55:24Z is after 2024-06-10T15:01:52Z","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234"}
2024/06/28 18:56:07 http: TLS handshake error from 10.233.1.69:59134: EOF
{"level":"error","ts":1719600967.0798962,"msg":"Reconciler error","controller":"vmalert","controllerGroup":"operator.victoriametrics.com","controllerKind":"VMAlert","VMAlert":{"name":"k3s-frb","namespace":"monitoring"},"namespace":"monitoring","name":"k3s-frb","reconcileID":"5260d84c-4719-4ea5-b53e-3a34cfd5e9f4","error":"Internal error occurred: failed calling webhook \"vmalert.victoriametrics.com\": failed to call webhook: Post \"https://vm-2-victoria-metrics-operator.monitoring.svc:443/validate-operator-victoriametrics-com-v1beta1-vmalert?timeout=10s\": http: server gave HTTP response to HTTPS client","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234"}
2024/06/28 18:56:07 http: TLS handshake error from 10.233.1.69:59140: EOF
{"level":"error","ts":1719600967.1918516,"msg":"Reconciler error","controller":"vmalert","controllerGroup":"operator.victoriametrics.com","controllerKind":"VMAlert","VMAlert":{"name":"k3s-mvd-port","namespace":"monitoring"},"namespace":"monitoring","name":"k3s-mvd-port","reconcileID":"235e164c-2793-4f57-8598-b9c2a5d1dde0","error":"Internal error occurred: failed calling webhook \"vmalert.victoriametrics.com\": failed to call webhook: Post \"https://vm-2-victoria-metrics-operator.monitoring.svc:443/validate-operator-victoriametrics-com-v1beta1-vmalert?timeout=10s\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-06-28T18:56:07Z is after 2024-06-10T15:01:52Z","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234"}
2024/06/28 18:56:07 http: TLS handshake error from 10.233.7.228:49162: EOF
{"level":"error","ts":1719600967.2811003,"msg":"Reconciler error","controller":"vmalert","controllerGroup":"operator.victoriametrics.com","controllerKind":"VMAlert","VMAlert":{"name":"k3s-pdt","namespace":"monitoring"},"namespace":"monitoring","name":"k3s-pdt","reconcileID":"4bbed69e-37d8-4c74-9e0a-43f6a132aea7","error":"Internal error occurred: failed calling webhook \"vmalert.victoriametrics.com\": failed to call webhook: Post \"https://vm-victoria-metrics-operator.monitoring.svc:443/validate-operator-victoriametrics-com-v1beta1-vmalert?timeout=10s\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-06-28T18:56:07Z is after 2024-06-10T15:01:52Z","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234"}
2024/06/28 18:56:07 http: TLS handshake error from 10.233.1.69:59152: EOF
{"level":"error","ts":1719600967.4129446,"msg":"Reconciler error","controller":"vmalert","controllerGroup":"operator.victoriametrics.com","controllerKind":"VMAlert","VMAlert":{"name":"k3s-pdu-norte","namespace":"monitoring"},"namespace":"monitoring","name":"k3s-pdu-norte","reconcileID":"a5b67d14-7dbe-49c2-bc68-74e82f66dcac","error":"Internal error occurred: failed calling webhook \"vmalert.victoriametrics.com\": failed to call webhook: Post \"https://vm-victoria-metrics-operator.monitoring.svc:443/validate-operator-victoriametrics-com-v1beta1-vmalert?timeout=10s\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-06-28T18:56:07Z is after 2024-06-10T15:01:52Z","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234"}
2024/06/28 18:56:07 http: TLS handshake error from 10.233.1.69:59158: remote error: tls: bad certificate
{"level":"error","ts":1719600967.5166848,"msg":"Reconciler error","controller":"vmalert","controllerGroup":"operator.victoriametrics.com","controllerKind":"VMAlert","VMAlert":{"name":"k3s-tacuarembo","namespace":"monitoring"},"namespace":"monitoring","name":"k3s-tacuarembo","reconcileID":"51a887ad-ffd1-4c29-94d0-ba36636a85da","error":"Internal error occurred: failed calling webhook \"vmalert.victoriametrics.com\": failed to call webhook: Post \"https://vm-victoria-metrics-operator.monitoring.svc:443/validate-operator-victoriametrics-com-v1beta1-vmalert?timeout=10s\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-06-28T18:56:07Z is after 2024-06-10T15:01:52Z","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234"}
2024/06/28 18:56:07 http: TLS handshake error from 10.233.7.228:49168: EOF
{"level":"error","ts":1719600967.6002195,"msg":"Reconciler error","controller":"vmalert","controllerGroup":"operator.victoriametrics.com","controllerKind":"VMAlert","VMAlert":{"name":"k3s-capricho","namespace":"monitoring"},"namespace":"monitoring","name":"k3s-capricho","reconcileID":"3df1b18f-ed44-466c-abf3-46f20b8e8692","error":"Internal error occurred: failed calling webhook \"vmalert.victoriametrics.com\": failed to call webhook: Post \"https://vm-2-victoria-metrics-operator.monitoring.svc:443/validate-operator-victoriametrics-com-v1beta1-vmalert?timeout=10s\": http: server gave HTTP response to HTTPS client","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234"}
2024/06/28 18:56:07 http: TLS handshake error from 10.233.7.228:49178: EOF
{"level":"error","ts":1719600967.6873116,"msg":"Reconciler error","controller":"vmalert","controllerGroup":"operator.victoriametrics.com","controllerKind":"VMAlert","VMAlert":{"name":"k3s-cerrezuelo","namespace":"monitoring"},"namespace":"monitoring","name":"k3s-cerrezuelo","reconcileID":"ebc9adfd-e513-44b0-96b8-f9ce3e40f9f1","error":"Internal error occurred: failed calling webhook \"vmalert.victoriametrics.com\": failed to call webhook: Post \"https://vm-2-victoria-metrics-operator.monitoring.svc:443/validate-operator-victoriametrics-com-v1beta1-vmalert?timeout=10s\": http: server gave HTTP response to HTTPS client","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234"}
2024/06/28 18:56:07 http: TLS handshake error from 10.233.1.69:59168: EOF
{"level":"error","ts":1719600967.7819726,"msg":"Reconciler error","controller":"vmalert","controllerGroup":"operator.victoriametrics.com","controllerKind":"VMAlert","VMAlert":{"name":"k3s-el-criollo","namespace":"monitoring"},"namespace":"monitoring","name":"k3s-el-criollo","reconcileID":"2677e432-225a-43ae-9e4e-1d887bf38bea","error":"Internal error occurred: failed calling webhook \"vmalert.victoriametrics.com\": failed to call webhook: Post \"https://vm-2-victoria-metrics-operator.monitoring.svc:443/validate-operator-victoriametrics-com-v1beta1-vmalert?timeout=10s\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-06-28T18:56:07Z is after 2024-06-10T15:01:52Z","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234"}
2024/06/28 18:56:07 http: TLS handshake error from 10.233.1.69:59182: remote error: tls: bad certificate
{"level":"error","ts":1719600967.87296,"msg":"Reconciler error","controller":"vmalert","controllerGroup":"operator.victoriametrics.com","controllerKind":"VMAlert","VMAlert":{"name":"k3s-sarandi","namespace":"monitoring"},"namespace":"monitoring","name":"k3s-sarandi","reconcileID":"b7d78f3f-875a-4642-91ca-53f797300992","error":"Internal error occurred: failed calling webhook \"vmalert.victoriametrics.com\": failed to call webhook: Post \"https://vm-victoria-metrics-operator.monitoring.svc:443/validate-operator-victoriametrics-com-v1beta1-vmalert?timeout=10s\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-06-28T18:56:07Z is after 2024-06-10T15:01:52Z","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234"}
2024/06/28 18:56:07 http: TLS handshake error from 10.233.1.69:59188: remote error: tls: bad certificate
{"level":"error","ts":1719600967.9818642,"msg":"Reconciler error","controller":"vmalert","controllerGroup":"operator.victoriametrics.com","controllerKind":"VMAlert","VMAlert":{"name":"application-01-aks-main","namespace":"monitoring"},"namespace":"monitoring","name":"application-01-aks-main","reconcileID":"941c0b36-829f-4300-8920-d5bcefbd946c","error":"Internal error occurred: failed calling webhook \"vmalert.victoriametrics.com\": failed to call webhook: Post \"https://vm-2-victoria-metrics-operator.monitoring.svc:443/validate-operator-victoriametrics-com-v1beta1-vmalert?timeout=10s\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-06-28T18:56:07Z is after 2024-06-10T15:01:52Z","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234"}
2024/06/28 18:56:08 http: TLS handshake error from 10.233.1.69:59202: remote error: tls: bad certificate
{"level":"error","ts":1719600968.0813084,"msg":"Reconciler error","controller":"vmalert","controllerGroup":"operator.victoriametrics.com","controllerKind":"VMAlert","VMAlert":{"name":"application-01-aks-stream","namespace":"monitoring"},"namespace":"monitoring","name":"application-01-aks-stream","reconcileID":"2383067b-1e32-4a04-bc81-a4207c788a1e","error":"Internal error occurred: failed calling webhook \"vmalert.victoriametrics.com\": failed to call webhook: Post \"https://vm-2-victoria-metrics-operator.monitoring.svc:443/validate-operator-victoriametrics-com-v1beta1-vmalert?timeout=10s\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-06-28T18:56:08Z is after 2024-06-10T15:01:52Z","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234"}
2024/06/28 18:56:42 http: TLS handshake error from 10.233.7.228:44160: EOF
{"level":"error","ts":1719601002.7849095,"msg":"Reconciler error","controller":"vmalertmanager","controllerGroup":"operator.victoriametrics.com","controllerKind":"VMAlertmanager","VMAlertmanager":{"name":"vm-victoria-metrics-k8s-stack","namespace":"monitoring"},"namespace":"monitoring","name":"vm-victoria-metrics-k8s-stack","reconcileID":"aae777c3-0f7e-4666-b1dd-681cefee0108","error":"Internal error occurred: failed calling webhook \"vmalertmanager.victoriametrics.com\": failed to call webhook: Post \"https://vm-2-victoria-metrics-operator.monitoring.svc:443/validate-operator-victoriametrics-com-v1beta1-vmalertmanager?timeout=10s\": http: server gave HTTP response to HTTPS client","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234"}
{"level":"info","ts":1719601044.8846753,"logger":"controller","msg":"Reconciling VMCluster","Request.Namespace":"monitoring","Request.Name":"vm-victoria-metrics-k8s-stack"}
2024/06/28 18:57:24 http: TLS handshake error from 10.233.7.228:40138: EOF
{"level":"error","ts":1719601044.9755416,"msg":"Reconciler error","controller":"vmcluster","controllerGroup":"operator.victoriametrics.com","controllerKind":"VMCluster","VMCluster":{"name":"vm-victoria-metrics-k8s-stack","namespace":"monitoring"},"namespace":"monitoring","name":"vm-victoria-metrics-k8s-stack","reconcileID":"14a40a4e-f9a9-4d73-b6ac-d0d4a686985f","error":"Internal error occurred: failed calling webhook \"vmcluster.victoriametrics.com\": failed to call webhook: Post \"https://vm-victoria-metrics-operator.monitoring.svc:443/validate-operator-victoriametrics-com-v1beta1-vmcluster?timeout=10s\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-06-28T18:57:24Z is after 2024-06-10T15:01:52Z","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234"}
2024/06/28 18:58:07 http: TLS handshake error from 10.233.1.69:39748: EOF
{"level":"error","ts":1719601087.1605337,"msg":"Reconciler error","controller":"vmalert","controllerGroup":"operator.victoriametrics.com","controllerKind":"VMAlert","VMAlert":{"name":"k3s-frb","namespace":"monitoring"},"namespace":"monitoring","name":"k3s-frb","reconcileID":"0d0b5175-571b-488a-b2c8-ba49f8540c86","error":"Internal error occurred: failed calling webhook \"vmalert.victoriametrics.com\": failed to call webhook: Post \"https://vm-victoria-metrics-operator.monitoring.svc:443/validate-operator-victoriametrics-com-v1beta1-vmalert?timeout=10s\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-06-28T18:58:07Z is after 2024-06-10T15:01:52Z","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234"}
2024/06/28 18:58:07 http: TLS handshake error from 10.233.7.228:39458: EOF
{"level":"error","ts":1719601087.268722,"msg":"Reconciler error","controller":"vmalert","controllerGroup":"operator.victoriametrics.com","controllerKind":"VMAlert","VMAlert":{"name":"k3s-mvd-port","namespace":"monitoring"},"namespace":"monitoring","name":"k3s-mvd-port","reconcileID":"7ee3522e-d46a-4995-9f91-5dfc1453ca07","error":"Internal error occurred: failed calling webhook \"vmalert.victoriametrics.com\": failed to call webhook: Post \"https://vm-victoria-metrics-operator.monitoring.svc:443/validate-operator-victoriametrics-com-v1beta1-vmalert?timeout=10s\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-06-28T18:58:07Z is after 2024-06-10T15:01:52Z","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234"}
2024/06/28 18:58:07 http: TLS handshake error from 10.233.1.69:39764: EOF
{"level":"error","ts":1719601087.3767102,"msg":"Reconciler error","controller":"vmalert","controllerGroup":"operator.victoriametrics.com","controllerKind":"VMAlert","VMAlert":{"name":"k3s-pdt","namespace":"monitoring"},"namespace":"monitoring","name":"k3s-pdt","reconcileID":"cdaf0192-e4b3-4008-bba9-7d8c57449336","error":"Internal error occurred: failed calling webhook \"vmalert.victoriametrics.com\": failed to call webhook: Post \"https://vm-2-victoria-metrics-operator.monitoring.svc:443/validate-operator-victoriametrics-com-v1beta1-vmalert?timeout=10s\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-06-28T18:58:07Z is after 2024-06-10T15:01:52Z","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234"}
2024/06/28 18:58:07 http: TLS handshake error from 10.233.1.69:39772: EOF
{"level":"error","ts":1719601087.5031838,"msg":"Reconciler error","controller":"vmalert","controllerGroup":"operator.victoriametrics.com","controllerKind":"VMAlert","VMAlert":{"name":"k3s-pdu-norte","namespace":"monitoring"},"namespace":"monitoring","name":"k3s-pdu-norte","reconcileID":"c67ba7ee-bd6f-4d22-a63d-72081ebbc4ef","error":"Internal error occurred: failed calling webhook \"vmalert.victoriametrics.com\": failed to call webhook: Post \"https://vm-victoria-metrics-operator.monitoring.svc:443/validate-operator-victoriametrics-com-v1beta1-vmalert?timeout=10s\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-06-28T18:58:07Z is after 2024-06-10T15:01:52Z","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234"}
2024/06/28 18:58:07 http: TLS handshake error from 10.233.1.69:39782: EOF
{"level":"error","ts":1719601087.6053696,"msg":"Reconciler error","controller":"vmalert","controllerGroup":"operator.victoriametrics.com","controllerKind":"VMAlert","VMAlert":{"name":"k3s-tacuarembo","namespace":"monitoring"},"namespace":"monitoring","name":"k3s-tacuarembo","reconcileID":"19113df5-d594-4b0d-9be2-3cdfc988b9d2","error":"Internal error occurred: failed calling webhook \"vmalert.victoriametrics.com\": failed to call webhook: Post \"https://vm-2-victoria-metrics-operator.monitoring.svc:443/validate-operator-victoriametrics-com-v1beta1-vmalert?timeout=10s\": http: server gave HTTP response to HTTPS client","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234"}
2024/06/28 18:58:07 http: TLS handshake error from 10.233.1.69:39790: EOF
{"level":"error","ts":1719601087.703518,"msg":"Reconciler error","controller":"vmalert","controllerGroup":"operator.victoriametrics.com","controllerKind":"VMAlert","VMAlert":{"name":"k3s-capricho","namespace":"monitoring"},"namespace":"monitoring","name":"k3s-capricho","reconcileID":"1998ac8d-fc38-41b3-bd21-ceeed25ea879","error":"Internal error occurred: failed calling webhook \"vmalert.victoriametrics.com\": failed to call webhook: Post \"https://vm-2-victoria-metrics-operator.monitoring.svc:443/validate-operator-victoriametrics-com-v1beta1-vmalert?timeout=10s\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-06-28T18:58:07Z is after 2024-06-10T15:01:52Z","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234"}
2024/06/28 18:58:07 http: TLS handshake error from 10.233.1.69:39800: remote error: tls: bad certificate
{"level":"error","ts":1719601087.7909343,"msg":"Reconciler error","controller":"vmalert","controllerGroup":"operator.victoriametrics.com","controllerKind":"VMAlert","VMAlert":{"name":"k3s-cerrezuelo","namespace":"monitoring"},"namespace":"monitoring","name":"k3s-cerrezuelo","reconcileID":"ec475f38-ddd8-44ab-886c-a54e9da458a8","error":"Internal error occurred: failed calling webhook \"vmalert.victoriametrics.com\": failed to call webhook: Post \"https://vm-victoria-metrics-operator.monitoring.svc:443/validate-operator-victoriametrics-com-v1beta1-vmalert?timeout=10s\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-06-28T18:58:07Z is after 2024-06-10T15:01:52Z","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234"}
2024/06/28 18:58:07 http: TLS handshake error from 10.233.7.228:57304: remote error: tls: bad certificate
{"level":"error","ts":1719601087.886413,"msg":"Reconciler error","controller":"vmalert","controllerGroup":"operator.victoriametrics.com","controllerKind":"VMAlert","VMAlert":{"name":"k3s-el-criollo","namespace":"monitoring"},"namespace":"monitoring","name":"k3s-el-criollo","reconcileID":"3b2b545a-753e-46bb-ab21-04b3c0d0a65c","error":"Internal error occurred: failed calling webhook \"vmalert.victoriametrics.com\": failed to call webhook: Post \"https://vm-victoria-metrics-operator.monitoring.svc:443/validate-operator-victoriametrics-com-v1beta1-vmalert?timeout=10s\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-06-28T18:58:07Z is after 2024-06-10T15:01:52Z","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234"}
2024/06/28 18:58:07 http: TLS handshake error from 10.233.7.228:57306: EOF
{"level":"error","ts":1719601087.9657698,"msg":"Reconciler error","controller":"vmalert","controllerGroup":"operator.victoriametrics.com","controllerKind":"VMAlert","VMAlert":{"name":"k3s-sarandi","namespace":"monitoring"},"namespace":"monitoring","name":"k3s-sarandi","reconcileID":"f8f9e48d-9cef-4a52-bdc7-1e36f3dd1034","error":"Internal error occurred: failed calling webhook \"vmalert.victoriametrics.com\": failed to call webhook: Post \"https://vm-victoria-metrics-operator.monitoring.svc:443/validate-operator-victoriametrics-com-v1beta1-vmalert?timeout=10s\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-06-28T18:58:07Z is after 2024-06-10T15:01:52Z","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234"}
2024/06/28 18:58:08 http: TLS handshake error from 10.233.7.228:57320: remote error: tls: bad certificate
{"level":"error","ts":1719601088.0572538,"msg":"Reconciler error","controller":"vmalert","controllerGroup":"operator.victoriametrics.com","controllerKind":"VMAlert","VMAlert":{"name":"application-01-aks-main","namespace":"monitoring"},"namespace":"monitoring","name":"application-01-aks-main","reconcileID":"23abc8d6-5747-48fc-8d86-1b40eacce120","error":"Internal error occurred: failed calling webhook \"vmalert.victoriametrics.com\": failed to call webhook: Post \"https://vm-2-victoria-metrics-operator.monitoring.svc:443/validate-operator-victoriametrics-com-v1beta1-vmalert?timeout=10s\": http: server gave HTTP response to HTTPS client","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234"}
2024/06/28 18:58:08 http: TLS handshake error from 10.233.7.228:57322: EOF
{"level":"error","ts":1719601088.1581085,"msg":"Reconciler error","controller":"vmalert","controllerGroup":"operator.victoriametrics.com","controllerKind":"VMAlert","VMAlert":{"name":"application-01-aks-stream","namespace":"monitoring"},"namespace":"monitoring","name":"application-01-aks-stream","reconcileID":"31520f3c-39b1-48f9-9d54-da65260ae3b4","error":"Internal error occurred: failed calling webhook \"vmalert.victoriametrics.com\": failed to call webhook: Post \"https://vm-victoria-metrics-operator.monitoring.svc:443/validate-operator-victoriametrics-com-v1beta1-vmalert?timeout=10s\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-06-28T18:58:08Z is after 2024-06-10T15:01:52Z","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234"}

Clean Up logs image

Here an example of the error: image

matrix-root commented 4 months ago

I've solved it by patching all resources with empty finalizers section and deleting all manually

SalvadorOlivar commented 4 months ago

thanks for answering @matrix-root

I ran the following command

kubectl patch vmcluster vm-victoria-metrics-k8s-stack --type=json -p='[{\"op\": \"replace\", \"path\": \"/metadata/finalizers\", \"value\":[]}]' -n monitoring

and it returned the following error

Error from server (InternalError): Internal error occurred: failed calling webhook "vmcluster.victoriametrics.com": failed to call webhook: Post "https://vm-victoria-metrics-operator.monitoring.svc:443/validate-operator-victoriametrics-com-v1beta1-vmcluster?timeout=10s": no endpoints available for service "vm-victoria-metrics-operator"
SalvadorOlivar commented 4 months ago

Ok, I was able to fix it.

What I did first was install a vmoperator and removed ValidatingWebhookConfiguration manually.