[victoria-metrics-k8s-stack] Operator fails to call webhooks - Server gave HTTP response to HTTPS client

Mahagon commented 2 weeks ago

👋

I am running into failing ValidatingAdmissionWebhooks since the victoria-metrics-k8s-stack update to 0.25.4 (issue persists in 0.25.5). It seems like the client is getting a HTTP, instead of a HTTPS response. Not sure if i misconfigured something, would be nice if someone could help me on this one :)

I already tried disabling the certmanager config entry, but i get the same result.

My helmfile for the Chart deployment:

---
helmDefaults:
  createNamespace: false
---
repositories:
  - name: vm
    url: https://victoriametrics.github.io/helm-charts/
---
releases:
  - name: victoria-metrics-k8s-stack
    namespace: victoria-metrics
    chart: vm/victoria-metrics-k8s-stack
    version: 0.25.5
    values:
      - victoria-metrics-operator:
          cleanupCRD: false
          env:
            - name: VM_ENABLESTRICTSECURITY
              value: true
            - name: VM_VMAGENTDEFAULT_CONFIGRELOADERMEMORY
              value: 50Mi
            - name: VM_USECUSTOMCONFIGRELOADER
              value: true
          podSecurityContext:
            seccompProfile:
              type: "RuntimeDefault"
          securityContext:
            allowPrivilegeEscalation: false
            runAsNonRoot: true
            runAsUser: 65534
            runAsGroup: 65534
            fsGroup: 65534
            capabilities:
              drop:
                - "ALL"
          operator:
            enable_converter_ownership: true
          admissionWebhooks:
            certManager:
              enabled: true

I am getting the following error messages after the upgrade from 0.25.3 to 0.25.4 and also 0.25.5

{"level":"error","ts":"2024-08-29T13:30:10Z","logger":"controller","msg":"cannot create AlertRule from Prometheusrule","kind":"alertRule","name":"pdl-de-single-sign-out-dispatcher-alerts","ns":"victoria-metrics","error":"Internal error occurred: failed calling webhook \"vmrule.victoriametrics.com\": failed to call webhook: Post \"https://victoria-metrics-k8s-stack-victoria-metrics-operator.victoria-metrics.svc:443/validate-operator-victoriametrics-com-v1beta1-vmrule?timeout=10s\": http: server gave HTTP response to HTTPS client"}
{"level":"error","ts":"2024-08-29T13:30:15Z","logger":"controller","msg":"cannot create AlertRule from Prometheusrule","kind":"alertRule","name":"pdl-next-alerts","ns":"victoria-metrics","error":"Internal error occurred: failed calling webhook \"vmrule.victoriametrics.com\": failed to call webhook: Post \"https://victoria-metrics-k8s-stack-victoria-metrics-operator.victoria-metrics.svc:443/validate-operator-victoriametrics-com-v1beta1-vmrule?timeout=10s\": http: server gave HTTP response to HTTPS client"}
{"level":"error","ts":"2024-08-29T13:30:21Z","logger":"controller","msg":"cannot create AlertRule from Prometheusrule","kind":"alertRule","name":"persistentvolume-alerts","ns":"victoria-metrics","error":"Internal error occurred: failed calling webhook \"vmrule.victoriametrics.com\": failed to call webhook: Post \"https://victoria-metrics-k8s-stack-victoria-metrics-operator.victoria-metrics.svc:443/validate-operator-victoriametrics-com-v1beta1-vmrule?timeout=10s\": http: server gave HTTP response to HTTPS client"}
{"level":"error","ts":"2024-08-29T13:30:27Z","logger":"controller","msg":"cannot create AlertRule from Prometheusrule","kind":"alertRule","name":"postgres-alerts","ns":"victoria-metrics","error":"Internal error occurred: failed calling webhook \"vmrule.victoriametrics.com\": failed to call webhook: Post \"https://victoria-metrics-k8s-stack-victoria-metrics-operator.victoria-metrics.svc:443/validate-operator-victoriametrics-com-v1beta1-vmrule?timeout=10s\": http: server gave HTTP response to HTTPS client"}
{"level":"error","ts":"2024-08-29T13:30:32Z","logger":"controller","msg":"cannot create AlertRule from Prometheusrule","kind":"alertRule","name":"prometheus-alerts","ns":"victoria-metrics","error":"Internal error occurred: failed calling webhook \"vmrule.victoriametrics.com\": failed to call webhook: Post \"https://victoria-metrics-k8s-stack-victoria-metrics-operator.victoria-metrics.svc:443/validate-operator-victoriametrics-com-v1beta1-vmrule?timeout=10s\": http: server gave HTTP response to HTTPS client"}
{"level":"error","ts":"2024-08-29T13:30:38Z","logger":"controller","msg":"cannot create AlertRule from Prometheusrule","kind":"alertRule","name":"rbac-manager-alerts","ns":"victoria-metrics","error":"Internal error occurred: failed calling webhook \"vmrule.victoriametrics.com\": failed to call webhook: Post \"https://victoria-metrics-k8s-stack-victoria-metrics-operator.victoria-metrics.svc:443/validate-operator-victoriametrics-com-v1beta1-vmrule?timeout=10s\": http: server gave HTTP response to HTTPS client"}
{"level":"error","ts":"2024-08-29T13:30:43Z","logger":"controller","msg":"cannot create AlertRule from Prometheusrule","kind":"alertRule","name":"sealed-secrets-alerts","ns":"victoria-metrics","error":"Internal error occurred: failed calling webhook \"vmrule.victoriametrics.com\": failed to call webhook: Post \"https://victoria-metrics-k8s-stack-victoria-metrics-operator.victoria-metrics.svc:443/validate-operator-victoriametrics-com-v1beta1-vmrule?timeout=10s\": http: server gave HTTP response to HTTPS client"}
{"level":"error","ts":"2024-08-29T13:30:49Z","logger":"controller","msg":"cannot create AlertRule from Prometheusrule","kind":"alertRule","name":"tempo-alerts","ns":"victoria-metrics","error":"Internal error occurred: failed calling webhook \"vmrule.victoriametrics.com\": failed to call webhook: Post \"https://victoria-metrics-k8s-stack-victoria-metrics-operator.victoria-metrics.svc:443/validate-operator-victoriametrics-com-v1beta1-vmrule?timeout=10s\": http: server gave HTTP response to HTTPS client"}
{"level":"error","ts":"2024-08-29T13:30:54Z","logger":"controller","msg":"cannot create AlertRule from Prometheusrule","kind":"alertRule","name":"tempo-rules","ns":"victoria-metrics","error":"Internal error occurred: failed calling webhook \"vmrule.victoriametrics.com\": failed to call webhook: Post \"https://victoria-metrics-k8s-stack-victoria-metrics-operator.victoria-metrics.svc:443/validate-operator-victoriametrics-com-v1beta1-vmrule?timeout=10s\": http: server gave HTTP response to HTTPS client"}
{"level":"error","ts":"2024-08-29T13:30:59Z","logger":"controller","msg":"cannot create AlertRule from Prometheusrule","kind":"alertRule","name":"tigera-operator-alerts","ns":"victoria-metrics","error":"Internal error occurred: failed calling webhook \"vmrule.victoriametrics.com\": failed to call webhook: Post \"https://victoria-metrics-k8s-stack-victoria-metrics-operator.victoria-metrics.svc:443/validate-operator-victoriametrics-com-v1beta1-vmrule?timeout=10s\": http: server gave HTTP response to HTTPS client"}
{"level":"error","ts":"2024-08-29T13:31:05Z","logger":"controller","msg":"cannot create AlertRule from Prometheusrule","kind":"alertRule","name":"timebooking-alerts","ns":"victoria-metrics","error":"Internal error occurred: failed calling webhook \"vmrule.victoriametrics.com\": failed to call webhook: Post \"https://victoria-metrics-k8s-stack-victoria-metrics-operator.victoria-metrics.svc:443/validate-operator-victoriametrics-com-v1beta1-vmrule?timeout=10s\": http: server gave HTTP response to HTTPS client"}
{"level":"error","ts":"2024-08-29T13:31:11Z","logger":"controller","msg":"cannot update VMRule","kind":"VMRule","name":"loki-scalable-loki-alerts","ns":"loki","error":"Internal error occurred: failed calling webhook \"vmrule.victoriametrics.com\": failed to call webhook: Post \"https://victoria-metrics-k8s-stack-victoria-metrics-operator.victoria-metrics.svc:443/validate-operator-victoriametrics-com-v1beta1-vmrule?timeout=10s\": http: server gave HTTP response to HTTPS client"}
{"level":"error","ts":"2024-08-29T13:31:17Z","logger":"controller","msg":"cannot update VMRule","kind":"VMRule","name":"loki-scalable-loki-rules","ns":"loki","error":"Internal error occurred: failed calling webhook \"vmrule.victoriametrics.com\": failed to call webhook: Post \"https://victoria-metrics-k8s-stack-victoria-metrics-operator.victoria-metrics.svc:443/validate-operator-victoriametrics-com-v1beta1-vmrule?timeout=10s\": http: server gave HTTP response to HTTPS client"}
{"level":"error","ts":"2024-08-29T13:31:23Z","logger":"controller","msg":"cannot update VMRule","kind":"VMRule","name":"promtail","ns":"loki","error":"Internal error occurred: failed calling webhook \"vmrule.victoriametrics.com\": failed to call webhook: Post \"https://victoria-metrics-k8s-stack-victoria-metrics-operator.victoria-metrics.svc:443/validate-operator-victoriametrics-com-v1beta1-vmrule?timeout=10s\": http: server gave HTTP response to HTTPS client"}
{"level":"error","ts":"2024-08-29T13:31:30Z","logger":"controller","msg":"cannot update VMRule","kind":"VMRule","name":"velero-alerts","ns":"velero","error":"Internal error occurred: failed calling webhook \"vmrule.victoriametrics.com\": failed to call webhook: Post \"https://victoria-metrics-k8s-stack-victoria-metrics-operator.victoria-metrics.svc:443/validate-operator-victoriametrics-com-v1beta1-vmrule?timeout=10s\": http: server gave HTTP response to HTTPS client"}
{"level":"error","ts":"2024-08-29T13:31:35Z","logger":"controller","msg":"cannot update VMRule","kind":"VMRule","name":"argocd-alerts","ns":"victoria-metrics","error":"Internal error occurred: failed calling webhook \"vmrule.victoriametrics.com\": failed to call webhook: Post \"https://victoria-metrics-k8s-stack-victoria-metrics-operator.victoria-metrics.svc:443/validate-operator-victoriametrics-com-v1beta1-vmrule?timeout=10s\": http: server gave HTTP response to HTTPS client"}
{"level":"error","ts":"2024-08-29T13:31:42Z","logger":"controller","msg":"cannot update VMRule","kind":"VMRule","name":"artifact-archiving-alerts","ns":"victoria-metrics","error":"Internal error occurred: failed calling webhook \"vmrule.victoriametrics.com\": failed to call webhook: Post \"https://victoria-metrics-k8s-stack-victoria-metrics-operator.victoria-metrics.svc:443/validate-operator-victoriametrics-com-v1beta1-vmrule?timeout=10s\": http: server gave HTTP response to HTTPS client"}
{"level":"error","ts":"2024-08-29T13:31:48Z","logger":"controller","msg":"cannot update VMRule","kind":"VMRule","name":"azure-quota-alerts","ns":"victoria-metrics","error":"Internal error occurred: failed calling webhook \"vmrule.victoriametrics.com\": failed to call webhook: Post \"https://victoria-metrics-k8s-stack-victoria-metrics-operator.victoria-metrics.svc:443/validate-operator-victoriametrics-com-v1beta1-vmrule?timeout=10s\": http: server gave HTTP response to HTTPS client"}
{"level":"error","ts":"2024-08-29T13:31:54Z","logger":"controller","msg":"cannot update VMRule","kind":"VMRule","name":"calico-alerts","ns":"victoria-metrics","error":"Internal error occurred: failed calling webhook \"vmrule.victoriametrics.com\": failed to call webhook: Post \"https://victoria-metrics-k8s-stack-victoria-metrics-operator.victoria-metrics.svc:443/validate-operator-victoriametrics-com-v1beta1-vmrule?timeout=10s\": http: server gave HTTP response to HTTPS client"}
{"level":"error","ts":"2024-08-29T13:31:59Z","logger":"controller","msg":"cannot update VMRule","kind":"VMRule","name":"cert-manager-alerts","ns":"victoria-metrics","error":"Internal error occurred: failed calling webhook \"vmrule.victoriametrics.com\": failed to call webhook: Post \"https://victoria-metrics-k8s-stack-victoria-metrics-operator.victoria-metrics.svc:443/validate-operator-victoriametrics-com-v1beta1-vmrule?timeout=10s\": http: server gave HTTP response to HTTPS client"}
{"level":"error","ts":"2024-08-29T13:32:05Z","logger":"controller","msg":"cannot update VMRule","kind":"VMRule","name":"core-shared-alerts","ns":"victoria-metrics","error":"Internal error occurred: failed calling webhook \"vmrule.victoriametrics.com\": failed to call webhook: Post \"https://victoria-metrics-k8s-stack-victoria-metrics-operator.victoria-metrics.svc:443/validate-operator-victoriametrics-com-v1beta1-vmrule?timeout=10s\": http: server gave HTTP response to HTTPS client"}
{"level":"error","ts":"2024-08-29T13:32:10Z","logger":"controller","msg":"cannot update VMRule","kind":"VMRule","name":"coredns-alerts","ns":"victoria-metrics","error":"Internal error occurred: failed calling webhook \"vmrule.victoriametrics.com\": failed to call webhook: Post \"https://victoria-metrics-k8s-stack-victoria-metrics-operator.victoria-metrics.svc:443/validate-operator-victoriametrics-com-v1beta1-vmrule?timeout=10s\": http: server gave HTTP response to HTTPS client"}
{"level":"error","ts":"2024-08-29T13:32:15Z","logger":"controller","msg":"cannot update VMRule","kind":"VMRule","name":"crossplane-alerts","ns":"victoria-metrics","error":"Internal error occurred: failed calling webhook \"vmrule.victoriametrics.com\": failed to call webhook: Post \"https://victoria-metrics-k8s-stack-victoria-metrics-operator.victoria-metrics.svc:443/validate-operator-victoriametrics-com-v1beta1-vmrule?timeout=10s\": http: server gave HTTP response to HTTPS client"}
{"level":"error","ts":"2024-08-29T13:32:20Z","logger":"controller","msg":"cannot update VMRule","kind":"VMRule","name":"eck-operator-alerts","ns":"victoria-metrics","error":"Internal error occurred: failed calling webhook \"vmrule.victoriametrics.com\": failed to call webhook: Post \"https://victoria-metrics-k8s-stack-victoria-metrics-operator.victoria-metrics.svc:443/validate-operator-victoriametrics-com-v1beta1-vmrule?timeout=10s\": http: server gave HTTP response to HTTPS client"}
{"level":"error","ts":"2024-08-29T13:32:25Z","logger":"controller","msg":"cannot update VMRule","kind":"VMRule","name":"elasticsearch-alerts","ns":"victoria-metrics","error":"Internal error occurred: failed calling webhook \"vmrule.victoriametrics.com\": failed to call webhook: Post \"https://victoria-metrics-k8s-stack-victoria-metrics-operator.victoria-metrics.svc:443/validate-operator-victoriametrics-com-v1beta1-vmrule?timeout=10s\": http: server gave HTTP response to HTTPS client"}
{"level":"error","ts":"2024-08-29T13:32:30Z","logger":"controller","msg":"cannot update VMRule","kind":"VMRule","name":"external-dns-alerts","ns":"victoria-metrics","error":"Internal error occurred: failed calling webhook \"vmrule.victoriametrics.com\": failed to call webhook: Post \"https://victoria-metrics-k8s-stack-victoria-metrics-operator.victoria-metrics.svc:443/validate-operator-victoriametrics-com-v1beta1-vmrule?timeout=10s\": http: server gave HTTP response to HTTPS client"}
{"level":"error","ts":"2024-08-29T13:32:35Z","logger":"controller","msg":"cannot update VMRule","kind":"VMRule","name":"grafana-alloy-alerts","ns":"victoria-metrics","error":"Internal error occurred: failed calling webhook \"vmrule.victoriametrics.com\": failed to call webhook: Post \"https://victoria-metrics-k8s-stack-victoria-metrics-operator.victoria-metrics.svc:443/validate-operator-victoriametrics-com-v1beta1-vmrule?timeout=10s\": http: server gave HTTP response to HTTPS client"}
{"level":"error","ts":"2024-08-29T13:32:40Z","logger":"controller","msg":"cannot update VMRule","kind":"VMRule","name":"ingress-nginx-alerts","ns":"victoria-metrics","error":"Internal error occurred: failed calling webhook \"vmrule.victoriametrics.com\": failed to call webhook: Post \"https://victoria-metrics-k8s-stack-victoria-metrics-operator.victoria-metrics.svc:443/validate-operator-victoriametrics-com-v1beta1-vmrule?timeout=10s\": http: server gave HTTP response to HTTPS client"}
{"level":"error","ts":"2024-08-29T13:32:45Z","logger":"controller","msg":"cannot update VMRule","kind":"VMRule","name":"jaeger-alerts","ns":"victoria-metrics","error":"Internal error occurred: failed calling webhook \"vmrule.victoriametrics.com\": failed to call webhook: Post \"https://victoria-metrics-k8s-stack-victoria-metrics-operator.victoria-metrics.svc:443/validate-operator-victoriametrics-com-v1beta1-vmrule?timeout=10s\": http: server gave HTTP response to HTTPS client"}
{"level":"error","ts":"2024-08-29T13:32:51Z","logger":"controller","msg":"cannot update VMRule","kind":"VMRule","name":"kube-api-alerts","ns":"victoria-metrics","error":"Internal error occurred: failed calling webhook \"vmrule.victoriametrics.com\": failed to call webhook: Post \"https://victoria-metrics-k8s-stack-victoria-metrics-operator.victoria-metrics.svc:443/validate-operator-victoriametrics-com-v1beta1-vmrule?timeout=10s\": http: server gave HTTP response to HTTPS client"}

AndrewChubatiuk commented 2 weeks ago

please try to run with valid podSecurityContext and securityContext values

victoria-metrics-operator:
  cleanupCRD: false
  env:
  - name: VM_ENABLESTRICTSECURITY
    value: true
  - name: VM_VMAGENTDEFAULT_CONFIGRELOADERMEMORY
    value: 50Mi
  - name: VM_USECUSTOMCONFIGRELOADER
    value: true
  securityContext:
    seccompProfile:
      type: "RuntimeDefault"
    allowPrivilegeEscalation: false
    capabilities:
      drop:
        - "ALL"
  podSecurityContext:
    runAsNonRoot: true
    runAsUser: 65534
    runAsGroup: 65534
    fsGroup: 65534
  operator:
    enable_converter_ownership: true
  admissionWebhooks:
    certManager:
      enabled: true

Mahagon commented 1 week ago

Removing fsGroup was an issue, but not the cause of this :)

But i was able to fix it anyways, it was missing a Network Policy so that the AKS control plane can reach the operator:

          - apiVersion: crd.projectcalico.org/v1
            kind: NetworkPolicy
            metadata:
              name: allow-control-plane-to-vmoperator-validating-webhook
            spec:
              selector: app.kubernetes.io/name == 'victoria-metrics-operator'
              types:
                - Ingress
              ingress:
                - action: Allow
                  source:
                    namespaceSelector: has(projectcalico.org/name) && projectcalico.org/name == 'kube-system'
                    selector: component == 'tunnel'
                  destination:
                    services:
                      name: victoria-metrics-k8s-stack-victoria-metrics-operator
                      namespace: victoria-metrics

So the mistake was on my side, thanks for helping. ❤️

VictoriaMetrics / helm-charts

[victoria-metrics-k8s-stack] Operator fails to call webhooks - Server gave HTTP response to HTTPS client #1367