search

VictoriaMetrics / operator

Kubernetes operator for Victoria Metrics
Apache License 2.0
425 stars 142 forks source link

PodMonitor conversion does not allow tlsConfig.serverName to be empty #1025

Closed bh-tt closed 3 months ago

bh-tt commented 3 months ago

We upgraded to version 0.46.2 from 0.45.0 today and the operator started crashing due to a panic while converting a PodMonitor to a VMPodScrape.

The error occurred at https://github.com/VictoriaMetrics/operator/blob/0c311bb251b8d87ec4ce299829cf555ca25b334c/internal/controller/operator/converter/apis.go#L267 while converting the following PodMonitor:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  annotations:
    meta.helm.sh/release-name: etcd
    meta.helm.sh/release-namespace: it
  creationTimestamp: "2024-02-10T09:37:54Z"
  generation: 2
  labels:
    app.kubernetes.io/instance: etcd
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: etcd
    app.kubernetes.io/version: 3.5.14
    helm.sh/chart: etcd-10.2.6
  name: etcd
  namespace: it
  resourceVersion: "1451335703"
  uid: 654bf596-edcf-4eb8-b50f-e5e906cc96fc
spec:
  namespaceSelector:
    matchNames:
    - it
  podMetricsEndpoints:
  - interval: 30s
    path: /metrics
    port: client
    scheme: https
    scrapeTimeout: 30s
    tlsConfig:
      insecureSkipVerify: true
  selector:
    matchLabels:
      app.kubernetes.io/component: etcd
      app.kubernetes.io/instance: etcd
      app.kubernetes.io/name: etcd

Adding serverName to the tlsConfig fixed the problem, but it appears the operator dereferences the tlsConfig.serverName without checking for nil.

Related logs ``` {"level":"info","ts":"2024-07-05T14:22:06Z","logger":"controller","msg":"api resource is ready","group":"monitoring.coreos.com/v1","kind":"PodMonitor"} {"level":"error","ts":"2024-07-05T14:22:06Z","msg":"Observed a panic: \"invalid memory address or nil pointer dereference\" (runtime error: invalid memory address or nil pointer dereference)\ngoroutine 736 [running]:\nk8s.io/apimachinery/pkg/util/runtime.logPanic({0x1be21a0, 0x311edd0})\n\t/go/pkg/mod/k8s.io/apimachinery@v0.30.2/pkg/util/runtime/runtime.go:75 +0x85\nk8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc00053fdc0?})\n\t/go/pkg/mod/k8s.io/apimachinery@v0.30.2/pkg/util/runtime/runtime.go:49 +0x6b\npanic({0x1be21a0?, 0x311edd0?})\n\t/usr/local/go/src/runtime/panic.go:770 +0x132\ngithub.com/VictoriaMetrics/operator/internal/controller/operator/converter.convertSafeTLSConfig(...)\n\t/workspace/internal/controller/operator/converter/apis.go:267\ngithub.com/VictoriaMetrics/operator/internal/controller/operator/converter.convertPodEndpoints({0xc001477208, 0x1, 0x224f1a0?})\n\t/workspace/internal/controller/operator/converter/apis.go:347 +0x3dc\ngithub.com/VictoriaMetrics/operator/internal/controller/operator/converter.ConvertPodMonitor(0xc0013ab000, 0xc0001ed508)\n\t/workspace/internal/controller/operator/converter/apis.go:375 +0x295\ngithub.com/VictoriaMetrics/operator/internal/controller/operator.(*ConverterController).CreatePodMonitor(0xc0031753b0, {0x1e3fb60?, 0xc0013ab000})\n\t/workspace/internal/controller/operator/vmprometheusconverter_controller.go:428 +0x1b6\nk8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd(...)\n\t/go/pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/controller.go:239\nk8s.io/client-go/tools/cache.(*processorListener).run.func1()\n\t/go/pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/shared_informer.go:978 +0x13e\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)\n\t/go/pkg/mod/k8s.io/apimachinery@v0.30.2/pkg/util/wait/backoff.go:226 +0x33\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00051bf70, {0x2234860, 0xc0015db620}, 0x1, 0xc001d88360)\n\t/go/pkg/mod/k8s.io/apimachinery@v0.30.2/pkg/util/wait/backoff.go:227 +0xaf\nk8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00051bf70, 0x3b9aca00, 0x0, 0x1, 0xc001d88360)\n\t/go/pkg/mod/k8s.io/apimachinery@v0.30.2/pkg/util/wait/backoff.go:204 +0x7f\nk8s.io/apimachinery/pkg/util/wait.Until(...)\n\t/go/pkg/mod/k8s.io/apimachinery@v0.30.2/pkg/util/wait/backoff.go:161\nk8s.io/client-go/tools/cache.(*processorListener).run(0xc0031754d0)\n\t/go/pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/shared_informer.go:972 +0x69\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()\n\t/go/pkg/mod/k8s.io/apimachinery@v0.30.2/pkg/util/wait/wait.go:72 +0x52\ncreated by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 730\n\t/go/pkg/mod/k8s.io/apimachinery@v0.30.2/pkg/util/wait/wait.go:70 +0x73","stacktrace":"k8s.io/klog/v2.(*loggingT).output\n\t/go/pkg/mod/k8s.io/klog/v2@v2.130.1/klog.go:882\nk8s.io/klog/v2.(*loggingT).printfDepth\n\t/go/pkg/mod/k8s.io/klog/v2@v2.130.1/klog.go:760\nk8s.io/klog/v2.(*loggingT).printf\n\t/go/pkg/mod/k8s.io/klog/v2@v2.130.1/klog.go:737\nk8s.io/klog/v2.Errorf\n\t/go/pkg/mod/k8s.io/klog/v2@v2.130.1/klog.go:1597\nk8s.io/apimachinery/pkg/util/runtime.logPanic\n\t/go/pkg/mod/k8s.io/apimachinery@v0.30.2/pkg/util/runtime/runtime.go:79\nk8s.io/apimachinery/pkg/util/runtime.HandleCrash\n\t/go/pkg/mod/k8s.io/apimachinery@v0.30.2/pkg/util/runtime/runtime.go:49\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:770\nruntime.panicmem\n\t/usr/local/go/src/runtime/panic.go:261\nruntime.sigpanic\n\t/usr/local/go/src/runtime/signal_unix.go:881\ngithub.com/VictoriaMetrics/operator/internal/controller/operator/converter.convertSafeTLSConfig\n\t/workspace/internal/controller/operator/converter/apis.go:267\ngithub.com/VictoriaMetrics/operator/internal/controller/operator/converter.convertPodEndpoints\n\t/workspace/internal/controller/operator/converter/apis.go:347\ngithub.com/VictoriaMetrics/operator/internal/controller/operator/converter.ConvertPodMonitor\n\t/workspace/internal/controller/operator/converter/apis.go:375\ngithub.com/VictoriaMetrics/operator/internal/controller/operator.(*ConverterController).CreatePodMonitor\n\t/workspace/internal/controller/operator/vmprometheusconverter_controller.go:428\nk8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd\n\t/go/pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/controller.go:239\nk8s.io/client-go/tools/cache.(*processorListener).run.func1\n\t/go/pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/shared_informer.go:978\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.30.2/pkg/util/wait/backoff.go:226\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.30.2/pkg/util/wait/backoff.go:227\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.30.2/pkg/util/wait/backoff.go:204\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.30.2/pkg/util/wait/backoff.go:161\nk8s.io/client-go/tools/cache.(*processorListener).run\n\t/go/pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/shared_informer.go:972\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.30.2/pkg/util/wait/wait.go:72"} panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x193c51c] goroutine 736 [running]: k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc00053fdc0?}) /go/pkg/mod/k8s.io/apimachinery@v0.30.2/pkg/util/runtime/runtime.go:56 +0xcd panic({0x1be21a0?, 0x311edd0?}) /usr/local/go/src/runtime/panic.go:770 +0x132 github.com/VictoriaMetrics/operator/internal/controller/operator/converter.convertSafeTLSConfig(...) /workspace/internal/controller/operator/converter/apis.go:267 github.com/VictoriaMetrics/operator/internal/controller/operator/converter.convertPodEndpoints({0xc001477208, 0x1, 0x224f1a0?}) /workspace/internal/controller/operator/converter/apis.go:347 +0x3dc github.com/VictoriaMetrics/operator/internal/controller/operator/converter.ConvertPodMonitor(0xc0013ab000, 0xc0001ed508) /workspace/internal/controller/operator/converter/apis.go:375 +0x295 github.com/VictoriaMetrics/operator/internal/controller/operator.(*ConverterController).CreatePodMonitor(0xc0031753b0, {0x1e3fb60?, 0xc0013ab000}) /workspace/internal/controller/operator/vmprometheusconverter_controller.go:428 +0x1b6 k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd(...) /go/pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/controller.go:239 k8s.io/client-go/tools/cache.(*processorListener).run.func1() /go/pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/shared_informer.go:978 +0x13e k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?) /go/pkg/mod/k8s.io/apimachinery@v0.30.2/pkg/util/wait/backoff.go:226 +0x33 k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0004cdf70, {0x2234860, 0xc0015db620}, 0x1, 0xc001d88360) /go/pkg/mod/k8s.io/apimachinery@v0.30.2/pkg/util/wait/backoff.go:227 +0xaf k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00051bf70, 0x3b9aca00, 0x0, 0x1, 0xc001d88360) /go/pkg/mod/k8s.io/apimachinery@v0.30.2/pkg/util/wait/backoff.go:204 +0x7f k8s.io/apimachinery/pkg/util/wait.Until(...) /go/pkg/mod/k8s.io/apimachinery@v0.30.2/pkg/util/wait/backoff.go:161 k8s.io/client-go/tools/cache.(*processorListener).run(0xc0031754d0) /go/pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/shared_informer.go:972 +0x69 k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1() /go/pkg/mod/k8s.io/apimachinery@v0.30.2/pkg/util/wait/wait.go:72 +0x52 created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 730 /go/pkg/mod/k8s.io/apimachinery@v0.30.2/pkg/util/wait/wait.go:70 +0x73 ```
f41gh7 commented 3 months ago

Crash must be fixed at v0.46.3 release