search

clastix / kamaji

Kamaji is the Hosted Control Plane Manager for Kubernetes.
https://kamaji.clastix.io
Apache License 2.0
1.07k stars 95 forks source link

Runtime error in Operator #447

Closed hwaastad closed 5 months ago

hwaastad commented 5 months ago

Hi, after a kubelet upgrade (unintended) from 1.28 -> 1.30 we see a panic runtime error in operator.

Stack trace:

panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
2024-04-23T17:47:41.589122320Z [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x1271a72]

goroutine 533 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.1-0.20240416095710-67b27f27e514/pkg/internal/controller/controller.go:120 +0x1e5
panic({0x1fb67e0?, 0x389ad80?})
    /usr/local/go/src/runtime/panic.go:770 +0x132
k8s.io/apimachinery/pkg/util/version.(*Version).compareInternal(0x26949b0?, 0xc0026f9500?)
    /go/pkg/mod/k8s.io/apimachinery@v0.30.0/pkg/util/version/version.go:272 +0x12
2024-04-23T17:47:41.589156191Z k8s.io/apimachinery/pkg/util/version.(*Version).LessThan(...)
2024-04-23T17:47:41.589158238Z  /go/pkg/mod/k8s.io/apimachinery@v0.30.0/pkg/util/version/version.go:360
k8s.io/kubernetes/cmd/kubeadm/app/phases/upgrade.GetAvailableUpgrades({0x2676760, 0xc001213bf0}, 0x0, 0x0, {0x26949b0, 0xc0026f9500}, {0x2676720, 0x3943980})
2024-04-23T17:47:41.589169273Z  /go/pkg/mod/k8s.io/kubernetes@v1.30.0/cmd/kubeadm/app/phases/upgrade/compute.go:167 +0x688
2024-04-23T17:47:41.589171971Z github.com/clastix/kamaji/internal/resources.(*KubernetesUpgrade).CreateOrUpdate(0xc00331c800, {0x2673420, 0xc000424620}, 0xc000ab7508)
    /workspace/internal/resources/kubeadm_upgrade.go:75 +0x1f6
github.com/clastix/kamaji/internal/resources.createOrUpdate({0x2673420, 0xc000424620}, {0x267b520, 0xc00331c800}, 0xc000ab7508)
2024-04-23T17:47:41.589184789Z  /workspace/internal/resources/resource.go:92 +0x43
github.com/clastix/kamaji/internal/resources.Handle({0x2673420, 0xc000424620}, {0x267b520, 0xc00331c800}, 0xc000ab7508)
2024-04-23T17:47:41.589189043Z  /workspace/internal/resources/resource.go:67 +0xf2
github.com/clastix/kamaji/controllers.(*TenantControlPlaneReconciler).Reconcile(0xc0001bb680, {0x2673378, 0xc002fbf020}, {{{0xc0008ca360?, 0x5?}, {0xc000483020?, 0xc0015e3d10?}}})
    /workspace/controllers/tenantcontrolplane_controller.go:178 +0xd7e
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x26791d0?, {0x2673378?, 0xc002fbf020?}, {{{0xc0008ca360?, 0xb?}, {0xc000483020?, 0x0?}}})
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.1-0.20240416095710-67b27f27e514/pkg/internal/controller/controller.go:123 +0xb7
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000600420, {0x26733b0, 0xc0007c60a0}, {0x20adb20, 0xc0008ac600})
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.1-0.20240416095710-67b27f27e514/pkg/internal/controller/controller.go:320 +0x3bc
2024-04-23T17:47:41.589257720Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000600420, {0x26733b0, 0xc0007c60a0})
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.1-0.20240416095710-67b27f27e514/pkg/internal/controller/controller.go:270 +0x1be
2024-04-23T17:47:41.589266732Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.1-0.20240416095710-67b27f27e514/pkg/internal/controller/controller.go:231 +0x79
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 149
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.1-0.20240416095710-67b27f27e514/pkg/internal/controller/controller.go:227 +0x50f
prometherion commented 5 months ago

Closed via #448, backported on active branch via #449.