search

infinispan / infinispan-operator

Infinispan Operator
https://infinispan.org/docs/infinispan-operator/main/operator.html
Apache License 2.0
48 stars 54 forks source link

Nil pointer derefence on upgrade #2104

Closed ryanemerson closed 3 months ago

ryanemerson commented 3 months ago

It's possible for Operator to get nil pointer dereference for Infinipsan cluster after Operator upgrade. In such case once user updates Infinispan cluster version the Operator will take down the cluster but doesn't bring it back up.

Operator stacktrace:

2024-06-05T16:29:50.097Z    ERROR   controllers.Infinispan  goroutine 572 [running]:
runtime/debug.Stack()
    /usr/lib/golang/src/runtime/debug/stack.go:24 +0x65
github.com/infinispan/infinispan-operator/pkg/reconcile/pipeline/infinispan/pipeline.invokeHandler.func1()
    /remote-source/deps/gomod/src/github.com/infinispan/infinispan-operator/pkg/reconcile/pipeline/infinispan/pipeline/pipeline.go:55 +0xba
panic({0x179ace0, 0x29eaa90})
    /usr/lib/golang/src/runtime/panic.go:884 +0x213
github.com/infinispan/infinispan-operator/pkg/reconcile/pipeline/infinispan/handler/provision.probe({0x0, 0x0, 0x0, 0x0, 0x0})
    /remote-source/deps/gomod/src/github.com/infinispan/infinispan-operator/pkg/reconcile/pipeline/infinispan/handler/provision/pods.go:71 +0xf0
github.com/infinispan/infinispan-operator/pkg/reconcile/pipeline/infinispan/handler/provision.PodLivenessProbe(...)
    /remote-source/deps/gomod/src/github.com/infinispan/infinispan-operator/pkg/reconcile/pipeline/infinispan/handler/provision/pods.go:52
github.com/infinispan/infinispan-operator/pkg/reconcile/pipeline/infinispan/handler/manage.StatefulSetRollingUpgrade(0xc00080c380, {0x1c39c50, 0xc002a4c2a0})
    /remote-source/deps/gomod/src/github.com/infinispan/infinispan-operator/pkg/reconcile/pipeline/infinispan/handler/manage/statefulset_updates.go:95 +0x1708
github.com/infinispan/infinispan-operator/pkg/reconcile/pipeline/infinispan.HandlerFunc.Handle(0x19381a0?, 0x1aa3a01?, {0x1c39c50?, 0xc002a4c2a0?})
    /remote-source/deps/gomod/src/github.com/infinispan/infinispan-operator/pkg/reconcile/pipeline/infinispan/api.go:40 +0x2f
github.com/infinispan/infinispan-operator/pkg/reconcile/pipeline/infinispan/pipeline.invokeHandler({0x1c18180?, 0x1aa3b40?}, 0xc0029a62d0?, {0x1c39c50?, 0xc002a4c2a0?})
    /remote-source/deps/gomod/src/github.com/infinispan/infinispan-operator/pkg/reconcile/pipeline/infinispan/pipeline/pipeline.go:60 +0x83
github.com/infinispan/infinispan-operator/pkg/reconcile/pipeline/infinispan/pipeline.(*impl).Process(0xc0029a6480, {0x1c2cbc8?, 0xc0029a62d0?})
    /remote-source/deps/gomod/src/github.com/infinispan/infinispan-operator/pkg/reconcile/pipeline/infinispan/pipeline/pipeline.go:42 +0x1a5
github.com/infinispan/infinispan-operator/controllers.(*InfinispanReconciler).Reconcile(0xc0009ce120, {0x1c2cbc8, 0xc0029a62d0}, {{{0xc002339780, 0x8}, {0xc002339776, 0xa}}})
    /remote-source/deps/gomod/src/github.com/infinispan/infinispan-operator/controllers/infinispan.go:238 +0x354
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x1c2cb20?, {0x1c2cbc8?, 0xc0029a62d0?}, {{{0xc002339780?, 0x18e5840?}, {0xc002339776?, 0x1c1b758?}}})
    /remote-source/deps/gomod/src/github.com/infinispan/infinispan-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121 +0xc8
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000458960, {0x1c2cb20, 0xc00034b5e0}, {0x17fe480?, 0xc000bd8340?})
    /remote-source/deps/gomod/src/github.com/infinispan/infinispan-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320 +0x327
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000458960, {0x1c2cb20, 0xc00034b5e0})
    /remote-source/deps/gomod/src/github.com/infinispan/infinispan-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273 +0x1d9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
    /remote-source/deps/gomod/src/github.com/infinispan/infinispan-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234 +0x85
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
    /remote-source/deps/gomod/src/github.com/infinispan/infinispan-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:230 +0x587
    {"infinispan": "datagrid/infinispan", "error": "panic occurred: runtime error: invalid memory address or nil pointer dereference"}
runtime.gopanic
    /usr/lib/golang/src/runtime/panic.go:884
runtime.panicmem
    /usr/lib/golang/src/runtime/panic.go:260
runtime.sigpanic
    /usr/lib/golang/src/runtime/signal_unix.go:841
github.com/infinispan/infinispan-operator/pkg/reconcile/pipeline/infinispan/handler/provision.probe
    /remote-source/deps/gomod/src/github.com/infinispan/infinispan-operator/pkg/reconcile/pipeline/infinispan/handler/provision/pods.go:71
github.com/infinispan/infinispan-operator/pkg/reconcile/pipeline/infinispan/handler/provision.PodLivenessProbe
    /remote-source/deps/gomod/src/github.com/infinispan/infinispan-operator/pkg/reconcile/pipeline/infinispan/handler/provision/pods.go:52
github.com/infinispan/infinispan-operator/pkg/reconcile/pipeline/infinispan/handler/manage.StatefulSetRollingUpgrade
    /remote-source/deps/gomod/src/github.com/infinispan/infinispan-operator/pkg/reconcile/pipeline/infinispan/handler/manage/statefulset_updates.go:95
github.com/infinispan/infinispan-operator/pkg/reconcile/pipeline/infinispan.HandlerFunc.Handle
    /remote-source/deps/gomod/src/github.com/infinispan/infinispan-operator/pkg/reconcile/pipeline/infinispan/api.go:40
github.com/infinispan/infinispan-operator/pkg/reconcile/pipeline/infinispan/pipeline.invokeHandler
    /remote-source/deps/gomod/src/github.com/infinispan/infinispan-operator/pkg/reconcile/pipeline/infinispan/pipeline/pipeline.go:60
github.com/infinispan/infinispan-operator/pkg/reconcile/pipeline/infinispan/pipeline.(*impl).Process
    /remote-source/deps/gomod/src/github.com/infinispan/infinispan-operator/pkg/reconcile/pipeline/infinispan/pipeline/pipeline.go:42
github.com/infinispan/infinispan-operator/controllers.(*InfinispanReconciler).Reconcile
    /remote-source/deps/gomod/src/github.com/infinispan/infinispan-operator/controllers/infinispan.go:238
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    /remote-source/deps/gomod/src/github.com/infinispan/infinispan-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /remote-source/deps/gomod/src/github.com/infinispan/infinispan-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /remote-source/deps/gomod/src/github.com/infinispan/infinispan-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /remote-source/deps/gomod/src/github.com/infinispan/infinispan-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234
2024-06-05T16:29:50.097Z    INFO    controllers.Infinispan  Done    {"infinispan": "datagrid/infinispan", "requeue": true, "requeueAfter": "0s", "error": "panic occurred: runtime error: invalid memory address or nil pointer dereference"}
2024-06-05T16:29:50.097Z    ERROR   Reconciler error    {"controller": "infinispan", "controllerGroup": "infinispan.org", "controllerKind": "Infinispan", "infinispan": {"name":"infinispan","namespace":"datagrid"}, "namespace": "datagrid", "name": "infinispan", "reconcileID": "043bd093-7402-457f-8e04-de155fcaf228", "error": "panic occurred: runtime error: invalid memory address or nil pointer dereference"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /remote-source/deps/gomod/src/github.com/infinispan/infinispan-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /remote-source/deps/gomod/src/github.com/infinispan/infinispan-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234

Cause

When the Spec.Service.Container element is not set prior to upgrade, the the nil pointer dereference error is thrown. This was introduced by #2019, as this code assumes that Spec.Service.Container and the probe values are initialized by the defaulting webhook and calls Spec.Service.Container.*Probe fields regardless of whether the container or probe value pointers are nill or not.

Workaround

Restarting the Operator should allow upgrades to progress as expected