Closed mh013370 closed 1 year ago
NifiCluster
NifiDataflow
NifiUser
NifiUserGroup
NifiRegistryClient
NifiParameterContext
On upgrades, nifikop should not crash
I observed the following in the logs:
{"level":"error","time":1661515532734146420,"logger":"parametercontext-method","caller":"clientwrappers/common.go:47","msg":"could not communicate with nifi node","action":"Create parameter-context update-request","error":"non 200 response from NiFi cluster","errorVerbose":"non 200 response from NiFi cluster\ngithub.com/konpyutaika/nifikop/pkg/nificlient.init\n\t/workspace/pkg/nificlient/common.go:11\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6222\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6199\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6199\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:233\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1571","stacktrace":"github.com/konpyutaika/nifikop/pkg/clientwrappers.ErrorCreateOperation\n\t/workspace/pkg/clientwrappers/common.go:47\ngithub.com/konpyutaika/nifikop/pkg/clientwrappers/parametercontext.SyncParameterContext\n\t/workspace/pkg/clientwrappers/parametercontext/parametercontext.go:120\ngithub.com/konpyutaika/nifikop/controllers.(*NifiParameterContextReconciler).Reconcile\n\t/workspace/controllers/nifiparametercontext_controller.go:270\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.1/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.1/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.1/pkg/internal/controller/controller.go:234"} -- Fri, Aug 26 2022 1:05:32 pm | {"level":"info","time":1661515532736150233,"logger":"controllers.NifiParameterContext","caller":"controllers/controller_common.go:26","msg":"failed to sync NifiParameterContext t-nifi-default-parameter-context"} Fri, Aug 26 2022 1:05:32 pm | {"level":"error","time":1661515532738732222,"caller":"controller/controller.go:326","msg":"Reconciler error","controller":"nifinodegroupautoscaler","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiNodeGroupAutoscaler","nifiParameterContext":{"name":"t-nifi-default-parameter-context","namespace":"nifi"},"namespace":"nifi","name":"t-nifi-default-parameter-context","reconcileID":"2ba277ec-8fff-40b6-b7c8-9c0c79c6daf6","error":"non 200 response from NiFi cluster","errorVerbose":"non 200 response from NiFi cluster\ngithub.com/konpyutaika/nifikop/pkg/nificlient.init\n\t/workspace/pkg/nificlient/common.go:11\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6222\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6199\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6199\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:233\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1571","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.1/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.1/pkg/internal/controller/controller.go:234"} Fri, Aug 26 2022 1:05:32 pm | {"level":"error","time":1661515532832712860,"logger":"nifi_client","caller":"nificlient/common.go:26","msg":"Non 200 response from nifi node","statusCode":"500 Internal Server Error","body":"java.net.UnknownHostException: t-nifi-2-node.t-nifi-headless.nifi.svc.cluster.local","stacktrace":"github.com/konpyutaika/nifikop/pkg/nificlient.errorGetOperation\n\t/workspace/pkg/nificlient/common.go:26\ngithub.com/konpyutaika/nifikop/pkg/nificlient.(*nifiClient).GetFlow\n\t/workspace/pkg/nificlient/flow.go:20\ngithub.com/konpyutaika/nifikop/pkg/clientwrappers/dataflow.listComponents\n\t/workspace/pkg/clientwrappers/dataflow/dataflow.go:650\ngithub.com/konpyutaika/nifikop/pkg/clientwrappers/dataflow.IsOutOfSyncDataflow\n\t/workspace/pkg/clientwrappers/dataflow/dataflow.go:151\ngithub.com/konpyutaika/nifikop/controllers.(*NifiDataflowReconciler).Reconcile\n\t/workspace/controllers/nifidataflow_controller.go:389\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.1/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.1/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.1/pkg/internal/controller/controller.go:234"} Fri, Aug 26 2022 1:05:32 pm | {"level":"info","time":1661515532832843522,"caller":"controller/controller.go:117","msg":"Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference","controller":"nifinodegroupautoscaler","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiNodeGroupAutoscaler","nifiDataflow":{"name":"t-nifi-worker-dataflow","namespace":"nifi"},"namespace":"nifi","name":"t-nifi-worker-dataflow","reconcileID":"277bffb7-ee75-4c3b-9d6e-673cd8da9a9f"} Fri, Aug 26 2022 1:05:32 pm | panic: runtime error: invalid memory address or nil pointer dereference [recovered] Fri, Aug 26 2022 1:05:32 pm | panic: runtime error: invalid memory address or nil pointer dereference Fri, Aug 26 2022 1:05:32 pm | [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x14c7117] Fri, Aug 26 2022 1:05:32 pm | Fri, Aug 26 2022 1:05:32 pm | goroutine 575 [running]: Fri, Aug 26 2022 1:05:32 pm | sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1() Fri, Aug 26 2022 1:05:32 pm | /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.1/pkg/internal/controller/controller.go:118 +0x1f4 Fri, Aug 26 2022 1:05:32 pm | panic({0x1690340, 0x263a1c0}) Fri, Aug 26 2022 1:05:32 pm | /usr/local/go/src/runtime/panic.go:838 +0x207 Fri, Aug 26 2022 1:05:32 pm | github.com/konpyutaika/nifikop/pkg/clientwrappers/dataflow.listComponents(0x1b10e78?, {0xc001149140, 0x24}) Fri, Aug 26 2022 1:05:32 pm | /workspace/pkg/clientwrappers/dataflow/dataflow.go:651 +0x77 Fri, Aug 26 2022 1:05:32 pm | github.com/konpyutaika/nifikop/pkg/clientwrappers/dataflow.IsOutOfSyncDataflow(0xc0011c6960, 0xc0004a5180, 0xc0009c1980, 0xc00124a9c0) Fri, Aug 26 2022 1:05:32 pm | /workspace/pkg/clientwrappers/dataflow/dataflow.go:151 +0xda Fri, Aug 26 2022 1:05:32 pm | github.com/konpyutaika/nifikop/controllers.(*NifiDataflowReconciler).Reconcile(0xc0002e2d20, {0x1b10e78, 0xc000c32c60}, {{{0xc000981d18?, 0x10?}, {0xc0006adc20?, 0x40d787?}}}) Fri, Aug 26 2022 1:05:32 pm | /workspace/controllers/nifidataflow_controller.go:389 +0x396e Fri, Aug 26 2022 1:05:32 pm | sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x1b10dd0?, {0x1b10e78?, 0xc000c32c60?}, {{{0xc000981d18?, 0x17beb80?}, {0xc0006adc20?, 0x4041f4?}}}) Fri, Aug 26 2022 1:05:32 pm | /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.1/pkg/internal/controller/controller.go:121 +0xc8 Fri, Aug 26 2022 1:05:32 pm | sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0002e2e60, {0x1b10dd0, 0xc0007a2c80}, {0x16f28a0?, 0xc0005a1dc0?}) Fri, Aug 26 2022 1:05:32 pm | /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.1/pkg/internal/controller/controller.go:320 +0x33c Fri, Aug 26 2022 1:05:32 pm | sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0002e2e60, {0x1b10dd0, 0xc0007a2c80}) Fri, Aug 26 2022 1:05:32 pm | /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.1/pkg/internal/controller/controller.go:273 +0x1d9 Fri, Aug 26 2022 1:05:32 pm | sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2() Fri, Aug 26 2022 1:05:32 pm | /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.1/pkg/internal/controller/controller.go:234 +0x85 Fri, Aug 26 2022 1:05:32 pm | created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 Fri, Aug 26 2022 1:05:32 pm | /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.1/pkg/internal/controller/controller.go:230 +0x325
Avoid the nil pointer reference.
v0.13.1-release
1.18
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.8", GitCommit:"4a3b558c52eb6995b3c5c1db5e54111bd0645a64", GitTreeState:"clean", BuildDate:"2021-12-15T14:52:11Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.6+rke2r2", GitCommit:"ad3338546da947756e8a88aa6822e9c11e7eac22", GitTreeState:"clean", BuildDate:"2022-04-28T19:13:01Z", GoVersion:"go1.17.9b7", Compiler:"gc", Platform:"linux/amd64"}
1.17.0
No response
I haven't seen this re-occur in the last two upgrades so i'll close this and we can raise a new issue if something like this happens in the future.
What steps will reproduce the bug?
NifiCluster
withNifiDataflow
,NifiUser
,NifiUserGroup
,NifiRegistryClient
,NifiParameterContext
using nifikop v0.11.0-releaseWhat is the expected behavior?
On upgrades, nifikop should not crash
What do you see instead?
I observed the following in the logs:
Possible solution
Avoid the nil pointer reference.
NiFiKop version
v0.13.1-release
Golang version
1.18
Kubernetes version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.8", GitCommit:"4a3b558c52eb6995b3c5c1db5e54111bd0645a64", GitTreeState:"clean", BuildDate:"2021-12-15T14:52:11Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.6+rke2r2", GitCommit:"ad3338546da947756e8a88aa6822e9c11e7eac22", GitTreeState:"clean", BuildDate:"2022-04-28T19:13:01Z", GoVersion:"go1.17.9b7", Compiler:"gc", Platform:"linux/amd64"}
NiFi version
1.17.0
Additional context
No response