konpyutaika / nifikop

The NiFiKop NiFi Kubernetes operator makes it easy to run Apache NiFi on Kubernetes. Apache NiFI is a free, open-source solution that support powerful and scalable directed graphs of data routing, transformation, and system mediation logic.
https://konpyutaika.github.io/nifikop/
Apache License 2.0
133 stars 45 forks source link

Nifikop crash on v0.11.0 -> v0.13.1 upgrade #156

Closed mh013370 closed 1 year ago

mh013370 commented 2 years ago

What steps will reproduce the bug?

  1. Deploy a 2-node secured NifiCluster with NifiDataflow, NifiUser, NifiUserGroup, NifiRegistryClient, NifiParameterContext using nifikop v0.11.0-release
  2. Upgrade nifikop to v0.13.1-release
  3. Nifikop will crash on a nil pointer reference, restart, and complete the reconciliation

What is the expected behavior?

On upgrades, nifikop should not crash

What do you see instead?

I observed the following in the logs:

{"level":"error","time":1661515532734146420,"logger":"parametercontext-method","caller":"clientwrappers/common.go:47","msg":"could not communicate with nifi node","action":"Create parameter-context update-request","error":"non 200 response from NiFi cluster","errorVerbose":"non 200 response from NiFi cluster\ngithub.com/konpyutaika/nifikop/pkg/nificlient.init\n\t/workspace/pkg/nificlient/common.go:11\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6222\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6199\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6199\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:233\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1571","stacktrace":"github.com/konpyutaika/nifikop/pkg/clientwrappers.ErrorCreateOperation\n\t/workspace/pkg/clientwrappers/common.go:47\ngithub.com/konpyutaika/nifikop/pkg/clientwrappers/parametercontext.SyncParameterContext\n\t/workspace/pkg/clientwrappers/parametercontext/parametercontext.go:120\ngithub.com/konpyutaika/nifikop/controllers.(*NifiParameterContextReconciler).Reconcile\n\t/workspace/controllers/nifiparametercontext_controller.go:270\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.1/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.1/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.1/pkg/internal/controller/controller.go:234"}
--
Fri, Aug 26 2022 1:05:32 pm | {"level":"info","time":1661515532736150233,"logger":"controllers.NifiParameterContext","caller":"controllers/controller_common.go:26","msg":"failed to sync NifiParameterContext t-nifi-default-parameter-context"}
Fri, Aug 26 2022 1:05:32 pm | {"level":"error","time":1661515532738732222,"caller":"controller/controller.go:326","msg":"Reconciler error","controller":"nifinodegroupautoscaler","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiNodeGroupAutoscaler","nifiParameterContext":{"name":"t-nifi-default-parameter-context","namespace":"nifi"},"namespace":"nifi","name":"t-nifi-default-parameter-context","reconcileID":"2ba277ec-8fff-40b6-b7c8-9c0c79c6daf6","error":"non 200 response from NiFi cluster","errorVerbose":"non 200 response from NiFi cluster\ngithub.com/konpyutaika/nifikop/pkg/nificlient.init\n\t/workspace/pkg/nificlient/common.go:11\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6222\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6199\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6199\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:233\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1571","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.1/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.1/pkg/internal/controller/controller.go:234"}
Fri, Aug 26 2022 1:05:32 pm | {"level":"error","time":1661515532832712860,"logger":"nifi_client","caller":"nificlient/common.go:26","msg":"Non 200 response from nifi node","statusCode":"500 Internal Server Error","body":"java.net.UnknownHostException: t-nifi-2-node.t-nifi-headless.nifi.svc.cluster.local","stacktrace":"github.com/konpyutaika/nifikop/pkg/nificlient.errorGetOperation\n\t/workspace/pkg/nificlient/common.go:26\ngithub.com/konpyutaika/nifikop/pkg/nificlient.(*nifiClient).GetFlow\n\t/workspace/pkg/nificlient/flow.go:20\ngithub.com/konpyutaika/nifikop/pkg/clientwrappers/dataflow.listComponents\n\t/workspace/pkg/clientwrappers/dataflow/dataflow.go:650\ngithub.com/konpyutaika/nifikop/pkg/clientwrappers/dataflow.IsOutOfSyncDataflow\n\t/workspace/pkg/clientwrappers/dataflow/dataflow.go:151\ngithub.com/konpyutaika/nifikop/controllers.(*NifiDataflowReconciler).Reconcile\n\t/workspace/controllers/nifidataflow_controller.go:389\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.1/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.1/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.1/pkg/internal/controller/controller.go:234"}
Fri, Aug 26 2022 1:05:32 pm | {"level":"info","time":1661515532832843522,"caller":"controller/controller.go:117","msg":"Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference","controller":"nifinodegroupautoscaler","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiNodeGroupAutoscaler","nifiDataflow":{"name":"t-nifi-worker-dataflow","namespace":"nifi"},"namespace":"nifi","name":"t-nifi-worker-dataflow","reconcileID":"277bffb7-ee75-4c3b-9d6e-673cd8da9a9f"}
Fri, Aug 26 2022 1:05:32 pm | panic: runtime error: invalid memory address or nil pointer dereference [recovered]
Fri, Aug 26 2022 1:05:32 pm | panic: runtime error: invalid memory address or nil pointer dereference
Fri, Aug 26 2022 1:05:32 pm | [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x14c7117]
Fri, Aug 26 2022 1:05:32 pm |  
Fri, Aug 26 2022 1:05:32 pm | goroutine 575 [running]:
Fri, Aug 26 2022 1:05:32 pm | sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
Fri, Aug 26 2022 1:05:32 pm | /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.1/pkg/internal/controller/controller.go:118 +0x1f4
Fri, Aug 26 2022 1:05:32 pm | panic({0x1690340, 0x263a1c0})
Fri, Aug 26 2022 1:05:32 pm | /usr/local/go/src/runtime/panic.go:838 +0x207
Fri, Aug 26 2022 1:05:32 pm | github.com/konpyutaika/nifikop/pkg/clientwrappers/dataflow.listComponents(0x1b10e78?, {0xc001149140, 0x24})
Fri, Aug 26 2022 1:05:32 pm | /workspace/pkg/clientwrappers/dataflow/dataflow.go:651 +0x77
Fri, Aug 26 2022 1:05:32 pm | github.com/konpyutaika/nifikop/pkg/clientwrappers/dataflow.IsOutOfSyncDataflow(0xc0011c6960, 0xc0004a5180, 0xc0009c1980, 0xc00124a9c0)
Fri, Aug 26 2022 1:05:32 pm | /workspace/pkg/clientwrappers/dataflow/dataflow.go:151 +0xda
Fri, Aug 26 2022 1:05:32 pm | github.com/konpyutaika/nifikop/controllers.(*NifiDataflowReconciler).Reconcile(0xc0002e2d20, {0x1b10e78, 0xc000c32c60}, {{{0xc000981d18?, 0x10?}, {0xc0006adc20?, 0x40d787?}}})
Fri, Aug 26 2022 1:05:32 pm | /workspace/controllers/nifidataflow_controller.go:389 +0x396e
Fri, Aug 26 2022 1:05:32 pm | sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x1b10dd0?, {0x1b10e78?, 0xc000c32c60?}, {{{0xc000981d18?, 0x17beb80?}, {0xc0006adc20?, 0x4041f4?}}})
Fri, Aug 26 2022 1:05:32 pm | /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.1/pkg/internal/controller/controller.go:121 +0xc8
Fri, Aug 26 2022 1:05:32 pm | sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0002e2e60, {0x1b10dd0, 0xc0007a2c80}, {0x16f28a0?, 0xc0005a1dc0?})
Fri, Aug 26 2022 1:05:32 pm | /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.1/pkg/internal/controller/controller.go:320 +0x33c
Fri, Aug 26 2022 1:05:32 pm | sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0002e2e60, {0x1b10dd0, 0xc0007a2c80})
Fri, Aug 26 2022 1:05:32 pm | /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.1/pkg/internal/controller/controller.go:273 +0x1d9
Fri, Aug 26 2022 1:05:32 pm | sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
Fri, Aug 26 2022 1:05:32 pm | /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.1/pkg/internal/controller/controller.go:234 +0x85
Fri, Aug 26 2022 1:05:32 pm | created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
Fri, Aug 26 2022 1:05:32 pm | /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.1/pkg/internal/controller/controller.go:230 +0x325

Possible solution

Avoid the nil pointer reference.

NiFiKop version

v0.13.1-release

Golang version

1.18

Kubernetes version

Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.8", GitCommit:"4a3b558c52eb6995b3c5c1db5e54111bd0645a64", GitTreeState:"clean", BuildDate:"2021-12-15T14:52:11Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.6+rke2r2", GitCommit:"ad3338546da947756e8a88aa6822e9c11e7eac22", GitTreeState:"clean", BuildDate:"2022-04-28T19:13:01Z", GoVersion:"go1.17.9b7", Compiler:"gc", Platform:"linux/amd64"}

NiFi version

1.17.0

Additional context

No response

mh013370 commented 1 year ago

I haven't seen this re-occur in the last two upgrades so i'll close this and we can raise a new issue if something like this happens in the future.