Installing the latest kubevirt hyperconverged via OLM throws a nil pointer exception

[nuwang]

/kind bug

What happened: Installing hco-operator v1.10.0 via OperatorHub throws a nil pointer exception and does not start up.

How to reproduce it (as minimally and precisely as possible):

Install v1.10.0 of HCO on EKS or upgrade from v1.9.0 to v1.10.0

Anything else we need to know?:

Stack trace

{"level":"info","ts":"2023-12-30T20:59:55Z","logger":"hyperconverged-operator-cmd","msg":"Go Version: go1.19.2"}
{"level":"info","ts":"2023-12-30T20:59:55Z","logger":"hyperconverged-operator-cmd","msg":"Go OS/Arch: linux/amd64"}
{"level":"info","ts":"2023-12-30T20:59:55Z","logger":"hyperconverged-operator-cmd","msg":"Found namespace","Namespace":"kubevirt-hyperconverged"}
{"level":"info","ts":"2023-12-30T20:59:55Z","logger":"controller-runtime.metrics","msg":"Metrics server is starting to listen","addr":"0.0.0.0:8383"}
{"level":"info","ts":"2023-12-30T20:59:55Z","logger":"hyperconverged-operator-cmd","msg":"Registering Components."}
{"level":"info","ts":"2023-12-30T20:59:55Z","logger":"hyperconverged-operator-cmd","msg":"Cluster type = kubernetes"}
2023-12-30T20:59:55.634421491Z {"level":"info","ts":"2023-12-30T20:59:55Z","logger":"hyperconverged-operator-cmd","msg":"Found namespace","Namespace":"kubevirt-hyperconverged"}
{"level":"info","ts":"2023-12-30T20:59:55Z","logger":"hyperconverged-operator-cmd","msg":"Found Pod","Pod.Namespace":"kubevirt-hyperconverged","Pod.Name":"hco-operator-74d4b569d7-ctszt"}
{"level":"info","ts":"2023-12-30T20:59:55Z","logger":"hyperconverged-operator-cmd","msg":"Found namespace","Namespace":"kubevirt-hyperconverged"}
{"level":"info","ts":"2023-12-30T20:59:55Z","logger":"hyperconverged-operator-cmd","msg":"Setting OperatorCondition."}
2023-12-30T20:59:55.664508621Z panic: runtime error: invalid memory address or nil pointer dereference
2023-12-30T20:59:55.664526308Z [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xf1ac80]

2023-12-30T20:59:55.664532992Z goroutine 1 [running]:
2023-12-30T20:59:55.664537168Z github.com/operator-framework/api/pkg/operators/v1alpha1.(*ClusterServiceVersion).GetNamespace(0x50?)
    <autogenerated>:1
2023-12-30T20:59:55.664543518Z sigs.k8s.io/controller-runtime/pkg/client.ObjectKeyFromObject({0x1ee5508, 0x0})
2023-12-30T20:59:55.664546925Z  /go/src/github.com/kubevirt/hyperconverged-cluster-operator/vendor/sigs.k8s.io/controller-runtime/pkg/client/interfaces.go:36 +0x27
2023-12-30T20:59:55.664550141Z github.com/kubevirt/hyperconverged-cluster-operator/controllers/operands.newCsvHandler({0x1eda100?, 0xc000246bd0}, {0x1edbe30?, 0xc0005beaf0?})
2023-12-30T20:59:55.664553375Z  /go/src/github.com/kubevirt/hyperconverged-cluster-operator/controllers/operands/csv.go:29 +0x49
2023-12-30T20:59:55.664556718Z github.com/kubevirt/hyperconverged-cluster-operator/controllers/operands.NewOperandHandler({0x1eda100?, 0xc000246bd0}, 0xc000272d90, {0x1edbe30, 0xc0005beaf0}, {0x1ebb878?, 0x2ccaac0})
2023-12-30T20:59:55.664559809Z  /go/src/github.com/kubevirt/hyperconverged-cluster-operator/controllers/operands/operandHandler.go:77 +0xdea
2023-12-30T20:59:55.664574739Z github.com/kubevirt/hyperconverged-cluster-operator/controllers/hyperconverged.newReconciler({0x1edd5f0, 0xc000695380}, {0x1edbe30, 0xc0005beaf0}, {0x1eb6f60?, 0xc00094c430})
2023-12-30T20:59:55.664578372Z  /go/src/github.com/kubevirt/hyperconverged-cluster-operator/controllers/hyperconverged/hyperconverged_controller.go:118 +0x139
github.com/kubevirt/hyperconverged-cluster-operator/controllers/hyperconverged.RegisterReconciler({0x1edd5f0, 0xc000695380}, {0x1edbe30, 0xc0005beaf0}, {0x1eb6f60?, 0xc00094c430?})
2023-12-30T20:59:55.664584702Z  /go/src/github.com/kubevirt/hyperconverged-cluster-operator/controllers/hyperconverged/hyperconverged_controller.go:104 +0x37
2023-12-30T20:59:55.664587885Z main.main()
    /go/src/github.com/kubevirt/hyperconverged-cluster-operator/cmd/hyperconverged-cluster-operator/main.go:160 +0xd25

Image version: quay.io/kubevirt/hyperconverged-cluster-operator@sha256:9a5f03e3fa84aaa07cc334036083a66ffbd467a8e1a1a9acfdfaed733873e6fe

v1.9.0 can be installed and works as expected.

Environment:

• HCO version (use oc get csv -n kubevirt-hyperconverged): v1.10.0
• Kubernetes version (use kubectl version): Client Version: v1.28.3 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.27.8-eks-8cb36c9
• Cloud provider or hardware configuration: EKS
• Install tools: OLM from catalogue: https://operatorhub.io/operator/community-kubevirt-hyperconverged

Fixing process

• [x] Fix issue in main - #2704
• [x] Fix issue in release-1.10 - #2705
• [x] Release new version
• [x] make sure the new index image created and solve the issue

[nunnatsa]

Thanks @nuwang for letting us know!

The problem is that following the instructions on the catalog page, makes OLM to install HCO and the other components on the wrong namespace. The wrong namespace is operators while it should be installed at kubevirt-hyperconverged.

@orenc1 - do you know what may cause it? I suspect it's olm/operator-sdk issue.

[nuwang]

@nunnatsa Thanks for the quick response. Ran into the namespace issue too, but even after installing in the kubevirt-hyperconverged namespace, this still happens with v1.10.0 but works fine with v1.9.0.

[nunnatsa]

Yes, after working with @orenc1 on it - we found that it's a real bug in HCO code. Working on a solution now

[nunnatsa]

@nuwang hi.

Version v1.10.1 fixes this issue.