operator-framework / operator-sdk

SDK for building Kubernetes applications. Provides high level APIs, useful abstractions, and project scaffolding.
https://sdk.operatorframework.io
Apache License 2.0
7.25k stars 1.75k forks source link

Failed to run bundle: install plan is not available for the subscription #5410

Closed pohly closed 2 years ago

pohly commented 2 years ago

Bug Report

I also reported this in https://github.com/operator-framework/operator-lifecycle-manager/issues/2454 but as this might also be an issue in operator-sdk, let me also file an issue here.

What did you do?

What did you expect to see?

The operator should start to run.

What did you see instead? Under which circumstances?

operator-sdk run bundle --namespace default --timeout 5m 172.17.42.1:5001/pmem-csi-bundle:v100.0.0 --skip-tls
time="2021-11-17T14:21:35+01:00" level=info msg="Successfully created registry pod: 172-17-42-1-5001-pmem-csi-bundle-v100-0-0"
time="2021-11-17T14:21:35+01:00" level=info msg="Created CatalogSource: pmem-csi-operator-catalog"
time="2021-11-17T14:21:35+01:00" level=info msg="OperatorGroup \"operator-sdk-og\" created"
time="2021-11-17T14:21:35+01:00" level=info msg="Created Subscription: pmem-csi-operator-v100-0-0-sub"
time="2021-11-17T14:26:27+01:00" level=fatal msg="Failed to run bundle: install plan is not available for the subscription pmem-csi-operator-v100-0-0-sub: timed out waiting for the condition\n"

This only happens with OLM 1.19.1. The same commands work when installing OLM 0.18.3 with operator-sdk olm install --version=v0.18.3. UPDATE: there is some randomness involved and it may depend on cluster load and/or state, see https://github.com/operator-framework/operator-sdk/issues/5410#issuecomment-1099581077 and https://github.com/operator-framework/operator-sdk/issues/5410#issuecomment-1105476692.

Environment

Operator type:

/language go

Kubernetes cluster type:

kubeadm in VMs with Kubernetes 1.21.1

$ operator-sdk version

operator-sdk version: "v1.15.0", commit: "f6326e832a8a5e5453d0ad25e86714a0de2c0fc8", kubernetes version: "1.21", go version: "go1.16.10", GOOS: "linux", GOARCH: "amd64"

$ go version (if language is Go)

go version go1.17.2 linux/amd64

$ kubectl version

Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-27T08:53:39Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-05-12T14:12:29Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}

Additional context

I encountered this in PMEM-CSI, tracked there as https://github.com/intel/pmem-csi/issues/1050

More diagnostics:

``` $ kubectl describe --all-namespaces subscriptions Name: pmem-csi-operator-v100-0-0-sub Namespace: default Labels: operators.coreos.com/pmem-csi-operator.default= Annotations: API Version: operators.coreos.com/v1alpha1 Kind: Subscription Metadata: Creation Timestamp: 2021-11-23T15:36:52Z Generation: 1 Resource Version: 3171 UID: 267d37b7-94c6-428c-a472-eab6ee98289a Spec: Channel: alpha Install Plan Approval: Manual Name: pmem-csi-operator Source: pmem-csi-operator-catalog Source Namespace: default Starting CSV: pmem-csi-operator.v100.0.0 Status: Catalog Health: Catalog Source Ref: API Version: operators.coreos.com/v1alpha1 Kind: CatalogSource Name: pmem-csi-operator-catalog Namespace: default Resource Version: 3164 UID: 306ec63a-1b78-4fd2-b06e-085194131acf Healthy: true Last Updated: 2021-11-23T15:36:52Z Catalog Source Ref: API Version: operators.coreos.com/v1alpha1 Kind: CatalogSource Name: operatorhubio-catalog Namespace: olm Resource Version: 3143 UID: 0ff45726-9642-4187-bff3-3738df547344 Healthy: true Last Updated: 2021-11-23T15:36:52Z Conditions: Last Transition Time: 2021-11-23T15:36:52Z Message: all available catalogsources are healthy Reason: AllCatalogSourcesHealthy Status: False Type: CatalogSourcesUnhealthy Last Updated: 2021-11-23T15:36:52Z Events: ```

Note the odd "AllCatalogSourcesHealthy: False". The catalog-operator pod here might be responsible for it (not sure) and reports an error (E1123 15:36:52.688776 1 queueinformer_operator.go:290] sync {"update" "default/pmem-csi-operator-v100-0-0-sub"} failed: Operation cannot be fulfilled on subscriptions.operators.coreos.com "pmem-csi-operator-v100-0-0-sub": the object has been modified; please apply your changes to the latest version and try again):

``` time="2021-11-23T15:36:32Z" level=info msg="log level info" time="2021-11-23T15:36:32Z" level=info msg="TLS keys not set, using non-https for metrics" W1123 15:36:32.456457 1 client_config.go:615] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. time="2021-11-23T15:36:32Z" level=info msg="Using in-cluster kube client config" time="2021-11-23T15:36:32Z" level=info msg="Using in-cluster kube client config" W1123 15:36:32.458198 1 client_config.go:615] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. W1123 15:36:32.486566 1 client_config.go:615] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. time="2021-11-23T15:36:32Z" level=info msg="connection established. cluster-version: v1.21.1" time="2021-11-23T15:36:32Z" level=info msg="operator ready" time="2021-11-23T15:36:32Z" level=info msg="starting informers..." time="2021-11-23T15:36:32Z" level=info msg="informers started" time="2021-11-23T15:36:32Z" level=info msg="waiting for caches to sync..." time="2021-11-23T15:36:32Z" level=info msg="starting workers..." time="2021-11-23T15:36:32Z" level=info msg="connection established. cluster-version: v1.21.1" time="2021-11-23T15:36:32Z" level=info msg="operator ready" time="2021-11-23T15:36:32Z" level=info msg="starting informers..." time="2021-11-23T15:36:32Z" level=info msg="informers started" time="2021-11-23T15:36:32Z" level=info msg="waiting for caches to sync..." time="2021-11-23T15:36:32Z" level=warning msg="couldn't find service in cache" service=operatorhubio-catalog time="2021-11-23T15:36:32Z" level=info msg="starting workers..." time="2021-11-23T15:36:32Z" level=info msg="syncing catalog source for annotation templates" catSrcName=operatorhubio-catalog catSrcNamespace=olm id=Fy47U time="2021-11-23T15:36:32Z" level=info msg="syncing catalog source for annotation templates" catSrcName=operatorhubio-catalog catSrcNamespace=olm id=CUJ52 time="2021-11-23T15:36:33Z" level=error msg="UpdateStatus - error while setting CatalogSource status" error="Operation cannot be fulfilled on catalogsources.operators.coreos.com \"operatorhubio-catalog\": the object has been modified; please apply your changes to the latest version and try again" id=oXhvn source=operatorhubio-catalog E1123 15:36:33.394238 1 queueinformer_operator.go:290] sync {"update" "olm/operatorhubio-catalog"} failed: Operation cannot be fulfilled on catalogsources.operators.coreos.com "operatorhubio-catalog": the object has been modified; please apply your changes to the latest version and try again time="2021-11-23T15:36:33Z" level=warning msg="multiple pods found for selector" selector="olm.catalogSource=operatorhubio-catalog" time="2021-11-23T15:36:33Z" level=warning msg="multiple pods found for selector" selector="olm.catalogSource=operatorhubio-catalog" time="2021-11-23T15:36:33Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=CONNECTING" time="2021-11-23T15:36:33Z" level=info msg="syncing catalog source for annotation templates" catSrcName=operatorhubio-catalog catSrcNamespace=olm id=O+6l9 time="2021-11-23T15:36:33Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=TRANSIENT_FAILURE" time="2021-11-23T15:36:33Z" level=warning msg="multiple pods found for selector" selector="olm.catalogSource=operatorhubio-catalog" time="2021-11-23T15:36:33Z" level=warning msg="multiple pods found for selector" selector="olm.catalogSource=operatorhubio-catalog" time="2021-11-23T15:36:33Z" level=warning msg="multiple pods found for selector" selector="olm.catalogSource=operatorhubio-catalog" time="2021-11-23T15:36:33Z" level=warning msg="multiple pods found for selector" selector="olm.catalogSource=operatorhubio-catalog" time="2021-11-23T15:36:33Z" level=info msg="syncing catalog source for annotation templates" catSrcName=operatorhubio-catalog catSrcNamespace=olm id=jQGRk time="2021-11-23T15:36:34Z" level=warning msg="multiple pods found for selector" selector="olm.catalogSource=operatorhubio-catalog" time="2021-11-23T15:36:34Z" level=warning msg="multiple pods found for selector" selector="olm.catalogSource=operatorhubio-catalog" time="2021-11-23T15:36:34Z" level=info msg="syncing catalog source for annotation templates" catSrcName=operatorhubio-catalog catSrcNamespace=olm id=QyOmv time="2021-11-23T15:36:34Z" level=warning msg="multiple pods found for selector" selector="olm.catalogSource=operatorhubio-catalog" time="2021-11-23T15:36:34Z" level=warning msg="multiple pods found for selector" selector="olm.catalogSource=operatorhubio-catalog" time="2021-11-23T15:36:34Z" level=warning msg="multiple pods found for selector" selector="olm.catalogSource=operatorhubio-catalog" time="2021-11-23T15:36:34Z" level=warning msg="multiple pods found for selector" selector="olm.catalogSource=operatorhubio-catalog" time="2021-11-23T15:36:34Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=CONNECTING" time="2021-11-23T15:36:34Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=TRANSIENT_FAILURE" time="2021-11-23T15:36:34Z" level=warning msg="multiple pods found for selector" selector="olm.catalogSource=operatorhubio-catalog" time="2021-11-23T15:36:34Z" level=warning msg="multiple pods found for selector" selector="olm.catalogSource=operatorhubio-catalog" time="2021-11-23T15:36:34Z" level=warning msg="multiple pods found for selector" selector="olm.catalogSource=operatorhubio-catalog" time="2021-11-23T15:36:34Z" level=warning msg="multiple pods found for selector" selector="olm.catalogSource=operatorhubio-catalog" time="2021-11-23T15:36:35Z" level=info msg="syncing catalog source for annotation templates" catSrcName=operatorhubio-catalog catSrcNamespace=olm id=ByiaJ time="2021-11-23T15:36:36Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=CONNECTING" time="2021-11-23T15:36:36Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=TRANSIENT_FAILURE" time="2021-11-23T15:36:36Z" level=info msg="syncing catalog source for annotation templates" catSrcName=operatorhubio-catalog catSrcNamespace=olm id=9+k2z time="2021-11-23T15:36:36Z" level=info msg="syncing catalog source for annotation templates" catSrcName=operatorhubio-catalog catSrcNamespace=olm id=YUXeH time="2021-11-23T15:36:38Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=CONNECTING" time="2021-11-23T15:36:38Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=TRANSIENT_FAILURE" time="2021-11-23T15:36:38Z" level=info msg="syncing catalog source for annotation templates" catSrcName=operatorhubio-catalog catSrcNamespace=olm id=Y6+kZ time="2021-11-23T15:36:38Z" level=error msg="UpdateStatus - error while setting CatalogSource status" error="Operation cannot be fulfilled on catalogsources.operators.coreos.com \"operatorhubio-catalog\": the object has been modified; please apply your changes to the latest version and try again" id=nDl4j source=operatorhubio-catalog E1123 15:36:38.488617 1 queueinformer_operator.go:290] sync {"update" "olm/operatorhubio-catalog"} failed: Operation cannot be fulfilled on catalogsources.operators.coreos.com "operatorhubio-catalog": the object has been modified; please apply your changes to the latest version and try again time="2021-11-23T15:36:42Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=CONNECTING" time="2021-11-23T15:36:42Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=TRANSIENT_FAILURE" time="2021-11-23T15:36:42Z" level=info msg="syncing catalog source for annotation templates" catSrcName=operatorhubio-catalog catSrcNamespace=olm id=H6Zl9 time="2021-11-23T15:36:42Z" level=info msg="syncing catalog source for annotation templates" catSrcName=operatorhubio-catalog catSrcNamespace=olm id=1g8ma time="2021-11-23T15:36:43Z" level=info msg="syncing catalog source for annotation templates" catSrcName=operatorhubio-catalog catSrcNamespace=olm id=SrQ8o time="2021-11-23T15:36:45Z" level=info msg="syncing catalog source for annotation templates" catSrcName=pmem-csi-operator-catalog catSrcNamespace=default id=vkSRo time="2021-11-23T15:36:45Z" level=info msg="syncing catalog source for annotation templates" catSrcName=pmem-csi-operator-catalog catSrcNamespace=default id=T8OdF time="2021-11-23T15:36:48Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=CONNECTING" time="2021-11-23T15:36:48Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=READY" time="2021-11-23T15:36:48Z" level=info msg="syncing catalog source for annotation templates" catSrcName=operatorhubio-catalog catSrcNamespace=olm id=ZJxsk time="2021-11-23T15:36:52Z" level=info msg="syncing catalog source for annotation templates" catSrcName=pmem-csi-operator-catalog catSrcNamespace=default id=sqS0R time="2021-11-23T15:36:52Z" level=info msg="state.Key.Namespace=default state.Key.Name=pmem-csi-operator-catalog state.State=CONNECTING" time="2021-11-23T15:36:52Z" level=info msg="state.Key.Namespace=default state.Key.Name=pmem-csi-operator-catalog state.State=READY" time="2021-11-23T15:36:52Z" level=info msg="syncing catalog source for annotation templates" catSrcName=pmem-csi-operator-catalog catSrcNamespace=default id=voIzU time="2021-11-23T15:36:52Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink= time="2021-11-23T15:36:52Z" level=info msg="syncing catalog source for annotation templates" catSrcName=pmem-csi-operator-catalog catSrcNamespace=default id=JK73i time="2021-11-23T15:36:52Z" level=info msg="syncing catalog source for annotation templates" catSrcName=pmem-csi-operator-catalog catSrcNamespace=default id=WDa3B time="2021-11-23T15:36:52Z" level=warning msg="an error was encountered during reconciliation" error="Operation cannot be fulfilled on subscriptions.operators.coreos.com \"pmem-csi-operator-v100-0-0-sub\": the object has been modified; please apply your changes to the latest version and try again" event=update reconciling="*v1alpha1.Subscription" selflink= E1123 15:36:52.688776 1 queueinformer_operator.go:290] sync {"update" "default/pmem-csi-operator-v100-0-0-sub"} failed: Operation cannot be fulfilled on subscriptions.operators.coreos.com "pmem-csi-operator-v100-0-0-sub": the object has been modified; please apply your changes to the latest version and try again time="2021-11-23T15:36:52Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink= time="2021-11-23T15:36:52Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink= time="2021-11-23T15:36:52Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink= ```

This repeats a few times but then not anymore. Deleting that pod doesn't help, the recreated one has the same problem.

For comparison, here is the output with OLM 0.18.3. It has the same update error, so that might be a red herring:

``` Name: pmem-csi-operator-v100-0-0-sub Namespace: default Labels: operators.coreos.com/pmem-csi-operator.default= Annotations: API Version: operators.coreos.com/v1alpha1 Kind: Subscription Metadata: Creation Timestamp: 2021-11-23T15:26:12Z Generation: 1 Managed Fields: [...] Resource Version: 1310 UID: 28bd1f43-0d4b-4f7f-bc3a-ddfbc9e4b080 Spec: Channel: alpha Install Plan Approval: Manual Name: pmem-csi-operator Source: pmem-csi-operator-catalog Source Namespace: default Starting CSV: pmem-csi-operator.v100.0.0 Status: Catalog Health: Catalog Source Ref: API Version: operators.coreos.com/v1alpha1 Kind: CatalogSource Name: pmem-csi-operator-catalog Namespace: default Resource Version: 1227 UID: 7d43c145-9139-4b58-8552-5aa9dc1128be Healthy: true Last Updated: 2021-11-23T15:26:24Z Catalog Source Ref: API Version: operators.coreos.com/v1alpha1 Kind: CatalogSource Name: operatorhubio-catalog Namespace: olm Resource Version: 1154 UID: 85a71d8b-ba21-484b-a669-a2fc4d237c8a Healthy: true Last Updated: 2021-11-23T15:26:24Z Conditions: Last Transition Time: 2021-11-23T15:26:24Z Message: all available catalogsources are healthy Reason: AllCatalogSourcesHealthy Status: False Type: CatalogSourcesUnhealthy Last Transition Time: 2021-11-23T15:26:23Z Reason: RequiresApproval Status: True Type: InstallPlanPending Current CSV: pmem-csi-operator.v100.0.0 Install Plan Generation: 1 Install Plan Ref: API Version: operators.coreos.com/v1alpha1 Kind: InstallPlan Name: install-vknrx Namespace: default Resource Version: 1209 UID: 8de30864-1b84-4edb-8425-0f2104649665 Installed CSV: pmem-csi-operator.v100.0.0 Installplan: API Version: operators.coreos.com/v1alpha1 Kind: InstallPlan Name: install-vknrx Uuid: 8de30864-1b84-4edb-8425-0f2104649665 Last Updated: 2021-11-23T15:26:26Z State: AtLatestKnown Events: time="2021-11-23T15:25:48Z" level=info msg="log level info" time="2021-11-23T15:25:48Z" level=info msg="TLS keys not set, using non-https for metrics" W1123 15:25:48.353944 1 client_config.go:614] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. time="2021-11-23T15:25:48Z" level=info msg="Using in-cluster kube client config" time="2021-11-23T15:25:48Z" level=info msg="Using in-cluster kube client config" W1123 15:25:48.355255 1 client_config.go:614] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. time="2021-11-23T15:25:48Z" level=info msg="connection established. cluster-version: v1.21.1" time="2021-11-23T15:25:48Z" level=info msg="operator ready" time="2021-11-23T15:25:48Z" level=info msg="starting informers..." time="2021-11-23T15:25:48Z" level=info msg="informers started" time="2021-11-23T15:25:48Z" level=info msg="waiting for caches to sync..." time="2021-11-23T15:25:48Z" level=info msg="starting workers..." time="2021-11-23T15:25:48Z" level=warning msg="couldn't find service in cache" service=operatorhubio-catalog time="2021-11-23T15:25:48Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=CONNECTING" time="2021-11-23T15:25:49Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=TRANSIENT_FAILURE" time="2021-11-23T15:25:50Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=CONNECTING" time="2021-11-23T15:25:50Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=TRANSIENT_FAILURE" time="2021-11-23T15:25:51Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=CONNECTING" time="2021-11-23T15:25:51Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=TRANSIENT_FAILURE" time="2021-11-23T15:25:54Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=CONNECTING" time="2021-11-23T15:25:54Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=TRANSIENT_FAILURE" time="2021-11-23T15:25:57Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=CONNECTING" time="2021-11-23T15:25:57Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=TRANSIENT_FAILURE" time="2021-11-23T15:26:04Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=CONNECTING" time="2021-11-23T15:26:04Z" level=info msg="state.Key.Namespace=olm state.Key.Name=operatorhubio-catalog state.State=READY" time="2021-11-23T15:26:12Z" level=info msg="state.Key.Namespace=default state.Key.Name=pmem-csi-operator-catalog state.State=CONNECTING" time="2021-11-23T15:26:12Z" level=info msg="state.Key.Namespace=default state.Key.Name=pmem-csi-operator-catalog state.State=READY" time="2021-11-23T15:26:12Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink= time="2021-11-23T15:26:13Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink= time="2021-11-23T15:26:13Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink= time="2021-11-23T15:26:14Z" level=warning msg="no installplan found with matching generation, creating new one" id=n45Np namespace=default time="2021-11-23T15:26:14Z" level=warning msg="no installplan found with matching generation, creating new one" id=jqDzy namespace=default time="2021-11-23T15:26:14Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink= time="2021-11-23T15:26:14Z" level=info msg=syncing id=13yW6 ip=install-7k6m5 namespace=default phase= time="2021-11-23T15:26:14Z" level=info msg="skip processing installplan without status - subscription sync responsible for initial status" id=13yW6 ip=install-7k6m5 namespace=default phase= time="2021-11-23T15:26:14Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink= time="2021-11-23T15:26:14Z" level=info msg=syncing id=p6Zl9 ip=install-vknrx namespace=default phase= time="2021-11-23T15:26:14Z" level=info msg="skip processing installplan without status - subscription sync responsible for initial status" id=p6Zl9 ip=install-vknrx namespace=default phase= time="2021-11-23T15:26:14Z" level=info msg=syncing id=9qY7f ip=install-7k6m5 namespace=default phase=RequiresApproval time="2021-11-23T15:26:14Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink= time="2021-11-23T15:26:14Z" level=warning msg="status not equal, updating..." id=9qY7f ip=install-7k6m5 namespace=default phase=RequiresApproval time="2021-11-23T15:26:15Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink= time="2021-11-23T15:26:15Z" level=info msg=syncing id=eeTWD ip=install-vknrx namespace=default phase=RequiresApproval time="2021-11-23T15:26:15Z" level=warning msg="status not equal, updating..." id=eeTWD ip=install-vknrx namespace=default phase=RequiresApproval time="2021-11-23T15:26:15Z" level=info msg=syncing id=aG2b8 ip=install-7k6m5 namespace=default phase=RequiresApproval time="2021-11-23T15:26:15Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink= time="2021-11-23T15:26:15Z" level=warning msg="status not equal, updating..." id=aG2b8 ip=install-7k6m5 namespace=default phase=RequiresApproval time="2021-11-23T15:26:16Z" level=info msg=syncing id=JskB/ ip=install-vknrx namespace=default phase=RequiresApproval time="2021-11-23T15:26:16Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink= time="2021-11-23T15:26:16Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink= time="2021-11-23T15:26:16Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink= E1123 15:26:16.668254 1 queueinformer_operator.go:290] sync {"update" "default/install-7k6m5"} failed: failed to update installplan bundle lookups: Operation cannot be fulfilled on installplans.operators.coreos.com "install-7k6m5": the object has been modified; please apply your changes to the latest version and try again time="2021-11-23T15:26:16Z" level=info msg=syncing id=fHgiu ip=install-7k6m5 namespace=default phase=RequiresApproval time="2021-11-23T15:26:16Z" level=warning msg="status not equal, updating..." id=fHgiu ip=install-7k6m5 namespace=default phase=RequiresApproval I1123 15:26:17.458456 1 request.go:655] Throttling request took 1.174767095s, request: PUT:https://10.96.0.1:443/apis/operators.coreos.com/v1alpha1/namespaces/default/subscriptions/pmem-csi-operator-v100-0-0-sub/status time="2021-11-23T15:26:17Z" level=warning msg="an error was encountered during reconciliation" error="Operation cannot be fulfilled on subscriptions.operators.coreos.com \"pmem-csi-operator-v100-0-0-sub\": the object has been modified; please apply your changes to the latest version and try again" event=update reconciling="*v1alpha1.Subscription" selflink= E1123 15:26:17.497324 1 queueinformer_operator.go:290] sync {"update" "default/pmem-csi-operator-v100-0-0-sub"} failed: Operation cannot be fulfilled on subscriptions.operators.coreos.com "pmem-csi-operator-v100-0-0-sub": the object has been modified; please apply your changes to the latest version and try again time="2021-11-23T15:26:17Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink= time="2021-11-23T15:26:17Z" level=info msg=syncing id=+Eq0P ip=install-7k6m5 namespace=default phase=RequiresApproval time="2021-11-23T15:26:17Z" level=info msg=syncing id=iZBDQ ip=install-7k6m5 namespace=default phase=RequiresApproval time="2021-11-23T15:26:17Z" level=warning msg="status not equal, updating..." id=iZBDQ ip=install-7k6m5 namespace=default phase=RequiresApproval time="2021-11-23T15:26:18Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink= time="2021-11-23T15:26:18Z" level=warning msg="an error was encountered during reconciliation" error="Operation cannot be fulfilled on subscriptions.operators.coreos.com \"pmem-csi-operator-v100-0-0-sub\": the object has been modified; please apply your changes to the latest version and try again" event=update reconciling="*v1alpha1.Subscription" selflink= E1123 15:26:18.474691 1 queueinformer_operator.go:290] sync {"update" "default/pmem-csi-operator-v100-0-0-sub"} failed: Operation cannot be fulfilled on subscriptions.operators.coreos.com "pmem-csi-operator-v100-0-0-sub": the object has been modified; please apply your changes to the latest version and try again time="2021-11-23T15:26:18Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink= time="2021-11-23T15:26:18Z" level=info msg=syncing id=aQEtu ip=install-7k6m5 namespace=default phase=RequiresApproval time="2021-11-23T15:26:19Z" level=warning msg="an error was encountered during reconciliation" error="Operation cannot be fulfilled on subscriptions.operators.coreos.com \"pmem-csi-operator-v100-0-0-sub\": the object has been modified; please apply your changes to the latest version and try again" event=update reconciling="*v1alpha1.Subscription" selflink= E1123 15:26:19.067953 1 queueinformer_operator.go:290] sync {"update" "default/pmem-csi-operator-v100-0-0-sub"} failed: Operation cannot be fulfilled on subscriptions.operators.coreos.com "pmem-csi-operator-v100-0-0-sub": the object has been modified; please apply your changes to the latest version and try again time="2021-11-23T15:26:19Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink= time="2021-11-23T15:26:19Z" level=warning msg="an error was encountered during reconciliation" error="Operation cannot be fulfilled on subscriptions.operators.coreos.com \"pmem-csi-operator-v100-0-0-sub\": the object has been modified; please apply your changes to the latest version and try again" event=update reconciling="*v1alpha1.Subscription" selflink= E1123 15:26:19.867193 1 queueinformer_operator.go:290] sync {"update" "default/pmem-csi-operator-v100-0-0-sub"} failed: Operation cannot be fulfilled on subscriptions.operators.coreos.com "pmem-csi-operator-v100-0-0-sub": the object has been modified; please apply your changes to the latest version and try again time="2021-11-23T15:26:19Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink= time="2021-11-23T15:26:20Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink= time="2021-11-23T15:26:20Z" level=warning msg="an error was encountered during reconciliation" error="Operation cannot be fulfilled on subscriptions.operators.coreos.com \"pmem-csi-operator-v100-0-0-sub\": the object has been modified; please apply your changes to the latest version and try again" event=update reconciling="*v1alpha1.Subscription" selflink= E1123 15:26:20.670721 1 queueinformer_operator.go:290] sync {"update" "default/pmem-csi-operator-v100-0-0-sub"} failed: Operation cannot be fulfilled on subscriptions.operators.coreos.com "pmem-csi-operator-v100-0-0-sub": the object has been modified; please apply your changes to the latest version and try again time="2021-11-23T15:26:20Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink= time="2021-11-23T15:26:20Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink= time="2021-11-23T15:26:21Z" level=info msg=syncing id=6ZFOt ip=install-vknrx namespace=default phase=RequiresApproval time="2021-11-23T15:26:21Z" level=warning msg="status not equal, updating..." id=6ZFOt ip=install-vknrx namespace=default phase=RequiresApproval time="2021-11-23T15:26:21Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink= time="2021-11-23T15:26:21Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink= time="2021-11-23T15:26:21Z" level=info msg=syncing id=CXnRg ip=install-7k6m5 namespace=default phase=RequiresApproval time="2021-11-23T15:26:21Z" level=info msg="added to bundle, Kind=Service" configmap=default/d328e672e216bb184fcf787a2bfee9769a8ecada9e454ce9618f52aa5809cf1 key=pmem-csi-operator-metrics_v1_service.yaml time="2021-11-23T15:26:21Z" level=info msg="added to bundle, Kind=ClusterServiceVersion" configmap=default/d328e672e216bb184fcf787a2bfee9769a8ecada9e454ce9618f52aa5809cf1 key=pmem-csi-operator.clusterserviceversion.yaml time="2021-11-23T15:26:21Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=default/d328e672e216bb184fcf787a2bfee9769a8ecada9e454ce9618f52aa5809cf1 key=pmem-csi.intel.com_pmemcsideployments.yaml time="2021-11-23T15:26:21Z" level=warning msg="status not equal, updating..." id=CXnRg ip=install-7k6m5 namespace=default phase=RequiresApproval E1123 15:26:21.665355 1 queueinformer_operator.go:290] sync {"update" "default/install-vknrx"} failed: failed to update installplan bundle lookups: Operation cannot be fulfilled on installplans.operators.coreos.com "install-vknrx": the object has been modified; please apply your changes to the latest version and try again time="2021-11-23T15:26:21Z" level=info msg=syncing id=VvKlT ip=install-vknrx namespace=default phase=RequiresApproval time="2021-11-23T15:26:21Z" level=info msg="added to bundle, Kind=ClusterServiceVersion" configmap=default/d328e672e216bb184fcf787a2bfee9769a8ecada9e454ce9618f52aa5809cf1 key=pmem-csi-operator.clusterserviceversion.yaml time="2021-11-23T15:26:21Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=default/d328e672e216bb184fcf787a2bfee9769a8ecada9e454ce9618f52aa5809cf1 key=pmem-csi.intel.com_pmemcsideployments.yaml time="2021-11-23T15:26:21Z" level=info msg="added to bundle, Kind=Service" configmap=default/d328e672e216bb184fcf787a2bfee9769a8ecada9e454ce9618f52aa5809cf1 key=pmem-csi-operator-metrics_v1_service.yaml time="2021-11-23T15:26:21Z" level=warning msg="status not equal, updating..." id=VvKlT ip=install-vknrx namespace=default phase=RequiresApproval time="2021-11-23T15:26:22Z" level=warning msg="an error was encountered during reconciliation" error="Operation cannot be fulfilled on subscriptions.operators.coreos.com \"pmem-csi-operator-v100-0-0-sub\": the object has been modified; please apply your changes to the latest version and try again" event=update reconciling="*v1alpha1.Subscription" selflink= E1123 15:26:22.267411 1 queueinformer_operator.go:290] sync {"update" "default/pmem-csi-operator-v100-0-0-sub"} failed: Operation cannot be fulfilled on subscriptions.operators.coreos.com "pmem-csi-operator-v100-0-0-sub": the object has been modified; please apply your changes to the latest version and try again time="2021-11-23T15:26:22Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink= time="2021-11-23T15:26:22Z" level=info msg=syncing id=6rKkt ip=install-7k6m5 namespace=default phase=RequiresApproval time="2021-11-23T15:26:22Z" level=info msg=syncing id=XWOvv ip=install-vknrx namespace=default phase=RequiresApproval time="2021-11-23T15:26:22Z" level=info msg="added to bundle, Kind=Service" configmap=default/d328e672e216bb184fcf787a2bfee9769a8ecada9e454ce9618f52aa5809cf1 key=pmem-csi-operator-metrics_v1_service.yaml time="2021-11-23T15:26:22Z" level=info msg="added to bundle, Kind=ClusterServiceVersion" configmap=default/d328e672e216bb184fcf787a2bfee9769a8ecada9e454ce9618f52aa5809cf1 key=pmem-csi-operator.clusterserviceversion.yaml time="2021-11-23T15:26:22Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=default/d328e672e216bb184fcf787a2bfee9769a8ecada9e454ce9618f52aa5809cf1 key=pmem-csi.intel.com_pmemcsideployments.yaml time="2021-11-23T15:26:22Z" level=warning msg="status not equal, updating..." id=XWOvv ip=install-vknrx namespace=default phase=RequiresApproval time="2021-11-23T15:26:23Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink= time="2021-11-23T15:26:23Z" level=warning msg="an error was encountered during reconciliation" error="Operation cannot be fulfilled on subscriptions.operators.coreos.com \"pmem-csi-operator-v100-0-0-sub\": the object has been modified; please apply your changes to the latest version and try again" event=update reconciling="*v1alpha1.Subscription" selflink= E1123 15:26:23.269889 1 queueinformer_operator.go:290] sync {"update" "default/pmem-csi-operator-v100-0-0-sub"} failed: Operation cannot be fulfilled on subscriptions.operators.coreos.com "pmem-csi-operator-v100-0-0-sub": the object has been modified; please apply your changes to the latest version and try again time="2021-11-23T15:26:23Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink= time="2021-11-23T15:26:23Z" level=info msg=syncing id=WBhSb ip=install-7k6m5 namespace=default phase=RequiresApproval E1123 15:26:23.671447 1 queueinformer_operator.go:290] sync {"update" "default/install-vknrx"} failed: failed to update installplan bundle lookups: Operation cannot be fulfilled on installplans.operators.coreos.com "install-vknrx": the object has been modified; please apply your changes to the latest version and try again time="2021-11-23T15:26:23Z" level=info msg=syncing id=FL6CM ip=install-vknrx namespace=default phase=RequiresApproval time="2021-11-23T15:26:23Z" level=warning msg="an error was encountered during reconciliation" error="Operation cannot be fulfilled on subscriptions.operators.coreos.com \"pmem-csi-operator-v100-0-0-sub\": the object has been modified; please apply your changes to the latest version and try again" event=update reconciling="*v1alpha1.Subscription" selflink= E1123 15:26:23.868491 1 queueinformer_operator.go:290] sync {"update" "default/pmem-csi-operator-v100-0-0-sub"} failed: Operation cannot be fulfilled on subscriptions.operators.coreos.com "pmem-csi-operator-v100-0-0-sub": the object has been modified; please apply your changes to the latest version and try again time="2021-11-23T15:26:23Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink= time="2021-11-23T15:26:24Z" level=info msg="error updating InstallPlan status" id=WBhSb ip=install-7k6m5 namespace=default phase=RequiresApproval updateError="Operation cannot be fulfilled on installplans.operators.coreos.com \"install-7k6m5\": the object has been modified; please apply your changes to the latest version and try again" E1123 15:26:24.469377 1 queueinformer_operator.go:290] sync {"update" "default/install-7k6m5"} failed: error updating InstallPlan status: Operation cannot be fulfilled on installplans.operators.coreos.com "install-7k6m5": the object has been modified; please apply your changes to the latest version and try again time="2021-11-23T15:26:24Z" level=info msg=syncing id=6rE5N ip=install-7k6m5 namespace=default phase=Installing time="2021-11-23T15:26:24Z" level=info msg=syncing id=HYRIR ip=install-vknrx namespace=default phase=RequiresApproval time="2021-11-23T15:26:24Z" level=warning msg="an error was encountered during reconciliation" error="Operation cannot be fulfilled on subscriptions.operators.coreos.com \"pmem-csi-operator-v100-0-0-sub\": the object has been modified; please apply your changes to the latest version and try again" event=update reconciling="*v1alpha1.Subscription" selflink= E1123 15:26:24.869032 1 queueinformer_operator.go:290] sync {"update" "default/pmem-csi-operator-v100-0-0-sub"} failed: Operation cannot be fulfilled on subscriptions.operators.coreos.com "pmem-csi-operator-v100-0-0-sub": the object has been modified; please apply your changes to the latest version and try again time="2021-11-23T15:26:24Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink= time="2021-11-23T15:26:25Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink= time="2021-11-23T15:26:25Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=default/d328e672e216bb184fcf787a2bfee9769a8ecada9e454ce9618f52aa5809cf1 key=pmem-csi.intel.com_pmemcsideployments.yaml time="2021-11-23T15:26:25Z" level=info msg="added to bundle, Kind=Service" configmap=default/d328e672e216bb184fcf787a2bfee9769a8ecada9e454ce9618f52aa5809cf1 key=pmem-csi-operator-metrics_v1_service.yaml time="2021-11-23T15:26:25Z" level=info msg="added to bundle, Kind=ClusterServiceVersion" configmap=default/d328e672e216bb184fcf787a2bfee9769a8ecada9e454ce9618f52aa5809cf1 key=pmem-csi-operator.clusterserviceversion.yaml time="2021-11-23T15:26:25Z" level=warning msg="an error was encountered during reconciliation" error="Operation cannot be fulfilled on subscriptions.operators.coreos.com \"pmem-csi-operator-v100-0-0-sub\": the object has been modified; please apply your changes to the latest version and try again" event=update reconciling="*v1alpha1.Subscription" selflink= E1123 15:26:25.872468 1 queueinformer_operator.go:290] sync {"update" "default/pmem-csi-operator-v100-0-0-sub"} failed: Operation cannot be fulfilled on subscriptions.operators.coreos.com "pmem-csi-operator-v100-0-0-sub": the object has been modified; please apply your changes to the latest version and try again time="2021-11-23T15:26:25Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink= time="2021-11-23T15:26:25Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink= time="2021-11-23T15:26:26Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink= time="2021-11-23T15:26:27Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink= time="2021-11-23T15:26:27Z" level=info msg=syncing id=klg2t ip=install-7k6m5 namespace=default phase=Installing time="2021-11-23T15:26:27Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink= time="2021-11-23T15:26:28Z" level=info msg="added to bundle, Kind=Service" configmap=default/d328e672e216bb184fcf787a2bfee9769a8ecada9e454ce9618f52aa5809cf1 key=pmem-csi-operator-metrics_v1_service.yaml time="2021-11-23T15:26:28Z" level=info msg="added to bundle, Kind=ClusterServiceVersion" configmap=default/d328e672e216bb184fcf787a2bfee9769a8ecada9e454ce9618f52aa5809cf1 key=pmem-csi-operator.clusterserviceversion.yaml time="2021-11-23T15:26:28Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=default/d328e672e216bb184fcf787a2bfee9769a8ecada9e454ce9618f52aa5809cf1 key=pmem-csi.intel.com_pmemcsideployments.yaml time="2021-11-23T15:26:28Z" level=info msg=syncing id=7Cmjc ip=install-7k6m5 namespace=default phase=Installing time="2021-11-23T15:26:28Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink= time="2021-11-23T15:26:28Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink= time="2021-11-23T15:26:29Z" level=info msg="added to bundle, Kind=Service" configmap=default/d328e672e216bb184fcf787a2bfee9769a8ecada9e454ce9618f52aa5809cf1 key=pmem-csi-operator-metrics_v1_service.yaml time="2021-11-23T15:26:29Z" level=info msg="added to bundle, Kind=ClusterServiceVersion" configmap=default/d328e672e216bb184fcf787a2bfee9769a8ecada9e454ce9618f52aa5809cf1 key=pmem-csi-operator.clusterserviceversion.yaml time="2021-11-23T15:26:29Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=default/d328e672e216bb184fcf787a2bfee9769a8ecada9e454ce9618f52aa5809cf1 key=pmem-csi.intel.com_pmemcsideployments.yaml time="2021-11-23T15:26:29Z" level=info msg="error updating InstallPlan status" id=7Cmjc ip=install-7k6m5 namespace=default phase=Installing updateError="Operation cannot be fulfilled on installplans.operators.coreos.com \"install-7k6m5\": the object has been modified; please apply your changes to the latest version and try again" E1123 15:26:29.471383 1 queueinformer_operator.go:290] sync {"update" "default/install-7k6m5"} failed: error updating InstallPlan status: Operation cannot be fulfilled on installplans.operators.coreos.com "install-7k6m5": the object has been modified; please apply your changes to the latest version and try again time="2021-11-23T15:26:29Z" level=info msg=syncing id=p351h ip=install-7k6m5 namespace=default phase=Complete time="2021-11-23T15:26:29Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink= time="2021-11-23T15:26:29Z" level=info msg=syncing id=6j3+Q ip=install-7k6m5 namespace=default phase=Complete ```
varshaprasad96 commented 2 years ago

@pohly do you have prometheus metrics enabled in your operator? If so, is the prometheus stack set up in the cluster. Just checking to see if that is the cause of the issue. Also can you please mention the logs from the installplan if it exists in the cluster.

camilamacedo86 commented 2 years ago

Hi @pohly,

Could you please let us know?

a) Have we enabled the metrics in your project? if yes, have you installed Prometheus on the cluster? b) Could you provide the project/link for us to try to reproduce your scenario? c) Could you please try to get the details of the deployment for we check if there has a reason for the failure as well?

pohly commented 2 years ago

a) Have we enabled the metrics in your project? if yes, have you installed Prometheus on the cluster?

The operator supports metrics collection and has a "metrics" port - see the manual deployment YAML. Is that what you mean with "enabled the metrics"?

Prometheus is not installed.

b) Could you provide the project/link for us to try to reproduce your scenario?

The project is https://github.com/intel/pmem-csi. First bring up some Kubernetes cluster without OLM installed and set KUBECONFIG.

Then run:

git clone https://github.com/intel/pmem-csi.git
cd pmem-csi
git checkout v1.0.1
sed -i -e 's/OLM_VERSION=v0.18.3/OLM_VERSION=v0.19.1/' test/start-stop-olm.sh
make _work/bin/operator-sdk-1.6.1 operator-generate-bundle
test/start-stop-olm.sh start # you can ignore errors about `_work/pmem-govm/ssh.0`, that's just for diagnostics
kubectl create ns pmem-csi
TEST_BUILD_PMEM_REGISTRY=localhost:5001 TEST_PMEM_REGISTRY=172.17.42.1:5001 TEST_LOCAL_REGISTRY=172.17.42.1:5001 TEST_LOCAL_REGISTRY_SKIP_TLS=true test/start-operator.sh -olm

This needs a Docker registry. You can use some external one like quay.io. In this example, I am running one on port 5001 of my build machine, which can be reached via 172.17.42.1 from inside the cluster. It doesn't use TLS.

Once test/start-operator.sh fails, you need to clean up before trying again:

test/stop-operator.sh -olm

c) Could you please try to get the details of the deployment for we check if there has a reason for the failure as well?

Which details do you need?

pohly commented 2 years ago

Also can you please mention the logs from the installplan if it exists in the cluster.

This is the installplans.operators.coreos.com CRD, right? There is no object of that kind after the failure and also none while operator-sdk is running.

camilamacedo86 commented 2 years ago

Hi @pohly,

Following are some comments inline:

The operator supports metrics collection and has a "metrics" port - see the manual deployment YAML. Is that what you mean with "enabled the metrics"?

You enable the metrics in the config/default/kustomization https://github.com/operator-framework/operator-sdk/blob/master/testdata/go/v3/memcached-operator/config/default/kustomization.yaml#L24-L25 with its layout. However, by looking at your project you deviated from the proposed layout. Also, please be aware of: https://sdk.operatorframework.io/docs/faqs/#can-i-customize-the-projects-initialized-with-operator-sdk

Prometheus is not installed.

If the Operator is integrated with OLM and the bundle has a PodMonitor or a ServiceMonitor (which will be the case when you enable to metrics) the complete InstallPlan will fail on a cluster, which does not have these CRD/the Prometheus operator installed. In this case, you might want to ensure the dependency requirement with OLM dependency or make clear its requirement for the Operator consumers.

If your bundle are you shipping the ServiceMonitor such as https://github.com/operator-framework/operator-sdk/blob/master/testdata/go/v3/memcached-operator/bundle/manifests/memcached-operator-controller-manager-metrics-monitor_monitoring.coreos.com_v1_servicemonitor.yaml#L1-L2 then the installPlan would fail.

That would happen with any CRD/API required for the operator works that do not exist on the cluster.

Could you please provide the bundle that are you using + ensure that the Operator image is published in a public space so we are able to easily reproduce the issue by running operator-sdk run bundle with?

pohly commented 2 years ago

This project existed before we started adding an operator. That determined the layout. We understand that this not how the SDK is normally meant to be used, but rewriting the entire project wasn't ideal either. Thanks for any assistance that you can provide here despite the unusual approach.

Looking at what you said about enabling metrics my conclusion is that we don't enable those.

Here's the bundle content:

``` pohly@pohly-desktop:/nvme/gopath/src/github.com/intel/pmem-csi/deploy/olm-bundle$ cat 1.0.1/manifests/pmem-csi.intel.com_pmemcsideployments.yaml apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: controller-gen.kubebuilder.io/version: v0.3.0 creationTimestamp: null name: pmemcsideployments.pmem-csi.intel.com spec: group: pmem-csi.intel.com names: kind: PmemCSIDeployment listKind: PmemCSIDeploymentList plural: pmemcsideployments shortNames: - pcd singular: pmemcsideployment scope: Cluster versions: - additionalPrinterColumns: - jsonPath: .spec.deviceMode name: DeviceMode type: string - jsonPath: .spec.nodeSelector name: NodeSelector type: string - jsonPath: .spec.image name: Image type: string - jsonPath: .status.phase name: Status type: string - jsonPath: .metadata.creationTimestamp name: Age type: date name: v1beta1 schema: openAPIV3Schema: description: PmemCSIDeployment is the Schema for the deployments API properties: apiVersion: description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' type: string kind: description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' type: string metadata: type: object spec: description: DeploymentSpec defines the desired state of Deployment properties: controllReplicas: description: ControllerReplicas determines how many copys of the controller Pod run concurrently. Zero (= unset) selects the builtin default, which is currently 1. minimum: 0 type: integer controllerDriverResources: description: ControllerDriverResources Compute resources required by central driver container properties: limits: additionalProperties: anyOf: - type: integer - type: string pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true description: 'Limits describes the maximum amount of compute resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/' type: object requests: additionalProperties: anyOf: - type: integer - type: string pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true description: 'Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/' type: object type: object controllerTLSSecret: description: ControllerTLSSecret is the name of a secret which contains ca.crt, tls.crt and tls.key data for the scheduler extender and pod mutation webhook. A controller is started if (and only if) this secret is specified. The special string "-openshift-" enables the usage of https://docs.openshift.com/container-platform/4.6/security/certificates/service-serving-certificate.html to create certificates. type: string deviceMode: description: DeviceMode to use to manage PMEM devices. enum: - lvm - direct type: string image: description: PMEM-CSI driver container image type: string imagePullPolicy: description: PullPolicy image pull policy one of Always, Never, IfNotPresent type: string kubeletDir: description: KubeletDir kubelet's root directory path type: string labels: additionalProperties: type: string description: Labels contains additional labels for all objects created by the operator. type: object logFormat: description: LogFormat enum: - text - json type: string logLevel: description: LogLevel number for the log verbosity type: integer maxUnavailable: anyOf: - type: integer - type: string description: DaemonSets use the default RollingUpdate strategy with at most 1 node not having a running driver pod. That limit can be increased with this setting, either with a higher integer or a percentage. x-kubernetes-int-or-string: true mutatePods: description: MutatePod defines how a mutating pod webhook is configured if a controller is started. The field is ignored if the controller is not enabled. The default is "Try". enum: - Always - Try - Never type: string nodeDriverResources: description: NodeDriverResources Compute resources required by driver container running on worker nodes properties: limits: additionalProperties: anyOf: - type: integer - type: string pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true description: 'Limits describes the maximum amount of compute resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/' type: object requests: additionalProperties: anyOf: - type: integer - type: string pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true description: 'Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/' type: object type: object nodeRegistrarImage: description: NodeRegistrarImage CSI node driver registrar sidecar image type: string nodeRegistrarResources: description: NodeRegistrarResources Compute resources required by node registrar sidecar container properties: limits: additionalProperties: anyOf: - type: integer - type: string pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true description: 'Limits describes the maximum amount of compute resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/' type: object requests: additionalProperties: anyOf: - type: integer - type: string pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true description: 'Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/' type: object type: object nodeSelector: additionalProperties: type: string description: NodeSelector node labels to use for selection of driver node type: object pmemPercentage: description: PMEMPercentage represents the percentage of space to be used by the driver in each PMEM region on every node. Unset (= zero) selects the default of 100%. This is only valid for driver in LVM mode. maximum: 100 minimum: 0 type: integer provisionerImage: description: ProvisionerImage CSI provisioner sidecar image type: string provisionerResources: description: ProvisionerResources Compute resources required by provisioner sidecar container properties: limits: additionalProperties: anyOf: - type: integer - type: string pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true description: 'Limits describes the maximum amount of compute resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/' type: object requests: additionalProperties: anyOf: - type: integer - type: string pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true description: 'Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/' type: object type: object schedulerNodePort: description: SchedulerNodePort, if non-zero, ensures that the "scheduler" service is created as a NodeService with that fixed port number. Otherwise that service is created as a cluster service. The number must be from the range reserved by Kubernetes for node ports. This is useful if the kube-scheduler cannot reach the scheduler extender via a cluster service. format: int32 type: integer type: object status: description: DeploymentStatus defines the observed state of Deployment properties: conditions: description: Conditions items: properties: lastUpdateTime: description: Last time the condition was probed. format: date-time nullable: true type: string reason: description: Message human readable text that explain why this condition is in this state type: string status: description: Status of the condition, one of True, False, Unknown. type: string type: description: Type of condition. type: string required: - status - type type: object type: array driverComponents: items: properties: component: description: 'DriverComponent represents type of the driver: controller or node' type: string lastUpdated: description: LastUpdated time of the driver status format: date-time nullable: true type: string reason: description: Reason represents the human readable text that explains why the driver is in this state. type: string status: description: Status represents the state of the component; one of `Ready` or `NotReady`. Component becomes `Ready` if all the instances(Pods) of the driver component are in running state. Otherwise, `NotReady`. type: string required: - component - reason - status type: object type: array lastUpdated: description: LastUpdated time of the deployment status format: date-time nullable: true type: string phase: description: Phase indicates the state of the deployment type: string reason: type: string type: object type: object served: true storage: true subresources: status: {} status: acceptedNames: kind: "" plural: "" conditions: [] storedVersions: [] pohly@pohly-desktop:/nvme/gopath/src/github.com/intel/pmem-csi/deploy/olm-bundle$ cat 1.0.1/manifests/pmem-csi-operator.clusterserviceversion.yaml apiVersion: operators.coreos.com/v1alpha1 kind: ClusterServiceVersion metadata: annotations: alm-examples: |- [ { "apiVersion": "pmem-csi.intel.com/v1beta1", "kind": "PmemCSIDeployment", "metadata": { "name": "pmem-csi.intel.com" }, "spec": { "deviceMode": "lvm", "nodeSelector": { "feature.node.kubernetes.io/memory-nv.dax": "true" } } } ] capabilities: Seamless Upgrades categories: Storage certified: "false" containerImage: intel/pmem-csi-driver:v1.0.1 createdAt: 2021-11-22T08:19:12Z description: An operator for deploying and managing the PMEM-CSI driver operators.operatorframework.io/builder: operator-sdk-v1.6.1+git operators.operatorframework.io/project_layout: "" repository: https://github.com/intel/pmem-csi/tree/release-1.0 support: Intel name: pmem-csi-operator.v1.0.1 namespace: placeholder spec: apiservicedefinitions: {} customresourcedefinitions: owned: - description: Represents a PMEM-CSI driver deployment in the cluster. displayName: PmemCSIDeployment kind: PmemCSIDeployment name: pmemcsideployments.pmem-csi.intel.com resources: - kind: StatefulSet name: "" replaces: pmem-csi-operator.v1.0.0 version: apps/v1 - kind: Deployment name: "" replaces: pmem-csi-operator.v1.0.0 version: apps/v1 - kind: DaemonSet name: "" replaces: pmem-csi-operator.v1.0.0 version: apps/v1 - kind: Secret name: "" replaces: pmem-csi-operator.v1.0.0 version: v1 - kind: Service name: "" replaces: pmem-csi-operator.v1.0.0 version: v1 replaces: pmem-csi-operator.v1.0.0 version: v1beta1 description: |2 Intel PMEM-CSI is a [CSI](https://github.com/container-storage-interface/spec) storage driver for container orchestrators like Kubernetes. It makes local persistent memory ([PMEM](https://pmem.io/)) available as a filesystem volume to container applications. This is the operator to deploy and manage the [PMEM-CSI](https://intel.github.io/pmem-csi/1.0/README.html) driver on a Kubernetes cluster. Refer to the [PMEM-CSI documentation](https://intel.github.io/pmem-csi/1.0/README.html) for more details on deploying and using the PMEM-CSI driver. displayName: Operator for PMEM-CSI driver icon: - base64data: iVBORw0KGgoAAAANSUhEUgAAAGQAAABkCAYAAABw4pVUAAAx6ElEQVR42s2dCXhcdbn/BxQRRVkVEOSK+4JXBFRUqrKI3CsI/FEUEBAURFAg3ehCCy1tKYVCm+4tdKH73qZrmrTZZ18zS5LJMpNJJslkstL73PuI9/H7f9/f75wzZ07OTCZt8f/v8/yeLE0mc36f8+7v7z0WmPz75z//if/93//Fyf/6L6R6e9He0YHWtrhYLa0xsZpb2rQVbW5FU7RFW41NzWI1NEbF4s/5+/xz/PP8OrF4AvH2DrQnOtHR2YXOZDeSXT3o6k6hJ9Ur/m5vXxrp/j70DwxgcHAQw8PDeP/9909pGX+Xv+bXHKDX7u/vRzqdRi/9zVQqhe7ubnR1dSGZTKKDrj2R6BDvty3Wru0BX4t6nZGGJoTCDQiGIqgPhRGoD8Hnr4fXF4DH64fb44Ofvo7SHqTTffj7Bx+IPTb7ZzF+g0EM0xvuTHbRm2hHW5xAxOI5gfAb0wPRg+A3agaDL0yFkehIZmD09KCnN4VUWgdjcAADCowzBYQ/HxoaMoXRQ+9BhdHZmSQYfOOMhKG/1nCkUcIIEoxgEL5ANgynywOH0w2b3Qm7w0U/F0J3TwofEJi8QPgH+M7kzVLvCP0bKQTGaJLBrxdrJyB0oQm6+1g6MpJhAmNoEIPDQ6cFwwjGCIOXGYwOulniLMnKXuSCoUqHkAwBwz8CBoNQF3/N/9dMr/Xf//3f5kAYRrKrOwuECqNQydDD4GUKQ1FViY5OIYWdXUkkSUV0p3qyYShqijfvTMPg12UYfX19JBkMo5dgpAhGD8HoJhhd4v3pb0x+/3wduWCwdPgDQaGqzGCo0sFf8/f5/3lFGhqzoFhUNdVDksF/PJ9kGIGYqSm9dKgw+PfaYnGhAln8WU1pdqO7S8BQVRXD6BvoF2rqw4LB0sEwVFUlYXRrMFgy2k8RBquq0WC43F7xMwyOf76tLaapLwsbl+Hh97W7Qf8mzoSq4t+TRrxdkwy9EdfDYOlgGKyqVBgfhqrSw5BGnCWjSwDJBUOvqrKMuAEGbzJvOG+8HgYvIwxe/Hv1pOpYUpmFhaUj2aUY8ByqaixqSoPRwjBYVZF0xA1GnPU0qyk24mw36M2kaZP6SI0IVTU0eMZA5PeoegmGlA5VMhIJqarywdAb8Vww9EacP+dllAz+Pf59tj38+h988A9YTp78L3k3xEe3GepdrwJRAZhLRrbdUGEII25QU0IyaKMGdHbjTMLI5VFJGBm7we9xNMlQVdVoMPQGXA9DBaLCUIEwYH6fFr47jXbDCESVjFzSwW9Ub8RVNZWB0ZFlN7p6dEY8LWGwmvpXGHHVo5Kxhs6I6yRD9aj0dkMPQ/WoeDNzScZodkMvHQyXX5PVuYXvCpaOXLHGaDCM0mEOI2M32L1VpUN4VH3SozodI57qH0Kkox81Tb04VN+N7a5OrK1LYEVVHMsqY1hW0YalJ9qw5Hgr1lS1YVNdG/a7YqgOtiPc2ilgSLvRLpyPzLXnh6F6VEbJOBUY/Nq8fxZ5R8RNbYYZEBWGUVWZubd6NaXB6JFReCqd1jwqNd4Yfn8kjGHD57z6h4bhaevDOmsCE/ZG8eu1Ify0OIBvz/fi6lkuXDzdjo9PsuLsojqcVWSlVaetT0y24dLpDlwz24nr33DjzmU+/G1rCGsryVBH+WYir5DsX1Mz33xNprGG0aPiDT9dGPw3GLwAokrHqcLg/1PdW30kbgajJ5VWYEgDLlQVBX5DBhjDJ8nDev+k8rkE0dDZjzlH2vCjRT5cPtOBT0+x42MTbfjohDp87mUnfrrYj4ffa8AkgjTvaAtWVcexproNq0kq3iprwaySJvxxYxC3L/Hiypl2nD1eAjuHfv/CKTZ8cbYD03YF4QjydfH1kXTQRoUj+WHoPSpeY4XBS1X7lraYuXt7SjBibSLVEqdInFVVlnurwujty8QaA1Iyhti9PTlSMvoHh9HSPYAd7i7csbye7m47Lpxqw1Wv0OYvCWDK/hbs83ehtXtQwBs+eVJI2SCpPobd1z8o7EZKiTWEe8selchRdcIaimHuwQhuftuFS6YzWKsA9MVZdhQfCcITbFCkwzzWyGU39DB46WHwygWD99NyxmCY5KgyaZEemTBME4y+/mw1pc8xKR/TBKI22ovZR1rxg7ekGrptaQB/29mELc4OREhSBoez1ZmZRyUj8V4lEpcuLmcHEp30HjsSiCdItdJ7bqRr2GVtxKPrfLjmFYdQbZ+cbMXj6zw47gyK3FOgPigShKPBMEbiY4GRBSRX9tYMRi67YQ4jJRJpmhHX5ahGqCm6y+vb+zB+T5T0uw/jSAXNPNiKEpKCpmQ/SVK2LRkNRiZHlXFxpXvbITIGwoiL6+frbkWYVNTGijB+XuwWauwjtO4oduKYjTe0Hr4cCcN8doNhqEBUGEa7od9Py1hS6cZfzqRF8sGgu5NcaxWGUFUKDKOaqmxM4d9IGn6+PICd7iSpIlI7QxlbMqTaFB3A0WH0CFWlwhAJQyUSF45Ma1zAkB4V3WwNjTjuiuAXBOIjZGPOITX24GqyD26SDIZBm60acaOa0sNgydBLB8PQSwfvpX4/1ZDBoqqqXO6teY6KVFUrxRttmdoGw1BrG6qaEpKR6s2CoaVFDHf6sAJkpycp1FFeV1cBOaT8HjsFA4YcVUYyZKzBN4qqqlT3NtTUhvrGFtQ30HUq1+oLNaI+1IByRwhfJltiIfV12XQrlh6k4M4TOKMelVlG3GK0G/lUlVEyjIUmtbZhpqZUGGoqXYI4iffsnbQ6UNXYQ5IzbKqOzFxh6XmRZCgw+gcGtUhcD0OmRZIiFmpPZGINTun8eaMfD6z2YsJ2P6q8DaihNW2nF+vKSDX5Q1h80IuPkpRYxlvxwEoHqu1kN5yn51EZYej3VADJp6r0hSY1Epcw2oSrHIvnKDSpVb+0kqPSwRgSmzgs7AFv7Pnkbp7/og1Pbm1Ed9+Q2OTRgZwUP3OYgsDNBLPE24lYV3pEwpBrG7KuoRhxrdAUE9dwFbm+bMC/MdeBPbVhrC2vxy1vO/HsRg9qXPVweQO4dq5N/MwXX7HiSI27IBi8jDB46WHoMxt6W2wpJArPhpGdvc2C0d2lpEVSWglW5qhInZCaYggDBIPV0tFQt1A5ImAbX4vHNkUIiM6ukAurX8OazTgpVBaDu508L3aDf/iWD1WRpEhScrKSk5Zd3d3So+LsrTDi0qNqbW1TnJhWXPWyAmSOA7trQnAHIlh4wI8tx2lj/bLQ9MwGl/iZj5CUbC8jGAoQFYYKRC8ZvMxg6IGo+23UOJZC7EbBJVi2G0r2NtuIDwmpGKS135vEtfM92O3tEpu8gVTWBptUWf2KAR9W7YSy1MBQjTOEESdpu+ltn1An185340SwQ4GREpnkpHBvuyguSqC5TantxOiidTmqDBA7dlUHEWTDK4I/vss5LeLDwv0uLcpfecAhgIxmxFUYhRhxY9hgyRVrGBOGeo8qU/XLVlOaZKTVql9/pgRLG8p5pqtnOYVb29Unv7eiuh0rqtpxLMxAZF5qs6NTfH9ljVz8+arqBAWIncLz4sBvgFTgTSQZHMhdRVH6i3sasbi8WdwU/J780QTWVzVjxr4wnt8WxJySMHZbm4RDElWuVwXydQKys6o+E4mr7i1t9tIDKhArlpSceSOenfsjIKOpqVww9N0i+hxVpgTLVb8haTfojvbG0rjuDS++T5sYpZhCtQfq3ffYpgZ0EYzmrgF8ba4nK/8kVUYdRdN2/HZ9CNFOmZDkoNH4cxyJW8PteGhtAJdMk14SbyanV74824H5B4KKR9WEK2faNCA7KqXe9/ozsYaLNn3iZof4fX6d9UedZNTPPIyM1klIIGptY7RYw7R1R9Q2Ulkw1BKs8JoISO/g+3h6eyM+OtGKPaSy9AZa2hAJpJuBdGeAXDzNge8t9FGQ6MFF0+xiYz41xY7llW3i76hAOKr+xjwXrl/gRnO8E89sDeKciTKpeDVJwY/fcuHTU2Ra5AqCsLkyJGrZV+mAbK/wZ3WKCPeWNn7cG/J1PvWiFfsrCIjLe9ow9CUKvQngZTGzG8a0iKyH5+ijIiNuLMEKNaWkRdgmHCMD/uU5Lty8ODDCY8oFhO/Ie9aEFPtxEsuq4uSNWUW+6W87GkR/0w8Wyp/71mtulPrjIkdlI+n4zEtSFd21wotoi/SoXj8YxPmT5e8/ttYjNuzKGRkg2074s2obrJZ2H3eJiP2solrc9LoV5XUeITX5PCpeehjGVigzGPq9teQsweoLTUpzQrto25HdIqwaunpSWgm2V5Rg+4RuH1SMONsIjrRfPtQqpGM9Ge/hAoHw9+9lIBwA0muxe8seFUfOz2yLiBvgBwu9BMRKQFwo9cVEwnBjdbOm81/cGcTaExG8S2vBgXrx+/x/ty0mt9UXJiCKDXnVhm3HfQoMn4g1TtS58ItFUlV9jN77Cxvpd5zegmGYpUXM7Aa3GLVrDhJJSC4YZrUNvZrSJINhpHX1cLXQpARwsdQgfrYkgM/SxYcSfSPSJTmBjJdA2FVmwJvsCdpQOwGxCSD8txkIb/y3SF2VemPigl49EFHsBtmcaVZ8ZrpNrEum2YQd4p+/eaETFc6QIiFWfI2AbC33ajmqOrsbEzc58MlJUjq+/qoVW8rcCowMEBWGCsQIQ7XNuY24hMEFMl4cM1lG6zDUe1T6EuyIDkOTPirOPXFm9kLS/+MW+RHrGShcQgSQIL2ebFDYaGvXgPxla1hUHfVAjnraRKyx8EhYbCJDuW+VB4+t85KKkutRWo+868LMnT5YPQxE2pWvzbbRhnuUuoYH83Y58JlpEh4XuiZsorjDRSAEkIxknI4R1/ZVgyGDWIuZeys6RXLAUFt3smGM7KNSE3+VDb1ig+5ZHUJH76AM7EYB8lUdkMFBmTTcaI1rKouBsJTepAD5xlwnDrlahNu4g1xbDuL4dV8rCYoik1r1s3oicPnVWCOoGHUJZPMxN6qtHkzZ4hBOgvTManHHIgesTmk3vATDd0bc24RmxBkGJzwZCMdOlmwj3qrAiOeGoeSo1Ka2fqVbxKwezlB2ezrF5vxuHW1iekgDpa6PT7KJu/mxjTJSb+4aJCBu8Tu/IpUlCk19/XivLq5ISB2e3hISrUu/XOET4FgdPUexxrKyBjhDLSQxTvH7n6MNn7u/HmvKglhQEsCP3nTiyfUURNplNP2tudKGMJBNpW68vN2pVBHZc6MbYpkdVbZT96iM2Q2j3ciG0S3ybhbzpjZzjypTgtV5VHlad1g97aGIXBjSpQHEU4MjVNbtywICyKMbGySQZMao/35DWJR62aPaUJuRkKc3h4RHtaYiKjaQDTu7uZ+gO7sh2op5JSFcPNUmvn8uff+zZEPUu/5Ccn+XHvKJtEjRJpcCxIqNFGMUvScBXTG9Ds9vdAivyu3xjgmGPi2iwshlN1QYvLg8wPk3SxaMWEwpwSZyw+jty+owzHdMgDe/tiklNvwLs11oos3mWrn+ZzyxPmH0n9jSKIBEyeaoEvL01oiAwV1962tjuEAB8ufNQSTI22uli1t4tAFfedUh3VNSVdHmNtQ3RLH0aJAkxaF5XBbh3trwZokHNpdf1DZq7D5M3uzEvUvt2HrMSZ/bcfcSticu1DkzMDw+8z4qterHQPTZWzMYaldo3AQGSwcX0AQQ8z4qY7dIT1Znur4Em691h3NPkc4BMurS3dytBIVail3NUZEa6x0cFgZcFprM+qgyTW0dSlObvtBk1tQWiURQ6w2jzBFCtStHU5tDaWqzZxrbRmtqy+fejmY32kfASIqsNPeJcTHNkg9GJkelpEVU97agYwLSVsQVt5fVB5dkk31DWVC0lbeprScnDLNjArKPSt69nDAUrqkuR5VVDycI1VZyCqpcsDpGT4vkK8EWZMQNkqGHwddq4aofq6mY0ili3rqT6aPqH9DBODl6IamP7vxXODAkVcNxwNSSZgoWJTA9ELXqp4chO9NlkUktwSZMYKjSMbKpLZzduiPSItlNbTvLHPjju3b84R0HKqzuMwZDTTcZvdVElhHvETBUIHy9lkwknq+Pqs+0BFtoZ2FlQwrffM0jdPkF5CnNL20TmV2uc4jAb1hp3ekfGAGjW6TS+SI6leNliREw9OkfY2d6IKAcLyOXVauHE5Dt5U48sMKGS6eS4aeb5Wdv2XHC6jntqp8ehmo3RnpUGTWllw62lRZ9bSMbRmrEmQ1hxE+hM51r5Jw+4a5BC8UkXCF8cR9dSJJfc0jEGv0mZzZGnvXji2sfIRlqcjSrM11RU36/bE6ocfhwtMaDRftduGWhDedNzO5qZCAVNu8pwcjnURnthl5NZcPoFddu0cPglekU6R0BQ21qO5XG5wQFhezanjNBbsLZ5BFxgWnR8RiqG7qR6OkT5V69ZGRgyN7b9hxGXA9D7Ux3+UI4VOvHmqNezNjhopjCgc/PtGlpFW2RGv34RK6ZOwW0XH1U+kKTEcZoRjyR5d52m8LQgGQb8S5x8DKrBKuTjKHh4TGpKqM9aesZxGSSDG4BVTfk3ElW/PvrHvzm3SDmHo5itzMOXzO9Jz7upsBIqH1UrJOVenizCqMpimCkCdUUhW+v4gDQj2c3unHXUidumG/HlTOt+NhE+bdGwKD17bk2TNjswt4Kn5AkYx+VseqXK+9nBkNNi3SY2A0Vhl46WENYsjwq5ZiA2i1idG/NjhcXDESpg/cMDIus7zUUl5w1PvtOZZV28TQ7rphhx1fncK+uD/et8uPPm4J4amM9ntwYwJPvBfCnDT489I4Xdy3z4IYFDtH6eflLNlxC9uBTkzMAVAiWIj2QWpKIOvz4DTsW73dT8Ec2xe0XgWK+prZCS7Ca3VBAqDBY2vWqygwGOzSWbPe2x7x154wcoJH18CElnR5JpPHsjgbaSAfOm2QTGyWSghqkWmFvznqhVvm/uhzLqiUTM69h1b7mvNYnJtXiIoL1jVcpyl/nwK7jTjjdHmHkZY7KLyqFp5KjMiZitdDBkKPSAzFTVRoQUdfo7jHto1JrG2fseBkZdw7+ZB/VgKifOKNdmH2wCXevCuDa19xCQs4en32HqxKkLksWjGxAXKr9zDSZMPwRScGvl9sxbYsd7x11oM6ur4f7TrsEq4cxWo6KYykObrnHWC8ZehichrLIs34pYcR7dWc2ciUMz+RZP7Uzndt2GuNJHPPHsL66BXMPNuDpzUHcs9KPnyxy47r5TlEPZ9X0xdl2XEMfv0JfXzffgR++6cCdxU48vNqF5ze5MGe3R3QZcv3iaDW37VC8cQY6DHP1UeljDXMYPZrdMFNTehi8NxbNiBub2v4FMIRHRVLJ74EdCtlHRZ4UGfDG1nbUN7XBE2mDM9QMa30Udb4oar1kwL0RVLnDqHaHxKpzB2H3yMY2j8mZDbsju6lN30dlbN3J19RWiBFX7QbDyBjx1Ag1ZQZDANHXNvqU42WDp+HejgkGHxFQzoiLIQLcCC0O7LeLYJWzCGpnelOU1EVTdMQp2Mz0BAlD60wXkiFzVLma2kbroxqLR9U+Inurh5HKaTP6DYddLVmtO/SfH/Yp2MyZDXWURfYpWP2RZLNI3AhDf/BShaFv9/xw+6iy1dRIu5HfgBthCCD5qn4fziiLXt0xgdzTE0ZLi5jBMB4TUI8k54Khr4efaglWDyRbTfWIazRG4vlg8H5Z/hWjLPLnqMYGQ1UtuWB8WMcE9JKh76PKZcRzwTCzG/p9t+jVVKEwhnUHMbUGaENpVhwTEKqqX6Ts2YMTzdBkN7qUURYdnUmtGXpk9lbpNVbTIlwMEqpKHi/zUdzgNYFhtzEMo2T4zbO3wUyhKSJgNIqORj64E43KczD6IQhcwIvF4uTeKjNblNYdbk6QrVGZyl9PAR6VXjrUfbOcinubXcs4OeJMhzTistDEMNLCvc10pie1MxsMI6EY8bh2TEDmqFTJIBARHYxAZjiY22uUDNcIj0qbK8KdIv6RxwRCWR4VwWhsFn9bwBDjQWSfgdprIA+0KufvtYSshMHXJcaFKEbcTFWZwdAaQ1hlmTYn5IHD6XLu011VkxCHMosr4uLkU99gtkeVTPXjWH0SB30dOOBNoMQTFyvQ3KHlqCItCey0NmNnXTO210axraYRW6sbsIXW5sowNtEKR7Ilg48KbD/hg9XpE5G23ohXW11YfdCJ7eX0tTM7lV7jDGBjuR9l9nrxWgxjW2U9Np4I0gphU0VY/M0tVRFsp79f7WvCbmsjjrijsIda4WuKYW+kE3vCSbh42AC9/8OeNux1tsHWkBBGvJbc8f2VHtTQRwZjFvgNGNql9DCEUS9UOlQpmHW4VRxL/tQUG86bZMX5L9px+QwHntnehGSa6xrDFMcMwtncg6/P5ePGDtEkzY3Pl0xz4AuzHJhV0oDmWAIHXC2iGeHiqVb5kSLsi7K+tokjycFQSKTTy+0B8X98iGZ9qVv02ernihysdIkUybVzbNhwzJ2ph/vqBYwvvGzDjJ1eUUVkNfXNuXbxdy5S1sVT7eL3+Qz7yrIGLD0WwdySEB5e68dRTyueqkvh4pJ+/KY6jSAFsm+VNonMwu1LfbA3duJIrQ+fuesl3PDEApRaA6KbU9jovmzJMALRe6KWQlUV1zSmH2gRlb+fFvtFfTycICkI9eCBdWGRGHxicwN6+weFx2Zt6sJlL9nx08VeHPS24wBJx+qKKL7/pgeffcmGdyqjCEXjdBdGsauuCWuPR3ApAfjaHDu2VtGdWhnE5oqgrupXj2c3ukSj3CfpRpi02SFKrnojzkC40YHTKw+ucpIUyaQhH2deV+YXm/3iNq+mpq4iQJ+dbsW26jB2VJNk1DRgZ20j9pBkeMItqKtvgS3YigBJRz2916Z4J75d2o9P7BvEHG8v2klV3bvSLxos/ro9jLbOHkxdvhcf+9kE3DdlDZpiHdJ29A/kNOLGOV6WQu0Gnxvn6Qk/eMsv5orobUhb9wAe3BAWPbAHfJ1kMwZgbZRA7l0dEKeZ2Iiz3agLx0Q295nN9WhobhOjLDjwc1AkfvkMG2543ZFVglWb2qppc786247vv27D7Yvs+OECG47VZHtUB6vcos5y0RSrOKi54rBHM+LrywISyHaf5lExEJYGNuKq3Whu1o2VIiPO+SnOHIiRhGQzDtON9qXDA7j60AD2NfXAHe0UTd9cUlhV2Sq67++Zshofv3Uypq0sISnJraqMMERysVAgc8henDfZLmyHUZVxVL/D1SFaRn+/PiSaIeoaOgmIA/es8itpEWk3vE1xfGe+U6iBYFOLdiTZXt8kUujXv+4UHlVQsRvqAZqF+z3iSMDb+1yYvs0pUu3rDjsUGDIKP1zlwUcJyO1vO3DTAjupNrs4K8gStr48A0Q14nxgh88ZirSIYsQznZvy2J4xLRKj9aY/hUsODOK7xwbgaU9hszVGqtiJz5MqLw10iBO833lsAS74xVS8e6BWhBSFwBD1kEKB/GFTRGRhg4m+7FhjWJZgw/TGriAJ4j5bjv5rIx24jCThntV+7RQsX1ipt5XuJhsmbA+Kpjbp3jbD5m8UQL5LEhIyDAdzkNv6f5Y7cfXLVlJTHrx31IWvzLLidyvt0qNyyuztoWqfODX7+zUuvL7PgwterMNT690C7IbyegFkyg6/FmswEO56LD4SQnFpGEtKG7C8LIJDrua8aZFoRzceqUvjnH1D+M+qPrSSqzttfyPZ1Dp89w0PgrEerNlXjU/fMRVfuH+2OOFrZsSNMETFsFAgD2+QXeV8DnCkeztEb7IXV9IF8hgMdm9rwx1CZfFRgZf2RTB9bxjPbQ2KTf/Kq3byqhqyjpdZfQ0aENG6o5srsv2EF98k28K1DAZQWefCnYvtuHBKLaqsfGbDK7pJDtd6xfSFR99xiUObdxRLo73pOBl1WheR0zCVgIjAjxarKwb4WXI65LIJB6WIbpZcJViRWSAVvKmhFxeVDOKyg4PY15ImjZAk2+jEBaQuV1a2IZ7swdcemkf2ZCKmLN9vCmPQpMumYCBPbWsUbTzO1rSmpoRnMDgojJanpVvYmO8u8AiXrzqUoDfoEHfg995043tvuISuvW+VF+sqImiMqn1UMvCrI1dRBSKMuALDRdIxa6dLtIL+6V07ZmyT6/a3ZRfjtK1Ocn99IvhjL4frIY+964afXOQj1iBJlQ2/XOpC8eF64aFN3RlQovBmoa4up5umlFzbY54oyjzNKPe2wN0QGzGZojOZicTt8R7ceqIfH9s7hMnufoTp64c3hIRxf5BUdkO8myDsw3lkR37yTDHCrR2mqkpNoehHDhYM5K3jMXzyRRvmHW3VzmwIGAMy8FtV1SZU0dNbwsInrwq2ExA7biv2oirQhgp/C31sRaipJXv2regwbEStRwEy36GoKpkWqbJ78Z90p7OR/ve5dfjOXCu+M8+G62idP5m+nmdHnVPGG0fq/ALIowRE9lGFMW+/nySV1Nsat7h7p+0KoEmZRiFtiENnxDOTUzmVo6kpMc5WZnDb6M5/zpHGeeRp/bZ2ANFkGgtKW0gdkqPxtg/1sV5sOerA5+97Bf92/yzU+qJa5lwvGdrIQTW/1yv73woGEuwYwJVktNh41UZT4kCnfMF++Fu7cdtSv2hsriQQbMQrgnHh3t690qukRZQhL0o6ItPuKT0qrmsIIK/ZhREXaRG3H/so0OLeqfHv2XGkyiWKTqU1bpTVunH/ClZbdVh12Cuk6ahVAnlsrVvELmGyRVZfGHeShPBxOJZwAUS07ShAaMkclYQhTzONzFHJymoKmyK9OJ9g3FhOWqEjjWoKFD8300nq2oFq8iwjrQlc94cF+OTtL2LXcY+wsbmMuJAMcoC4JsRq3htpLhwIq6k1Ne2i0Y3PC8453IISbwcWH2/F9xd6xWHMOYeatLRIBUkFq6y7V3jlKAtu92QvRte6I3pjFfeWe28ZyHUERD8C6fn3KJgkY/7OIUPCkGzGxmMe0dDwyDtOca681BbQAYmIoJIj/XfL6bVnyBO503fXS/dWGRzAwWcxGfPiY3ItKWvEiuNROCLtBhjdsMVSuPRAP644NIgtTX1o6kjhutfJ+yPJW1Tehg4C9sBLa3EO2Y3Jy/YilR7QJlaYwlBS8twLV1LhxjW/nj02ICx6G20J/MeKenx9nhtfnE3ezhwnflbsw8LSKGKJbm32LQ8H+/kSL57bFhilqa1BeEG17nrcSu7qb1Y4MvOoyHu6d6kNj6y2obyWYYycffvzxQ7yqpyodAZo1eNGskHTd3gVGNK99YYa8dwWH25c4MTio2EBg9/Tfyzz4Eaybd8Tyy0X2bufLPJgp601k07ns++02VNdfbitoh+z/f1IpNIUd7SJYxYTdjeihVRXmT2I259fhifnb6VgMp45SWaEIZKtMr8Xp71avO04vvibV2G5eXxhQFQYbDfYgDeRR3WsvhMlngQO+drhiWaSbOrsW56eUE02wxGii8/b1CZrGx66ww/W+ISnlMlRubGr3EHxhVLbMEmlHyM1ddRKxt/PQ/BDOGwNocYTzm5qo+UKNqHcLSNwcTCJgFQHWoRtq6T3yXauur4NNaE4akPtiMY6dfXwFJI9KYQ70wglKUJPpUX2uinRiwZa7d2yfBHrTFF03on2rpRmN9QEYpZ726fC6ELRol247K4ZOOvmojEAUWAMKLUNYYjS6cwoC26ETiY1GOqEz5a2WFZnulmHoZoWkSXY7GZozt7KDG5GTbndhkltXGgKhLSEYYjUXzjcOLIE20yrRVfbiMXlWRg24nF9o3lSXIts3ZEwukX2tlcxvunMNFXai341ChdHweVgHXXakR5G9kTttBi4efekVTj3lkkEg0CMGwOQXMPBZNNXakQJNqaMnc3dmZ55zka+Afb6YcRqc4LZpLbTOusXz3hUZiVYfWd6oVW/Ee6tAoOTjTzMrcodwbi/LMJZ48ZLCPo11np49qQ282MC+nq4CkM/blXACI6EYTap7cOsh+dr3VFBjKUEqweSNfhfVxPaVmrHjU8sxNm0+ZqaGgsQswM0+mborBJsu7lkGDvTJYx6LdZwm7bunDkYZke9zTsM9X1UPQX1UZkVmcxyVB0Uv7y24Qiu/j+zaOOLBIyzxo0BiD7Xwvn8kTPT1edsJKWqMpmZbgZDf2ZD1CsMMLKb2lQ15VXqGurxMn0fVVjzqBoEjEbZmR5Vpiew3ciaQx9XHijTISZTyKnbycx0iqyHBCjH+NK9o8IwAlEzGPw7re1JPPvmdnz6jikjARQCJFcflb51R5MMk9advDBoI120qVYvN7bJqh9X96rsHliVs36cQKxyeFFNEXi1KwC7JhkBCvTII9OMeBjOYCP8ZMTDoh4uD1/6G1vgbGyFv6lV8ahkJN5MK9yWQIRWUzwDI06ruZNuMKXHOdbVi46e3ozd6Mtd9TNTU2pjBxengtE4Hp39Hj760wliw03VVD4gufqoVH2qtu6M9miHXDC85J4utzXh4fJ2HHXUw0ZSsLC6AU+UteK4laSEYo336oL4bVk7Hi+P48njcSy1NgnJqPaGcH95BxZYabPp9U4EGvHgiSRWuWKaqtpD7uvjVV345YkUfl3Rg42BTi0tUhLpxEPVvXjamsIzNnLdm2SD3omWHjzvSMOT6EGQgr2JFG9UxNLZDYQGGPnS6f1KnMHjZTkuOVsYbxXEGIDklgz9I4FIMpJJ0W3BRZtcD8syg8F630XxRlFlDGfvGcTfKMovtftw3aFufKGkF4dtPjGPan5Vo/h6bnUzllkbsdcZIfc2iGO+CCz0ezcc6aX4pwl/qOjEx/cOYHxtp3Rx6e/eVNqLO4+nsNLbgYeqUvja4T5UNclJpMv9Xbi4ZACL/d346fE+PF7XK67HE+/GbSf6MN2Txnx/moK/Afg60yOMuFkJNltNDQjJ4EOy7+6vwbW/n69IRJHhYwFA8ncY9ma5twll9u1oHYbZsYbso3J6AyiqaMOVtOGX7u/D+BMtuII+v/5QF47Y5NCw+dVNuLwkjV+XJfBURRyH3DIdz0A+umcAnz/Qh0cqunDVwT5ccyiNCXWd4u/WRlpxwf5BFHs6RN7MStHyp+nr1YGkuHmW+5O4kIC86unBjcf68aQtLVs+6dqWB9P4/KEBfKt0AMvCfaLXeaxGnCejMoyFW8pw2d0zhGQI433zC8oaAxCz5zNlPKpUdoehyYNP8h281De1OSiwe+FEK+4uTeBGgnDx/l48fCyGmw8ncdTuF0Ber4kKIL9RgBx2yzODZf4GXLivH78nybhofz8eqezC9472CiD89+sibbigpB9LPFJNWaPtAsiaejnNemWgG+fuG8RXj/Tju8f6hGRwWoSbvdvp4w1l/bixbACJVF9BTW1GjypOdmjGqgP42C0TZbCnSAWvc2+djE/+/EUBZlQguRJf+kHEGY8q90MWTU/B1oeyYg22Gc+TVPzqaByra4KYdLwZS6ojEohDJhUZyL8dSGNmdSveqGvGFmdUvFY5AbloP6kcRwxzbQlUBlvxA1JRE61JrR7+w2O9+MXxXqzysb0glXWkDzXNncKjWlnfjSsODmB9uAfXkYQsre/NHOGjO/vemgGx0qM0tclIXIUxJOIMWyCKB2eu11STWOzWKsHfj//8Nv40b1MmIs+1cj8SqDdrXrrq3sZyPGRRgxEZCUOfvbWTR1VMAGacaEC1XXYY7rYFMLGiGVUumT7ZbI/gd+UJPHy8A7+n9bq1TXhU1vomkookjgRatA7DySQd73gTSnNCKw6EEvhTbQr3VqTxWzLgW4JdMvjrkA0KfyE1FUx0422CMcXdJzK0KWUwwsJgHy15kCiXZGgwlHRSmmzG4doAfvJsMXlSE7M2l+FcdOdUfPm3c/Dvj76OxdvKzGMP/dJ7BtlPvNSNsjBRU2ZGXEhGWN/uaf7kshpybasYBrd8utyw0//VuelnhHtbDzctmy8Imz8MOy13UKbSGbYnHEWoIdPuWU+ubSSqdIookXikNY5Aa4Jc3E4lRyUfndHe2am4t91op2trTcopFephpY7ePrH6lByVGYz3dYlWrgntq/TiC/fPwkd+MsGwudJe3Pq3Ymw7asUrq/fjBGmAs0aLQ/TN0GbHBKR7m9+jMjPixmfBZj8SKJOjcuoO0HiUeVTapDYO/LT+W4rCIzKDy9nbJiVhKGsbKozsqp/6NNEET7dOJrXaRrdyhI9VlZhNr0zFSyuNbf0mfVRDOveWYXD2d9G246JMK9SUIS/FkqAa8fPJdvxq8iqU1nrx0Ix3R0hS1tJ3pmc/frQrZ45q1MeP5oXx4eeojMcEsnNUqZxHknPmqIYzh1W5gTzcksDzb+/CJ26bbIAwHh+/ZRJu/sti3DtlNb7+4Dwt7mAw1/x6FhZtKcWlv5yebWsUcJf+8iVYRs4VyX4WLF9YvICE4ak8fvTDgqH2UWUPecmcZjImDFUg+QM/CYQLT/dNfUeUaI1B3vk/n4KiRTvx14Xb8fxb2/HewVpc+8hryoaPF+uzd79E6m28lCjNLS7CV8nOLNlaCsvIx4+qc0U6NRi5RlmcyceP6mHo51GpI5AKPUBjNsoiM49q5CnY0dxb1W6wmrKSJ/XNh18TaZCzTKLuG//4pmi4/tIDr+Jccn0fm72BoNRIGOOkJJx9c0aVqevbj87HrmNW+fwQ/TPE9UNejO7tmXj86KlIRiHDwfJNajObnlDQmY1hpdjEs7z6+rG51IFL9KpmnC4Von5OH59buA1vbjwqvKsvPTBbdFNaRtQ95GuwLbnpyYWooIBYnqeMwSLd2x5FOroEEGHEC3jiZSGPH7XrniZgfPyo2bjV0WCc2iiLnpxqKnfCUJauO3vSeG1DKT5378ua6lHtBXtWnMGVqXQJ5FNkwGes3IfJxbvFx7snrjQx3kW48BdT8cgra+Gpb1A8RAVIBob6QPfOnENexvrES6N0FPL4UePYPOOk7dGGvOhhZNRUKmfVL2cjNM9m6egWDQsX3Tkt466OU430eNxFm128vRzjyIhzh6Ja42A3eMLinbjtb8WmHtUVv3oJL63Yg0C4SQGhLgKiJgxZMrQnl7XL00zqBpwpGKczHEwPQz+PSp0SNHJSmzo9QZ5o0sPgZe5RKQ+aISD+xrjY6HOEvch4SpZxGc/obnJlF24qxeKt5Zj9zgFcLMAViRzWeeSBnT1ugiFQfAEX/8c0rNpVLhKhKgT9sqh2Y2yT2swfsvj/i3srpcO8BJv3eBm7tRTslTsjuOGJN5Wad5HOS+LPJ2jG/LxbJ2HfCRd+8cJybD5ch0NkL254fIEOWgYGOwJf/d0cHKl2yevgpkEDDA2I6lHlgpHrSHK+J16eCgzjPKpcY/PyTfjUl2DHAoOlom9gEOsP2fCNh18bkTL/CN3tNzzxhmgP1ewIrZkUgS/dcRxX3fcKrn/8DREEGqNxjlfumrAcVRSpZ6soEwnJPBIoYXgwr3mhKR8M4zPEzUZZ5HrI4mgPy9LPMMw95EUdf9STNT2hkLN+ia4Upq08gMsoTshOkReJTsR7Jq/G1GV7Rc+uPti7/g9vYN66wyIgPGtEEapIRPJ/fm0Tat1BA4CRMAQQfQk2H4yQkqPy13+4jx81PrvECCPfIGKz8Ue5YOgzt83tXXh09ka6kw09UspiW7K2pIakYza+/6eFZKyXaHEI91V9meIOtfaR8cSKxCmq19YeQLAharr5pkDiBY3NyyQMcz222ti6M5Y+qlOd1JYr1ii0j4phBJrbRZpDbOg4abSNRaRP3TEVCzaWYs2eSryzr0oUn3ido6uTSwgviI8cibMkbTpYLer4uaTBFIiwG3lKsIUnDP9FRjznXBFzuyEPXfbL0SGq3VDS5geq/bjhjwt1Bnu8ElWPx+W/mklGeK7WzHYlxSGbyHD/9c3tuPfF1Xh2wVaKOaYYclLSeP/oqbew42id6HjhJ4kWCkPaELrA1gJLsP8vm9rYbpgHfl15Zxhq0sEFOGHEh9Dd24fiHRW4Rm1w1tUvONj7wZNvYdqK/XhnfzVue26ptulfemAO1h+oxeNzNop+3EwuarwG7v6pq1Fa68lyRApdfI0Wzu20FOhR5WpqG+sTL/M9ZHFsk9oydkMPQ51HpVdTAgbFGT0EY+aag0qwZygWjXsBn7vnZdGgcNNTbwtJmLn6gK6oVCTiCDXlnkm7F4nAcFLxTgr2GsekovSLSwIWnlGSrwT7YcIwe3ZJIUY8mcydo1KlIwvGwKDSnd4jIm+RbTVtOCgSLurU5fvwCkHjbO06kohfkYfFIDL2IpNCYdvDwwLmvnsAjc2tOrd27ECYheUf//iHKOjk6qM6HRiFTmozez5TzEQyzCZD55pHxXZDjiocFOcxKt0NuKNoRd6+KHWzOaZYs7cKLy7dgx//5W08vWALvv3IfNp8nRc1Ttqbbz40D0u3laGRruNUIKg5LC4xMwvLP//5T1EhYwnJNxwsH4zRHj+qAjFKhvFhWcYnl+UeDtZVwHAwqabYeO8sd4nIe2SZ1SwLK8Fwmn3ToVrlIE1RJoGoqCmWjFv+Wox9xx1544qCoNA1s6PBLCygf0yGg8N8MESDggGGXjpGG5tnpqbGMsOw0EHEmqpSDPg7+2vJRZ0pNvOsm4tG6Rx8IZMmIWlgw32uGvAJ6ZAtPB8hT+rBGWth94bkBKO2ttMCwq1IzID/CSD873/+539EfTrfpLZ/5XCw9pyp9FTu4WAKDG5YSyR7xGgLi9KgNmqT86iSIxenRp6iyLs+0qRVUU9VTamq6u9//0DFkAGiQuGYhLsEvR8CDLOnwRlhmOWo9DCMEz717q3oNqfljrThsVc34dxbJ2le0OkBkevz971Mxr5ETKCQM7ROT02xZPz973/XI8gGwv8++OAfok+JT8eeDgzjU2gKkYzcTxNQx+b15i3BsmRUextEg/O5t0w2yS2d2mJ1d/3jC7C+pFobZiZWa+spwWCpYJuhqqm8QPgfG5cPPvhAnAvh3ifedD0Mp8tzypKhf+ZV/hJsUvOmOBIvZIbh3goPrr5/1og6xCmDUGD86M9v4SgFe6LNSHE8xhz0tSdEnMGuLYPgPTb7938BhWWtdubpXywAAAAASUVORK5CYII= mediatype: image/png install: spec: clusterPermissions: - rules: - apiGroups: - rbac.authorization.k8s.io resources: - clusterroles - clusterrolebindings verbs: - '*' - apiGroups: - storage.k8s.io resources: - csidrivers verbs: - '*' - apiGroups: - pmem-csi.intel.com resources: - pmemcsideployments - pmemcsideployments/status - pmemcsideployments/finalizers verbs: - '*' - apiGroups: - admissionregistration.k8s.io resources: - mutatingwebhookconfigurations verbs: - '*' serviceAccountName: pmem-csi-operator deployments: - name: pmem-csi-operator spec: replicas: 1 selector: matchLabels: name: pmem-csi-operator strategy: {} template: metadata: labels: app: pmem-csi-operator name: pmem-csi-operator pmem-csi.intel.com/webhook: ignore spec: containers: - command: - /usr/local/bin/pmem-csi-operator - -metrics-addr=:8080 - -v=3 env: - name: WATCH_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.annotations['olm.targetNamespaces'] - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: OPERATOR_NAME value: pmem-csi-operator - name: GODEBUG value: x509ignoreCN=0 image: intel/pmem-csi-driver:v1.0.1 imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 5 httpGet: path: /metrics port: metrics scheme: HTTP periodSeconds: 10 successThreshold: 1 timeoutSeconds: 5 name: pmem-csi-operator ports: - containerPort: 8080 name: metrics resources: {} securityContext: readOnlyRootFilesystem: true volumeMounts: - mountPath: /tmp name: tmp serviceAccountName: pmem-csi-operator volumes: - emptyDir: {} name: tmp permissions: - rules: - apiGroups: - "" resources: - events verbs: - '*' - apiGroups: - "" resources: - configmaps - services - services/finalizers - serviceaccounts - endpoints - events - secrets - pods verbs: - '*' - apiGroups: - apps resources: - daemonsets - statefulsets - deployments verbs: - '*' - apiGroups: - rbac.authorization.k8s.io resources: - roles - rolebindings verbs: - '*' - apiGroups: - "" resources: - pods - secrets verbs: - get serviceAccountName: pmem-csi-operator strategy: deployment installModes: - supported: true type: OwnNamespace - supported: true type: SingleNamespace - supported: false type: MultiNamespace - supported: false type: AllNamespaces keywords: - PMEM-CSI - PMEM-CSI operator - Persistent Memory - PMEM - Intel links: - name: PMEM-CSI url: https://intel.github.io/pmem-csi/1.0 maintainers: - email: olev.kartau@intel.com name: Olev Kartau - email: patrick.ohly@intel.com name: Patrick Ohly - email: amarnath.valluri@intel.com name: Amarnath Valluri maturity: alpha minKubeVersion: 1.18.0 provider: name: Intel® Corporation replaces: pmem-csi-operator.v1.0.0 version: 1.0.1 pohly@pohly-desktop:/nvme/gopath/src/github.com/intel/pmem-csi/deploy/olm-bundle$ cat 1.0.1/manifests/pmem-csi-operator-metrics_v1_service.yaml apiVersion: v1 kind: Service metadata: creationTimestamp: null name: pmem-csi-operator-metrics spec: ports: - port: 8080 targetPort: 8080 selector: name: pmem-csi-operator status: loadBalancer: {} pohly@pohly-desktop:/nvme/gopath/src/github.com/intel/pmem-csi/deploy/olm-bundle$ cat 1.0.1/manifests/pmem-csi-operator_v1_serviceaccount.yaml apiVersion: v1 kind: ServiceAccount metadata: creationTimestamp: null name: pmem-csi-operator pohly@pohly-desktop:/nvme/gopath/src/github.com/intel/pmem-csi/deploy/olm-bundle$ cat 1.0.1/metadata/annotations.yaml annotations: # Core bundle annotations. operators.operatorframework.io.bundle.mediatype.v1: registry+v1 operators.operatorframework.io.bundle.manifests.v1: manifests/ operators.operatorframework.io.bundle.metadata.v1: metadata/ operators.operatorframework.io.bundle.package.v1: pmem-csi-operator operators.operatorframework.io.bundle.channels.v1: alpha,stable operators.operatorframework.io.bundle.channel.default.v1: stable operators.operatorframework.io.metrics.mediatype.v1: metrics+v1 operators.operatorframework.io.metrics.builder: operator-sdk-v1.6.1+git operators.operatorframework.io.metrics.project_layout: # Annotations for testing. ```

I pushed the image to docker.io/pohly/pmem-csi-bundle:v1.0.1.

camilamacedo86 commented 2 years ago

Hi @pohly,

Would possible to just add the zip/dir with the bundle content? Otherwise, we need to copy and paste and manually generate it to try to test and see if we can help you out.

pohly commented 2 years ago

Would possible to just add the zip/dir with the bundle content? Otherwise, we need to copy and paste and manually generate it to try to test and see if we can help you out.

Do you still need that when the generated image is available (see docker.io/pohly/pmem-csi-bundle:v1.0.1)?

But I can of course also attach the original bundle files: bundle-1.0.1.tar.gz

openshift-bot commented 2 years ago

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close. Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

pohly commented 2 years ago

/remove-lifecycle stale

@camilamacedo86 is there anything further that I can do to investigate this?

We've not had less luck lately in our periodic CI runs with OLM 0.18.3 because of new failure (operatorhubio-catalog-lbdrz 0/1 CrashLoopBackOff). This didn't occur earlier. We would update to a new version, but this issue here is a blocker.

kevinrizza commented 2 years ago

Was a subscription actually created? I'm not super familiar with run bundle, but I assume that it's basically generating a catalog source at runtime and then creating a subscription pointing to the catalog it generated. if no installplan was generated, it could be a resolution error or a problem getting content from that catalog.

pohly commented 2 years ago

Was a subscription actually created?

I don't know. How do I check?

Our CI jobs capture the output of all pods, perhaps that would help? Unfortunately the ones with the more recent OLM expired. Let me kick one off once more...

kevinrizza commented 2 years ago

I don't know. How do I check?

The only real way to install an operator is to create a Subscription resource -- that's the entrypoint API used to install an operator with OLM. I'm not an sdk developer (I work on OLM), but I am making the assumption that this run bundle error is happening because that Subscription is failing. So, if we look at the status of the subscription, it may give us a better idea of why the install is failing.

camilamacedo86 commented 2 years ago

They provide the bundle files here: https://github.com/operator-framework/operator-sdk/issues/5410#issuecomment-983007339

@rashmigottipati @jmrodri could we try to check this one? I think we need to try to reproduce the issue and check the subscription status to know what is wrong here.

WDYT about we add this one in a milestone to be checked with the latest release?

pohly commented 2 years ago

I tried with operator-sdk 1.19.1 and OLM 0.20.0, with Kubernetes 1.19 and 1.22. The "install plan is not available for the subscription pmem-csi-operator-v100-0-0-sub" occurred for both.

Attached is the log output from https://cloudnative-k8sci.southcentralus.cloudapp.azure.com/view/pmem-csi/job/pmem-csi/view/change-requests/job/PR-1071/2/artifact/joblog-jenkins-pmem-csi-PR-1071-2-test-1.22.log

Beware that some output gets dumped repeatedly, for example:

operatorhubio-catalog-6rbbt/registry-server@pmem..ker1: time="2022-04-14T17:05:17Z" level=info msg="serving registry" configs=/configs port=50051
operatorhubio-catalog-6rbbt/registry-server@pmem..ker1: time="2022-04-14T17:05:18Z" level=info msg="shutting down..." configs=/configs port=50051

pmem-csi.txt

pohly commented 2 years ago

I also tried with operator-sdk 1.18.0 and OLM 0.20.0. That worked once (Kubernetes 1.22) and failed once (1.19). It seems to be a bit random, but usually it fails reliably.

pohly commented 2 years ago

Update: I've been successful when installing OLM 0.20.0 on a fresh test cluster. The failure only seems to occur when the cluster has been in use for a while, i.e. several other tests not involving OLM ran earlier.

J0zi commented 2 years ago

We have the same issue with operator-sdk run bundle quay.io/operatorhubio/hive-operator:v2.5.3508-6cb94c6

camilamacedo86 commented 2 years ago

I could also reproduce with: (SDK master branch)

Following the steps

$ operator-sdk run bundle quay.io/operatorhubio/hive-operator:v2.5.3508-6cb94c6
INFO[0014] Successfully created registry pod: quay-io-operatorhubio-hive-operator-v2-5-3508-6cb94c6 
INFO[0014] Created CatalogSource: hive-operator-catalog 
INFO[0014] OperatorGroup "operator-sdk-og" created      
INFO[0014] Created Subscription: hive-operator-v2-5-3508-6cb94c6-sub 
FATA[0120] Failed to run bundle: install plan is not available for the subscription hive-operator-v2-5-3508-6cb94c6-sub: timed out waiting for the condition 

And then, by checking the bundle logs: (kubectl logs pod/quay-io-operatorhubio-hive-operator-v2-5-3508-6cb94c6)

$ kubectl logs pod/quay-io-operatorhubio-hive-operator-v2-5-3508-6cb94c6 
time="2022-05-11T00:46:00Z" level=warning msg="\x1b[1;33mDEPRECATION NOTICE:\nSqlite-based catalogs and their related subcommands are deprecated. Support for\nthem will be removed in a future release. Please migrate your catalog workflows\nto the new file-based catalog format.\x1b[0m"
time="2022-05-11T00:46:00Z" level=info msg="adding to the registry" bundles="[quay.io/operatorhubio/hive-operator:v2.5.3508-6cb94c6]"
time="2022-05-11T00:46:01Z" level=info msg="Could not find optional dependencies file" file=bundle_tmp1603466453/metadata load=annotations with=./bundle_tmp1603466453
time="2022-05-11T00:46:01Z" level=info msg="Could not find optional properties file" file=bundle_tmp1603466453/metadata load=annotations with=./bundle_tmp1603466453
time="2022-05-11T00:46:01Z" level=info msg="Could not find optional dependencies file" file=bundle_tmp1603466453/metadata load=annotations with=./bundle_tmp1603466453
time="2022-05-11T00:46:01Z" level=info msg="Could not find optional properties file" file=bundle_tmp1603466453/metadata load=annotations with=./bundle_tmp1603466453
time="2022-05-11T00:46:01Z" level=error msg="permissive mode disabled" bundles="[quay.io/operatorhubio/hive-operator:v2.5.3508-6cb94c6]" error="error loading bundle into db: FOREIGN KEY constraint failed"
Error: error loading bundle into db: FOREIGN KEY constraint failed
Usage:
  opm registry add [flags]

Also, we found the same above issue by using the operator-sdk run bundle-upgrade, see: https://github.com/k8s-operatorhub/community-operators/runs/6364587418?check_suite_focus=true#step:3:7120 (More info: https://github.com/k8s-operatorhub/community-operators/issues/1195 )

It seems to be an issue associated with OPM.

camilamacedo86 commented 2 years ago

We could find a way to reproduce the issue with OPM and without SDK. So, we raise an issue for we get it fixed in OPM: https://github.com/operator-framework/operator-registry/issues/952

camilamacedo86 commented 2 years ago

This issue shows to be the scenario clarified and tracked via: https://github.com/operator-framework/operator-sdk/issues/5773. To avoid duplication and centralize the info, it seems like we can close this one in favour of https://github.com/operator-framework/operator-sdk/issues/5773.

Note that some workarounds were also proposed in the issue: https://github.com/operator-framework/operator-sdk/issues/5773. Please, ensure that you check if the proposed workarounds can help you out. If you check that your problem is not the same scenario, we would like to ask for you re-open this issue.

Thank you for your attention and collaboration.

pohly commented 2 years ago

That is not the root cause of the failure that I ran into with PMEM-CSI. If I install OLM on a fresh test cluster, running the bundle works. If I do the same thing after the cluster has been in use for a while, it fails for the same bundle.

I can't tell from the log files (see my earlier comments) what might be going wrong. As I have a workaround (run OLM tests first), I am not going to reopen this issue unless it pops up again.