search

percona / everest

Percona Everest is a cloud-native database platform to deploy and manage enterprise-grade PostgreSQL, MongoDB and MySQL database clusters.
https://docs.percona.com/everest/index.html
Apache License 2.0
258 stars 12 forks source link

Stuck on Installing Percona OLM Catalog #192

Open thecodeassassin opened 7 months ago

thecodeassassin commented 7 months ago

Hi all,

everestctl install is stuck on this step:

verestctl install --namespaces db-prod,db-dev --operator.mongodb=true --operator.postgresql=true --operator.xtradb-cluster=true --skip-wizard 2024-04-13T14:28:16Z info install/install.go:430 Installing Operator Lifecycle Manager {"component": "install"} 2024-04-13T14:28:16Z info kubernetes/kubernetes.go:328 OLM operator is already installed {"component": "install", "component": "kubernetes"} 2024-04-13T14:28:16Z info install/install.go:435 OLM has been installed {"component": "install"} 2024-04-13T14:28:16Z info install/install.go:436 Installing Percona OLM Catalog {"component": "install"}

it fails somewhere here:

func (k *Kubernetes) InstallPerconaCatalog(ctx context.Context, version *goversion.Version) error {

But with no errors.

mayankshah1607 commented 7 months ago

Hi @thecodeassassin, thanks for reporting!

Could you please also provide additional information, such as:

fideloper commented 4 months ago

I have a similar issue today, I can't seem to install it on a fresh EKS cluster:

cli

everestctl version
ProjectName: everestctl
Version: v1.0.1
FullCommit: 40b9e1c074f75b1452143102dcf062d91bb8ab93

cluster

status

k logs -f -n everest-olm olm-operator-6f7b945cbb-rjcdd

# outputs this multiple times per second forever:

time="2024-07-08T18:55:05Z" level=info msg="install strategy successful" csv=packageserver id=uGZ4N namespace=everest-olm phase=Installing strategy=deployment

and:

k logs -f packageserver-6b766dcf95-nc7lz -n everest-olm

# has grpc errors since I believe the pod above isn't running yet:

time="2024-07-08T18:53:43Z" level=info msg="updating PackageManifest based on CatalogSource changes: {everest-catalog everest-olm}" action="sync catalogsource" address="everest-catalog.everest-olm.svc:50051" name=everest-catalog namespace=everest-olm
W0708 18:54:02.897521       1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "everest-catalog.everest-olm.svc:50051", ServerName: "everest-catalog.everest-olm.svc:50051", }. Err: connection error: desc = "transport: Error while dialing: dial tcp [fda1:d88a:dc1::b912]:50051: i/o timeout"
time="2024-07-08T18:54:02Z" level=warning msg="error getting bundle stream" action="refresh cache" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp [fda1:d88a:dc1::b912]:50051: i/o timeout\"" source="{everest-catalog everest-olm}"
time="2024-07-08T18:54:02Z" level=info msg="updating PackageManifest based on CatalogSource changes: {everest-catalog everest-olm}" action="sync catalogsource" address="everest-catalog.everest-olm.svc:50051" name=everest-catalog namespace=everest-olm
time="2024-07-08T18:54:02Z" level=warning msg="error getting bundle stream" action="refresh cache" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp [fda1:d88a:dc1::b912]:50051: i/o timeout\"" source="{everest-catalog everest-olm}"
time="2024-07-08T18:54:03Z" level=info msg="updating PackageManifest based on CatalogSource changes: {everest-catalog everest-olm}" action="sync catalogsource" address="everest-catalog.everest-olm.svc:50051" name=everest-catalog namespace=everest-olm
time="2024-07-08T18:54:03Z" level=warning msg="error getting bundle stream" action="refresh cache" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp [fda1:d88a:dc1::b912]:50051: i/o timeout\"" source="{everest-catalog everest-olm}"
time="2024-07-08T18:54:04Z" level=info msg="updating PackageManifest based on CatalogSource changes: {everest-catalog everest-olm}" action="sync catalogsource" address="everest-catalog.everest-olm.svc:50051" name=everest-catalog namespace=everest-olm
time="2024-07-08T19:07:32Z" level=info msg="updating PackageManifest based on CatalogSource changes: {everest-catalog everest-olm}" action="sync catalogsource" address="everest-catalog.everest-olm.svc:50051" name=everest-catalog namespace=everest-olm
fideloper commented 4 months ago

This is occuring within an ipv4 EKS cluster as well for me.

mayankshah1607 commented 4 months ago

@fideloper from the package server logs, it seems to me that it cannot reach the catalog server. Can you check the status of the everest-catalog-xx pod? You can also describe the catalog source object for additional clues - kubectl describe catalogsource everest-catalog -n everest-olm

fideloper commented 4 months ago

I've opened up all ports, altho the error being "connection refused" sounds like nothing is listening there rather than there being a security group in the way (which would cause a timeout typically).

Some more info:

image

everestctl

#> everestctl -v install

2024-07-09T07:19:22.943-0400    DEBUG   commands/root.go:34     Debug logging enabled
? Namespaces managed by Everest [comma separated] everest,percona
? What operators do you want to install? MySQL, PostgreSQL
2024-07-09T07:19:31.661-0400    DEBUG   install/install.go:552  Enabling MySQL operator {"component": "install"}
2024-07-09T07:19:31.661-0400    DEBUG   install/install.go:552  Enabling PostgreSQL operator    {"component": "install"}
2024-07-09T07:19:31.852-0400    DEBUG   install/install.go:214  Everest latest version available: 1.0.1 {"component": "install"}
2024-07-09T07:19:31.852-0400    DEBUG   install/install.go:215  Everest version information &version.MetadataVersion{state:impl.MessageState{NoUnkeyedLiterals:pragma.NoUnkeyedLiterals{}, DoNotCompare:pragma.DoNotCompare{}, DoNotCopy:pragma.DoNotCopy{}, atomicMessageInfo:(*impl.MessageInfo)(nil)}, sizeCache:0, unknownFields:[]uint8(nil), Version:"1.0.1", Recommended:map[string]string{}, Supported:map[string]string{"cli":">= 1.0.0"}}   {"component": "install"}
2024-07-09T07:19:31.852-0400    INFO    install/install.go:580  Installing Operator Lifecycle Manager   {"component": "install"}
2024-07-09T07:19:31.902-0400    DEBUG   kubernetes/kubernetes.go:430    Applying "crds/olm/crds.yaml" file      {"component": "install", "component": "kubernetes"}
2024-07-09T07:19:34.821-0400    DEBUG   kubernetes/kubernetes.go:430    Applying "crds/olm/olm.yaml" file       {"component": "install", "component": "kubernetes"}
2024-07-09T07:19:39.779-0400    INFO    install/install.go:585  OLM has been installed  {"component": "install"}
2024-07-09T07:19:39.779-0400    INFO    install/install.go:586  Installing Percona OLM Catalog  {"component": "install"}
2024-07-09T07:19:39.780-0400    DEBUG   kubernetes/kubernetes.go:398    Using catalog image docker.io/percona/everest-catalog:1.0.1     {"component": "install", "component": "kubernetes"}

## This is where it just sits/waits

catalog-operator

catalog operator logs ``` time="2024-07-09T11:06:28Z" level=info msg="log level info" W0709 11:06:28.822007 1 client_config.go:618] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. time="2024-07-09T11:06:28Z" level=info msg="Using in-cluster kube client config" time="2024-07-09T11:06:28Z" level=info msg="Using in-cluster kube client config" W0709 11:06:28.822662 1 client_config.go:618] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. time="2024-07-09T11:06:28Z" level=info msg="skipping irrelevant gvr" gvr="rbac.authorization.k8s.io/v1, Resource=clusterrolebindings" time="2024-07-09T11:06:28Z" level=info msg="skipping irrelevant gvr" gvr="rbac.authorization.k8s.io/v1, Resource=clusterroles" time="2024-07-09T11:06:28Z" level=info msg="skipping irrelevant gvr" gvr="apps/v1, Resource=deployments" time="2024-07-09T11:06:28Z" level=info msg="detected ability to filter informers" canFilter=true time="2024-07-09T11:06:28Z" level=info msg="registering owner reference fixer" gvr="/v1, Resource=services" W0709 11:06:28.879504 1 client_config.go:618] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. time="2024-07-09T11:06:28Z" level=info msg="connection established. cluster-version: v1.30.2-eks-db838b0" time="2024-07-09T11:06:28Z" level=info msg="operator ready" time="2024-07-09T11:06:28Z" level=info msg="starting informers..." time="2024-07-09T11:06:28Z" level=info msg="informers started" time="2024-07-09T11:06:28Z" level=info msg="waiting for caches to sync..." time="2024-07-09T11:06:28Z" level=info msg="starting workers..." time="2024-07-09T11:06:28Z" level=info msg="resolving sources" id=dBSfr namespace=default time="2024-07-09T11:06:28Z" level=info msg="checking if subscriptions need update" id=dBSfr namespace=default time="2024-07-09T11:06:28Z" level=info msg="resolving sources" id=sIOme namespace=everest-olm time="2024-07-09T11:06:28Z" level=info msg="checking if subscriptions need update" id=sIOme namespace=everest-olm time="2024-07-09T11:06:28Z" level=info msg="connection established. cluster-version: v1.30.2-eks-db838b0" time="2024-07-09T11:06:28Z" level=info msg="operator ready" time="2024-07-09T11:06:28Z" level=info msg="starting informers..." time="2024-07-09T11:06:28Z" level=info msg="informers started" time="2024-07-09T11:06:28Z" level=info msg="waiting for caches to sync..." time="2024-07-09T11:06:29Z" level=info msg="No subscriptions were found in namespace everest-olm" id=sIOme namespace=everest-olm time="2024-07-09T11:06:29Z" level=info msg="resolving sources" id=mTObF namespace=kube-node-lease time="2024-07-09T11:06:29Z" level=info msg="checking if subscriptions need update" id=mTObF namespace=kube-node-lease time="2024-07-09T11:06:29Z" level=info msg="No subscriptions were found in namespace kube-node-lease" id=mTObF namespace=kube-node-lease time="2024-07-09T11:06:29Z" level=info msg="resolving sources" id=Af7Lz namespace=kube-public time="2024-07-09T11:06:29Z" level=info msg="checking if subscriptions need update" id=Af7Lz namespace=kube-public time="2024-07-09T11:06:29Z" level=info msg="No subscriptions were found in namespace kube-public" id=Af7Lz namespace=kube-public time="2024-07-09T11:06:29Z" level=info msg="resolving sources" id=k7qHT namespace=kube-system time="2024-07-09T11:06:29Z" level=info msg="checking if subscriptions need update" id=k7qHT namespace=kube-system time="2024-07-09T11:06:29Z" level=info msg="No subscriptions were found in namespace default" id=dBSfr namespace=default time="2024-07-09T11:06:29Z" level=info msg="No subscriptions were found in namespace kube-system" id=k7qHT namespace=kube-system time="2024-07-09T11:06:29Z" level=info msg="starting workers..." time="2024-07-09T11:06:29Z" level=info msg="resolving sources" id=lpBY7 namespace=everest-olm time="2024-07-09T11:06:29Z" level=info msg="checking if subscriptions need update" id=lpBY7 namespace=everest-olm time="2024-07-09T11:06:29Z" level=info msg="No subscriptions were found in namespace everest-olm" id=lpBY7 namespace=everest-olm time="2024-07-09T11:06:29Z" level=info msg="resolving sources" id=FWc4g namespace=everest-olm time="2024-07-09T11:06:29Z" level=info msg="checking if subscriptions need update" id=FWc4g namespace=everest-olm time="2024-07-09T11:06:29Z" level=info msg="No subscriptions were found in namespace everest-olm" id=FWc4g namespace=everest-olm time="2024-07-09T11:06:29Z" level=info msg="resolving sources" id=ppQL1 namespace=everest-olm time="2024-07-09T11:06:29Z" level=info msg="checking if subscriptions need update" id=ppQL1 namespace=everest-olm time="2024-07-09T11:06:29Z" level=info msg="resolving sources" id=6p8Zn namespace=everest-olm time="2024-07-09T11:06:29Z" level=info msg="checking if subscriptions need update" id=6p8Zn namespace=everest-olm time="2024-07-09T11:06:29Z" level=info msg="No subscriptions were found in namespace everest-olm" id=ppQL1 namespace=everest-olm time="2024-07-09T11:06:29Z" level=info msg="No subscriptions were found in namespace everest-olm" id=6p8Zn namespace=everest-olm time="2024-07-09T11:06:29Z" level=info msg="resolving sources" id=HXy3Y namespace=everest-olm time="2024-07-09T11:06:29Z" level=info msg="checking if subscriptions need update" id=HXy3Y namespace=everest-olm time="2024-07-09T11:06:29Z" level=info msg="No subscriptions were found in namespace everest-olm" id=HXy3Y namespace=everest-olm time="2024-07-09T11:06:29Z" level=info msg="resolving sources" id=G7ri2 namespace=everest-olm time="2024-07-09T11:06:29Z" level=info msg="checking if subscriptions need update" id=G7ri2 namespace=everest-olm time="2024-07-09T11:06:29Z" level=info msg="No subscriptions were found in namespace everest-olm" id=G7ri2 namespace=everest-olm time="2024-07-09T11:06:41Z" level=info msg="syncing catalog source" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=bWAk7 time="2024-07-09T11:06:41Z" level=info msg="synchronizing registry server" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=bWAk7 time="2024-07-09T11:06:41Z" level=info msg="checked registry server health" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm health=false id=bWAk7 time="2024-07-09T11:06:41Z" level=info msg="ensuring registry server" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=bWAk7 time="2024-07-09T11:06:41Z" level=info msg="registry service status invalid, need to overwrite" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=bWAk7 time="2024-07-09T11:06:41Z" level=info msg="searching for current pods" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=bWAk7 time="2024-07-09T11:06:41Z" level=info msg="of 0 pods matching label selector, 0 have the correct images and matching hash" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=bWAk7 time="2024-07-09T11:06:41Z" level=info msg="registry pods invalid, need to overwrite" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=bWAk7 time="2024-07-09T11:06:41Z" level=info msg="creating desired pod" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=bWAk7 pod.name= pod.namespace=everest-olm time="2024-07-09T11:06:41Z" level=info msg="ensured registry server" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=bWAk7 time="2024-07-09T11:06:41Z" level=info msg="requeuing registry server sync based on polling interval 45m0s" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=bWAk7 time="2024-07-09T11:06:41Z" level=info msg="syncing catalog source" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=FD2Hp time="2024-07-09T11:06:41Z" level=info msg="synchronizing registry server" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=FD2Hp time="2024-07-09T11:06:41Z" level=info msg="searching for current pods" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=FD2Hp time="2024-07-09T11:06:41Z" level=info msg="evaluating current pod" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm correctHash=true correctImages=true current-pod.name=everest-catalog-7pmp2 current-pod.namespace=everest-olm id=FD2Hp time="2024-07-09T11:06:41Z" level=info msg="of 1 pods matching label selector, 1 have the correct images and matching hash" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm correctHash=true correctImages=true current-pod.name=everest-catalog-7pmp2 current-pod.namespace=everest-olm id=FD2Hp time="2024-07-09T11:06:41Z" level=info msg="checked registry server health" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm health=true id=FD2Hp time="2024-07-09T11:06:41Z" level=info msg="registry state good" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=FD2Hp time="2024-07-09T11:06:41Z" level=info msg="ensuring registry server" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=FD2Hp time="2024-07-09T11:06:41Z" level=info msg="searching for current pods" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=FD2Hp time="2024-07-09T11:06:41Z" level=info msg="evaluating current pod" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm correctHash=true correctImages=true current-pod.name=everest-catalog-7pmp2 current-pod.namespace=everest-olm id=FD2Hp time="2024-07-09T11:06:41Z" level=info msg="of 1 pods matching label selector, 1 have the correct images and matching hash" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm correctHash=true correctImages=true current-pod.name=everest-catalog-7pmp2 current-pod.namespace=everest-olm id=FD2Hp time="2024-07-09T11:06:41Z" level=info msg="ensured registry server" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=FD2Hp time="2024-07-09T11:06:41Z" level=info msg="requeuing registry server sync based on polling interval 45m0s" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=FD2Hp time="2024-07-09T11:06:41Z" level=info msg="state.Key.Namespace=everest-olm state.Key.Name=everest-catalog state.State=CONNECTING" time="2024-07-09T11:06:41Z" level=info msg="state.Key.Namespace=everest-olm state.Key.Name=everest-catalog state.State=TRANSIENT_FAILURE" time="2024-07-09T11:06:41Z" level=info msg="syncing catalog source" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=W2OxN time="2024-07-09T11:06:41Z" level=info msg="synchronizing registry server" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=W2OxN time="2024-07-09T11:06:41Z" level=info msg="searching for current pods" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=W2OxN time="2024-07-09T11:06:41Z" level=info msg="evaluating current pod" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm correctHash=true correctImages=true current-pod.name=everest-catalog-7pmp2 current-pod.namespace=everest-olm id=W2OxN time="2024-07-09T11:06:41Z" level=info msg="of 1 pods matching label selector, 1 have the correct images and matching hash" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm correctHash=true correctImages=true current-pod.name=everest-catalog-7pmp2 current-pod.namespace=everest-olm id=W2OxN time="2024-07-09T11:06:41Z" level=info msg="checked registry server health" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm health=true id=W2OxN time="2024-07-09T11:06:41Z" level=info msg="registry state good" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=W2OxN time="2024-07-09T11:06:41Z" level=info msg="ensuring registry server" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=W2OxN time="2024-07-09T11:06:41Z" level=info msg="searching for current pods" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=W2OxN time="2024-07-09T11:06:41Z" level=info msg="evaluating current pod" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm correctHash=true correctImages=true current-pod.name=everest-catalog-7pmp2 current-pod.namespace=everest-olm id=W2OxN time="2024-07-09T11:06:41Z" level=info msg="of 1 pods matching label selector, 1 have the correct images and matching hash" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm correctHash=true correctImages=true current-pod.name=everest-catalog-7pmp2 current-pod.namespace=everest-olm id=W2OxN time="2024-07-09T11:06:41Z" level=info msg="ensured registry server" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=W2OxN time="2024-07-09T11:06:41Z" level=info msg="requeuing registry server sync based on polling interval 45m0s" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=W2OxN time="2024-07-09T11:06:41Z" level=error msg="UpdateStatus - error while setting CatalogSource status" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm error="Operation cannot be fulfilled on catalogsources.operators.coreos.com \"everest-catalog\": the object has been modified; please apply your changes to the latest version and try again" id=W2OxN E0709 11:06:41.551736 1 queueinformer_operator.go:319] sync {"update" "everest-olm/everest-catalog"} failed: Operation cannot be fulfilled on catalogsources.operators.coreos.com "everest-catalog": the object has been modified; please apply your changes to the latest version and try again time="2024-07-09T11:06:41Z" level=info msg="syncing catalog source" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=pHbqd time="2024-07-09T11:06:41Z" level=info msg="synchronizing registry server" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=pHbqd time="2024-07-09T11:06:41Z" level=info msg="searching for current pods" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=pHbqd time="2024-07-09T11:06:41Z" level=info msg="evaluating current pod" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm correctHash=true correctImages=true current-pod.name=everest-catalog-7pmp2 current-pod.namespace=everest-olm id=pHbqd time="2024-07-09T11:06:41Z" level=info msg="of 1 pods matching label selector, 1 have the correct images and matching hash" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm correctHash=true correctImages=true current-pod.name=everest-catalog-7pmp2 current-pod.namespace=everest-olm id=pHbqd time="2024-07-09T11:06:41Z" level=info msg="checked registry server health" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm health=true id=pHbqd time="2024-07-09T11:06:41Z" level=info msg="registry state good" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=pHbqd time="2024-07-09T11:06:41Z" level=info msg="ensuring registry server" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=pHbqd time="2024-07-09T11:06:42Z" level=info msg="searching for current pods" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=pHbqd time="2024-07-09T11:06:42Z" level=info msg="evaluating current pod" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm correctHash=true correctImages=true current-pod.name=everest-catalog-7pmp2 current-pod.namespace=everest-olm id=pHbqd time="2024-07-09T11:06:42Z" level=info msg="of 1 pods matching label selector, 1 have the correct images and matching hash" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm correctHash=true correctImages=true current-pod.name=everest-catalog-7pmp2 current-pod.namespace=everest-olm id=pHbqd time="2024-07-09T11:06:42Z" level=info msg="ensured registry server" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=pHbqd time="2024-07-09T11:06:42Z" level=info msg="requeuing registry server sync based on polling interval 45m0s" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=pHbqd time="2024-07-09T11:06:42Z" level=info msg="syncing catalog source" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=tA3pc time="2024-07-09T11:06:42Z" level=info msg="synchronizing registry server" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=tA3pc time="2024-07-09T11:06:42Z" level=info msg="searching for current pods" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=tA3pc time="2024-07-09T11:06:42Z" level=info msg="evaluating current pod" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm correctHash=true correctImages=true current-pod.name=everest-catalog-7pmp2 current-pod.namespace=everest-olm id=tA3pc time="2024-07-09T11:06:42Z" level=info msg="of 1 pods matching label selector, 1 have the correct images and matching hash" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm correctHash=true correctImages=true current-pod.name=everest-catalog-7pmp2 current-pod.namespace=everest-olm id=tA3pc time="2024-07-09T11:06:42Z" level=info msg="checked registry server health" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm health=true id=tA3pc time="2024-07-09T11:06:42Z" level=info msg="registry state good" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=tA3pc time="2024-07-09T11:06:42Z" level=info msg="ensuring registry server" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=tA3pc time="2024-07-09T11:06:42Z" level=info msg="searching for current pods" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=tA3pc time="2024-07-09T11:06:42Z" level=info msg="evaluating current pod" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm correctHash=true correctImages=true current-pod.name=everest-catalog-7pmp2 current-pod.namespace=everest-olm id=tA3pc time="2024-07-09T11:06:42Z" level=info msg="of 1 pods matching label selector, 1 have the correct images and matching hash" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm correctHash=true correctImages=true current-pod.name=everest-catalog-7pmp2 current-pod.namespace=everest-olm id=tA3pc time="2024-07-09T11:06:42Z" level=info msg="ensured registry server" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=tA3pc time="2024-07-09T11:06:42Z" level=info msg="requeuing registry server sync based on polling interval 45m0s" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=tA3pc time="2024-07-09T11:06:42Z" level=info msg="syncing catalog source" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=edjWM time="2024-07-09T11:06:42Z" level=info msg="synchronizing registry server" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=edjWM time="2024-07-09T11:06:42Z" level=info msg="searching for current pods" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=edjWM time="2024-07-09T11:06:42Z" level=info msg="evaluating current pod" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm correctHash=true correctImages=true current-pod.name=everest-catalog-7pmp2 current-pod.namespace=everest-olm id=edjWM time="2024-07-09T11:06:42Z" level=info msg="of 1 pods matching label selector, 1 have the correct images and matching hash" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm correctHash=true correctImages=true current-pod.name=everest-catalog-7pmp2 current-pod.namespace=everest-olm id=edjWM time="2024-07-09T11:06:42Z" level=info msg="checked registry server health" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm health=true id=edjWM time="2024-07-09T11:06:42Z" level=info msg="registry state good" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=edjWM time="2024-07-09T11:06:42Z" level=info msg="ensuring registry server" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=edjWM time="2024-07-09T11:06:43Z" level=info msg="searching for current pods" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=edjWM time="2024-07-09T11:06:43Z" level=info msg="evaluating current pod" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm correctHash=true correctImages=true current-pod.name=everest-catalog-7pmp2 current-pod.namespace=everest-olm id=edjWM time="2024-07-09T11:06:43Z" level=info msg="of 1 pods matching label selector, 1 have the correct images and matching hash" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm correctHash=true correctImages=true current-pod.name=everest-catalog-7pmp2 current-pod.namespace=everest-olm id=edjWM time="2024-07-09T11:06:43Z" level=info msg="ensured registry server" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=edjWM time="2024-07-09T11:06:43Z" level=info msg="requeuing registry server sync based on polling interval 45m0s" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=edjWM time="2024-07-09T11:06:43Z" level=info msg="syncing catalog source" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=3e0Mt time="2024-07-09T11:06:43Z" level=info msg="synchronizing registry server" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=3e0Mt time="2024-07-09T11:06:43Z" level=info msg="searching for current pods" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=3e0Mt time="2024-07-09T11:06:43Z" level=info msg="evaluating current pod" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm correctHash=true correctImages=true current-pod.name=everest-catalog-7pmp2 current-pod.namespace=everest-olm id=3e0Mt time="2024-07-09T11:06:43Z" level=info msg="of 1 pods matching label selector, 1 have the correct images and matching hash" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm correctHash=true correctImages=true current-pod.name=everest-catalog-7pmp2 current-pod.namespace=everest-olm id=3e0Mt time="2024-07-09T11:06:43Z" level=info msg="checked registry server health" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm health=true id=3e0Mt time="2024-07-09T11:06:43Z" level=info msg="registry state good" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=3e0Mt time="2024-07-09T11:06:43Z" level=info msg="ensuring registry server" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=3e0Mt time="2024-07-09T11:06:43Z" level=info msg="searching for current pods" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=3e0Mt time="2024-07-09T11:06:43Z" level=info msg="evaluating current pod" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm correctHash=true correctImages=true current-pod.name=everest-catalog-7pmp2 current-pod.namespace=everest-olm id=3e0Mt time="2024-07-09T11:06:43Z" level=info msg="of 1 pods matching label selector, 1 have the correct images and matching hash" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm correctHash=true correctImages=true current-pod.name=everest-catalog-7pmp2 current-pod.namespace=everest-olm id=3e0Mt time="2024-07-09T11:06:43Z" level=info msg="ensured registry server" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=3e0Mt time="2024-07-09T11:06:43Z" level=info msg="requeuing registry server sync based on polling interval 45m0s" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=3e0Mt time="2024-07-09T11:06:51Z" level=info msg="syncing catalog source" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=1TXEH time="2024-07-09T11:06:51Z" level=info msg="synchronizing registry server" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=1TXEH time="2024-07-09T11:06:51Z" level=info msg="searching for current pods" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=1TXEH time="2024-07-09T11:06:51Z" level=info msg="evaluating current pod" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm correctHash=true correctImages=true current-pod.name=everest-catalog-7pmp2 current-pod.namespace=everest-olm id=1TXEH time="2024-07-09T11:06:51Z" level=info msg="of 1 pods matching label selector, 1 have the correct images and matching hash" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm correctHash=true correctImages=true current-pod.name=everest-catalog-7pmp2 current-pod.namespace=everest-olm id=1TXEH time="2024-07-09T11:06:51Z" level=info msg="checked registry server health" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm health=true id=1TXEH time="2024-07-09T11:06:51Z" level=info msg="registry state good" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=1TXEH time="2024-07-09T11:06:51Z" level=info msg="ensuring registry server" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=1TXEH time="2024-07-09T11:06:51Z" level=info msg="searching for current pods" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=1TXEH time="2024-07-09T11:06:51Z" level=info msg="evaluating current pod" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm correctHash=true correctImages=true current-pod.name=everest-catalog-7pmp2 current-pod.namespace=everest-olm id=1TXEH time="2024-07-09T11:06:51Z" level=info msg="of 1 pods matching label selector, 1 have the correct images and matching hash" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm correctHash=true correctImages=true current-pod.name=everest-catalog-7pmp2 current-pod.namespace=everest-olm id=1TXEH time="2024-07-09T11:06:51Z" level=info msg="ensured registry server" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=1TXEH time="2024-07-09T11:06:51Z" level=info msg="requeuing registry server sync based on polling interval 45m0s" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=1TXEH time="2024-07-09T11:06:52Z" level=info msg="syncing catalog source" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=l/Im9 time="2024-07-09T11:06:52Z" level=info msg="synchronizing registry server" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=l/Im9 time="2024-07-09T11:06:52Z" level=info msg="searching for current pods" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=l/Im9 time="2024-07-09T11:06:52Z" level=info msg="evaluating current pod" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm correctHash=true correctImages=true current-pod.name=everest-catalog-7pmp2 current-pod.namespace=everest-olm id=l/Im9 time="2024-07-09T11:06:52Z" level=info msg="of 1 pods matching label selector, 1 have the correct images and matching hash" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm correctHash=true correctImages=true current-pod.name=everest-catalog-7pmp2 current-pod.namespace=everest-olm id=l/Im9 time="2024-07-09T11:06:52Z" level=info msg="checked registry server health" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm health=true id=l/Im9 time="2024-07-09T11:06:52Z" level=info msg="registry state good" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=l/Im9 time="2024-07-09T11:06:52Z" level=info msg="ensuring registry server" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=l/Im9 time="2024-07-09T11:06:52Z" level=info msg="searching for current pods" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=l/Im9 time="2024-07-09T11:06:52Z" level=info msg="evaluating current pod" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm correctHash=true correctImages=true current-pod.name=everest-catalog-7pmp2 current-pod.namespace=everest-olm id=l/Im9 time="2024-07-09T11:06:52Z" level=info msg="of 1 pods matching label selector, 1 have the correct images and matching hash" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm correctHash=true correctImages=true current-pod.name=everest-catalog-7pmp2 current-pod.namespace=everest-olm id=l/Im9 time="2024-07-09T11:06:52Z" level=info msg="ensured registry server" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=l/Im9 time="2024-07-09T11:06:52Z" level=info msg="requeuing registry server sync based on polling interval 45m0s" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=l/Im9 time="2024-07-09T11:06:55Z" level=info msg="state.Key.Namespace=everest-olm state.Key.Name=everest-catalog state.State=READY" time="2024-07-09T11:06:55Z" level=info msg="syncing catalog source" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=bi2pj time="2024-07-09T11:06:55Z" level=info msg="synchronizing registry server" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=bi2pj time="2024-07-09T11:06:55Z" level=info msg="resolving sources" id=J8M3K namespace=everest-olm time="2024-07-09T11:06:55Z" level=info msg="checking if subscriptions need update" id=J8M3K namespace=everest-olm time="2024-07-09T11:06:55Z" level=info msg="No subscriptions were found in namespace everest-olm" id=J8M3K namespace=everest-olm time="2024-07-09T11:06:55Z" level=info msg="searching for current pods" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=bi2pj time="2024-07-09T11:06:55Z" level=info msg="evaluating current pod" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm correctHash=true correctImages=true current-pod.name=everest-catalog-7pmp2 current-pod.namespace=everest-olm id=bi2pj time="2024-07-09T11:06:55Z" level=info msg="of 1 pods matching label selector, 1 have the correct images and matching hash" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm correctHash=true correctImages=true current-pod.name=everest-catalog-7pmp2 current-pod.namespace=everest-olm id=bi2pj time="2024-07-09T11:06:55Z" level=info msg="checked registry server health" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm health=true id=bi2pj time="2024-07-09T11:06:55Z" level=info msg="registry state good" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=bi2pj time="2024-07-09T11:06:55Z" level=info msg="ensuring registry server" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=bi2pj time="2024-07-09T11:06:55Z" level=info msg="searching for current pods" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=bi2pj time="2024-07-09T11:06:55Z" level=info msg="evaluating current pod" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm correctHash=true correctImages=true current-pod.name=everest-catalog-7pmp2 current-pod.namespace=everest-olm id=bi2pj time="2024-07-09T11:06:55Z" level=info msg="of 1 pods matching label selector, 1 have the correct images and matching hash" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm correctHash=true correctImages=true current-pod.name=everest-catalog-7pmp2 current-pod.namespace=everest-olm id=bi2pj time="2024-07-09T11:06:55Z" level=info msg="ensured registry server" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=bi2pj time="2024-07-09T11:06:55Z" level=info msg="requeuing registry server sync based on polling interval 45m0s" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=bi2pj time="2024-07-09T11:06:55Z" level=info msg="syncing catalog source" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=dSE4I time="2024-07-09T11:06:55Z" level=info msg="synchronizing registry server" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=dSE4I time="2024-07-09T11:06:55Z" level=info msg="searching for current pods" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=dSE4I time="2024-07-09T11:06:55Z" level=info msg="evaluating current pod" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm correctHash=true correctImages=true current-pod.name=everest-catalog-7pmp2 current-pod.namespace=everest-olm id=dSE4I time="2024-07-09T11:06:55Z" level=info msg="of 1 pods matching label selector, 1 have the correct images and matching hash" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm correctHash=true correctImages=true current-pod.name=everest-catalog-7pmp2 current-pod.namespace=everest-olm id=dSE4I time="2024-07-09T11:06:55Z" level=info msg="checked registry server health" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm health=true id=dSE4I time="2024-07-09T11:06:55Z" level=info msg="registry state good" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=dSE4I time="2024-07-09T11:06:55Z" level=info msg="ensuring registry server" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=dSE4I time="2024-07-09T11:06:55Z" level=info msg="searching for current pods" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=dSE4I time="2024-07-09T11:06:55Z" level=info msg="evaluating current pod" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm correctHash=true correctImages=true current-pod.name=everest-catalog-7pmp2 current-pod.namespace=everest-olm id=dSE4I time="2024-07-09T11:06:55Z" level=info msg="of 1 pods matching label selector, 1 have the correct images and matching hash" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm correctHash=true correctImages=true current-pod.name=everest-catalog-7pmp2 current-pod.namespace=everest-olm id=dSE4I time="2024-07-09T11:06:55Z" level=info msg="ensured registry server" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=dSE4I time="2024-07-09T11:06:55Z" level=info msg="requeuing registry server sync based on polling interval 45m0s" catalogsource.name=everest-catalog catalogsource.namespace=everest-olm id=dSE4I ```

everest-catalog

catalog logs ``` time="2024-07-09T11:06:42Z" level=info msg="starting pprof endpoint" address="localhost:6060" time="2024-07-09T11:06:42Z" level=info msg="found existing cache contents" backend=pogreb.v1 cache=/tmp/cache configs=/configs time="2024-07-09T11:06:43Z" level=info msg="serving registry" cache=/tmp/cache configs=/configs port=50051 time="2024-07-09T11:06:43Z" level=info msg="stopped caching cpu profile data" address="localhost:6060" ```

olm-operato

operator logs ``` time="2024-07-09T11:06:28Z" level=info msg="log level info" time="2024-07-09T11:06:28Z" level=info msg="skipping irrelevant gvr" gvr="/v1, Resource=services" time="2024-07-09T11:06:28Z" level=info msg="skipping irrelevant gvr" gvr="/v1, Resource=configmaps" time="2024-07-09T11:06:28Z" level=info msg="skipping irrelevant gvr" gvr="/v1, Resource=pods" time="2024-07-09T11:06:28Z" level=info msg="skipping irrelevant gvr" gvr="rbac.authorization.k8s.io/v1, Resource=roles" time="2024-07-09T11:06:28Z" level=info msg="skipping irrelevant gvr" gvr="rbac.authorization.k8s.io/v1, Resource=rolebindings" time="2024-07-09T11:06:28Z" level=info msg="skipping irrelevant gvr" gvr="batch/v1, Resource=jobs" time="2024-07-09T11:06:28Z" level=info msg="skipping irrelevant gvr" gvr="/v1, Resource=serviceaccounts" time="2024-07-09T11:06:28Z" level=info msg="detected ability to filter informers" canFilter=true time="2024-07-09T11:06:28Z" level=info msg="connection established. cluster-version: v1.30.2-eks-db838b0" time="2024-07-09T11:06:28Z" level=info msg="operator ready" time="2024-07-09T11:06:28Z" level=info msg="starting informers..." time="2024-07-09T11:06:28Z" level=info msg="informers started" time="2024-07-09T11:06:28Z" level=info msg="waiting for caches to sync..." time="2024-07-09T11:06:28Z" level=info msg="starting workers..." time="2024-07-09T11:06:28Z" level=warning msg="csv in namespace with no operatorgroups" csv=packageserver id=OhcR1 namespace=everest-olm phase=Installing time="2024-07-09T11:06:28Z" level=info msg="operatorgroup incorrect" csv=packageserver error="csv in namespace with no operatorgroups" id=OhcR1 namespace=everest-olm phase=Installing I0709 11:06:28.720224 1 event.go:298] Event(v1.ObjectReference{Kind:"ClusterServiceVersion", Namespace:"everest-olm", Name:"packageserver", UID:"e8d796bf-0500-4ba8-8150-7c12282b328e", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"220891", FieldPath:""}): type: 'Warning' reason: 'NoOperatorGroup' csv in namespace with no operatorgroups {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting EventSource","controller":"operatorcondition","controllerGroup":"operators.coreos.com","controllerKind":"OperatorCondition","source":"kind source: *v2.OperatorCondition"} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting EventSource","controller":"operatorcondition","controllerGroup":"operators.coreos.com","controllerKind":"OperatorCondition","source":"kind source: *v1.Role"} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting EventSource","controller":"operatorcondition","controllerGroup":"operators.coreos.com","controllerKind":"OperatorCondition","source":"kind source: *v1.RoleBinding"} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting EventSource","controller":"operatorcondition","controllerGroup":"operators.coreos.com","controllerKind":"OperatorCondition","source":"kind source: *v1.Deployment"} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting Controller","controller":"operatorcondition","controllerGroup":"operators.coreos.com","controllerKind":"OperatorCondition"} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting EventSource","controller":"operator","controllerGroup":"operators.coreos.com","controllerKind":"Operator","source":"kind source: *v1.Operator"} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting EventSource","controller":"operator","controllerGroup":"operators.coreos.com","controllerKind":"Operator","source":"kind source: *v1.Deployment"} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting EventSource","controller":"operator","controllerGroup":"operators.coreos.com","controllerKind":"Operator","source":"kind source: *v1.Namespace"} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting EventSource","controller":"operator","controllerGroup":"operators.coreos.com","controllerKind":"Operator","source":"kind source: *v1.CustomResourceDefinition"} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting EventSource","controller":"operator","controllerGroup":"operators.coreos.com","controllerKind":"Operator","source":"kind source: *v1.APIService"} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting EventSource","controller":"operator","controllerGroup":"operators.coreos.com","controllerKind":"Operator","source":"kind source: *v1alpha1.Subscription"} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting EventSource","controller":"operator","controllerGroup":"operators.coreos.com","controllerKind":"Operator","source":"kind source: *v1alpha1.InstallPlan"} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting EventSource","controller":"operator","controllerGroup":"operators.coreos.com","controllerKind":"Operator","source":"kind source: *v1alpha1.ClusterServiceVersion"} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting EventSource","controller":"operator","controllerGroup":"operators.coreos.com","controllerKind":"Operator","source":"kind source: *v2.OperatorCondition"} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting EventSource","controller":"operator","controllerGroup":"operators.coreos.com","controllerKind":"Operator","source":"kind source: *v1.PartialObjectMetadata"} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting EventSource","controller":"operator","controllerGroup":"operators.coreos.com","controllerKind":"Operator","source":"kind source: *v1.PartialObjectMetadata"} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting EventSource","controller":"operator","controllerGroup":"operators.coreos.com","controllerKind":"Operator","source":"kind source: *v1.PartialObjectMetadata"} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting EventSource","controller":"operator","controllerGroup":"operators.coreos.com","controllerKind":"Operator","source":"kind source: *v1.PartialObjectMetadata"} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting EventSource","controller":"operator","controllerGroup":"operators.coreos.com","controllerKind":"Operator","source":"kind source: *v1.PartialObjectMetadata"} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting EventSource","controller":"operator","controllerGroup":"operators.coreos.com","controllerKind":"Operator","source":"kind source: *v1.PartialObjectMetadata"} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting EventSource","controller":"operator","controllerGroup":"operators.coreos.com","controllerKind":"Operator","source":"kind source: *v1.PartialObjectMetadata"} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting Controller","controller":"operator","controllerGroup":"operators.coreos.com","controllerKind":"Operator"} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting EventSource","controller":"clusterserviceversion","controllerGroup":"operators.coreos.com","controllerKind":"ClusterServiceVersion","source":"kind source: *v1alpha1.ClusterServiceVersion"} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting EventSource","controller":"clusterserviceversion","controllerGroup":"operators.coreos.com","controllerKind":"ClusterServiceVersion","source":"kind source: *v2.OperatorCondition"} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting Controller","controller":"clusterserviceversion","controllerGroup":"operators.coreos.com","controllerKind":"ClusterServiceVersion"} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting EventSource","controller":"clusterserviceversion","controllerGroup":"operators.coreos.com","controllerKind":"ClusterServiceVersion","source":"kind source: *v1alpha1.ClusterServiceVersion"} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting EventSource","controller":"clusterserviceversion","controllerGroup":"operators.coreos.com","controllerKind":"ClusterServiceVersion","source":"kind source: *v1.Deployment"} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting EventSource","controller":"clusterserviceversion","controllerGroup":"operators.coreos.com","controllerKind":"ClusterServiceVersion","source":"kind source: *v1.Namespace"} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting EventSource","controller":"clusterserviceversion","controllerGroup":"operators.coreos.com","controllerKind":"ClusterServiceVersion","source":"kind source: *v1.Service"} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting EventSource","controller":"clusterserviceversion","controllerGroup":"operators.coreos.com","controllerKind":"ClusterServiceVersion","source":"kind source: *v1.CustomResourceDefinition"} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting EventSource","controller":"clusterserviceversion","controllerGroup":"operators.coreos.com","controllerKind":"ClusterServiceVersion","source":"kind source: *v1.APIService"} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting EventSource","controller":"clusterserviceversion","controllerGroup":"operators.coreos.com","controllerKind":"ClusterServiceVersion","source":"kind source: *v1alpha1.Subscription"} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting EventSource","controller":"clusterserviceversion","controllerGroup":"operators.coreos.com","controllerKind":"ClusterServiceVersion","source":"kind source: *v2.OperatorCondition"} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting EventSource","controller":"clusterserviceversion","controllerGroup":"operators.coreos.com","controllerKind":"ClusterServiceVersion","source":"kind source: *v1.PartialObjectMetadata"} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting EventSource","controller":"clusterserviceversion","controllerGroup":"operators.coreos.com","controllerKind":"ClusterServiceVersion","source":"kind source: *v1.PartialObjectMetadata"} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting EventSource","controller":"clusterserviceversion","controllerGroup":"operators.coreos.com","controllerKind":"ClusterServiceVersion","source":"kind source: *v1.PartialObjectMetadata"} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting EventSource","controller":"clusterserviceversion","controllerGroup":"operators.coreos.com","controllerKind":"ClusterServiceVersion","source":"kind source: *v1.PartialObjectMetadata"} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting EventSource","controller":"clusterserviceversion","controllerGroup":"operators.coreos.com","controllerKind":"ClusterServiceVersion","source":"kind source: *v1.PartialObjectMetadata"} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting EventSource","controller":"clusterserviceversion","controllerGroup":"operators.coreos.com","controllerKind":"ClusterServiceVersion","source":"kind source: *v1.PartialObjectMetadata"} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting EventSource","controller":"clusterserviceversion","controllerGroup":"operators.coreos.com","controllerKind":"ClusterServiceVersion","source":"kind source: *v1.PartialObjectMetadata"} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting Controller","controller":"clusterserviceversion","controllerGroup":"operators.coreos.com","controllerKind":"ClusterServiceVersion"} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting EventSource","controller":"subscription","controllerGroup":"operators.coreos.com","controllerKind":"Subscription","source":"kind source: *v1alpha1.Subscription"} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting EventSource","controller":"subscription","controllerGroup":"operators.coreos.com","controllerKind":"Subscription","source":"kind source: *v1alpha1.ClusterServiceVersion"} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting EventSource","controller":"subscription","controllerGroup":"operators.coreos.com","controllerKind":"Subscription","source":"kind source: *v1alpha1.InstallPlan"} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting Controller","controller":"subscription","controllerGroup":"operators.coreos.com","controllerKind":"Subscription"} time="2024-07-09T11:06:28Z" level=info msg="operatorgroup not found" csv=packageserver id=zWF7p namespace=everest-olm operatorgroup=olm-operators phase=Installing E0709 11:06:28.791945 1 queueinformer_operator.go:319] sync {"update" "everest-olm/packageserver"} failed: csv in namespace with no operatorgroups time="2024-07-09T11:06:28Z" level=warning msg="csv in namespace with no operatorgroups" csv=packageserver id=yJJ8F namespace=everest-olm phase=Failed time="2024-07-09T11:06:28Z" level=info msg="operatorgroup incorrect" csv=packageserver error="csv in namespace with no operatorgroups" id=yJJ8F namespace=everest-olm phase=Failed I0709 11:06:28.792239 1 event.go:298] Event(v1.ObjectReference{Kind:"ClusterServiceVersion", Namespace:"everest-olm", Name:"packageserver", UID:"e8d796bf-0500-4ba8-8150-7c12282b328e", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"222873", FieldPath:""}): type: 'Warning' reason: 'NoOperatorGroup' csv in namespace with no operatorgroups time="2024-07-09T11:06:28Z" level=info msg="operatorgroup not found" csv=packageserver id=TjEWD namespace=everest-olm operatorgroup=olm-operators phase=Failed E0709 11:06:28.813495 1 queueinformer_operator.go:319] sync {"update" "everest-olm/packageserver"} failed: csv in namespace with no operatorgroups time="2024-07-09T11:06:28Z" level=warning msg="csv in namespace with no operatorgroups" csv=packageserver id=esjGB namespace=everest-olm phase=Failed time="2024-07-09T11:06:28Z" level=info msg="operatorgroup incorrect" csv=packageserver error="csv in namespace with no operatorgroups" id=esjGB namespace=everest-olm phase=Failed I0709 11:06:28.813675 1 event.go:298] Event(v1.ObjectReference{Kind:"ClusterServiceVersion", Namespace:"everest-olm", Name:"packageserver", UID:"e8d796bf-0500-4ba8-8150-7c12282b328e", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"222873", FieldPath:""}): type: 'Warning' reason: 'NoOperatorGroup' csv in namespace with no operatorgroups {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting workers","controller":"operatorcondition","controllerGroup":"operators.coreos.com","controllerKind":"OperatorCondition","worker count":1} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting workers","controller":"clusterserviceversion","controllerGroup":"operators.coreos.com","controllerKind":"ClusterServiceVersion","worker count":1} {"level":"info","ts":"2024-07-09T11:06:28Z","logger":"controllers.operatorcondition-generator","msg":"Cleaning up resources after CSV deletion","name":"packageserver","namespace":"everest-olm","selector":"olm.owner=packageserver,olm.owner.kind=ClusterServiceVersion,olm.owner.namespace=everest-olm"} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting workers","controller":"clusterserviceversion","controllerGroup":"operators.coreos.com","controllerKind":"ClusterServiceVersion","worker count":1} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting workers","controller":"subscription","controllerGroup":"operators.coreos.com","controllerKind":"Subscription","worker count":1} {"level":"info","ts":"2024-07-09T11:06:28Z","msg":"Starting workers","controller":"operator","controllerGroup":"operators.coreos.com","controllerKind":"Operator","worker count":1} time="2024-07-09T11:06:28Z" level=info msg="operatorgroup not found" csv=packageserver id=iaksk namespace=everest-olm operatorgroup=olm-operators phase=Failed E0709 11:06:28.827347 1 queueinformer_operator.go:319] sync {"update" "everest-olm/packageserver"} failed: csv in namespace with no operatorgroups time="2024-07-09T11:06:28Z" level=warning msg="csv in namespace with no operatorgroups" csv=packageserver id=+6GEe namespace=everest-olm phase=Failed time="2024-07-09T11:06:28Z" level=info msg="operatorgroup incorrect" csv=packageserver error="csv in namespace with no operatorgroups" id=+6GEe namespace=everest-olm phase=Failed I0709 11:06:28.827583 1 event.go:298] Event(v1.ObjectReference{Kind:"ClusterServiceVersion", Namespace:"everest-olm", Name:"packageserver", UID:"e8d796bf-0500-4ba8-8150-7c12282b328e", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"222873", FieldPath:""}): type: 'Warning' reason: 'NoOperatorGroup' csv in namespace with no operatorgroups time="2024-07-09T11:06:28Z" level=info msg="operatorgroup not found" csv=packageserver id=34FF+ namespace=everest-olm operatorgroup=olm-operators phase=Failed E0709 11:06:28.867365 1 queueinformer_operator.go:319] sync {"update" "everest-olm/packageserver"} failed: csv in namespace with no operatorgroups time="2024-07-09T11:06:28Z" level=warning msg="csv in namespace with no operatorgroups" csv=packageserver id=QqyV1 namespace=everest-olm phase=Failed time="2024-07-09T11:06:28Z" level=info msg="operatorgroup incorrect" csv=packageserver error="csv in namespace with no operatorgroups" id=QqyV1 namespace=everest-olm phase=Failed I0709 11:06:28.867601 1 event.go:298] Event(v1.ObjectReference{Kind:"ClusterServiceVersion", Namespace:"everest-olm", Name:"packageserver", UID:"e8d796bf-0500-4ba8-8150-7c12282b328e", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"222873", FieldPath:""}): type: 'Warning' reason: 'NoOperatorGroup' csv in namespace with no operatorgroups time="2024-07-09T11:06:28Z" level=info msg="operatorgroup not found" csv=packageserver id=t1L5Z namespace=everest-olm operatorgroup=olm-operators phase=Failed E0709 11:06:28.893089 1 queueinformer_operator.go:319] sync {"update" "everest-olm/packageserver"} failed: csv in namespace with no operatorgroups time="2024-07-09T11:06:28Z" level=warning msg="csv in namespace with no operatorgroups" csv=packageserver id=9R0rs namespace=everest-olm phase=Failed time="2024-07-09T11:06:28Z" level=info msg="operatorgroup incorrect" csv=packageserver error="csv in namespace with no operatorgroups" id=9R0rs namespace=everest-olm phase=Failed I0709 11:06:28.907895 1 event.go:298] Event(v1.ObjectReference{Kind:"ClusterServiceVersion", Namespace:"everest-olm", Name:"packageserver", UID:"e8d796bf-0500-4ba8-8150-7c12282b328e", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"222873", FieldPath:""}): type: 'Warning' reason: 'NoOperatorGroup' csv in namespace with no operatorgroups time="2024-07-09T11:06:28Z" level=info msg="operatorgroup not found" csv=packageserver id=vz+J3 namespace=everest-olm operatorgroup=olm-operators phase=Failed E0709 11:06:28.926489 1 queueinformer_operator.go:319] sync {"update" "everest-olm/packageserver"} failed: csv in namespace with no operatorgroups time="2024-07-09T11:06:29Z" level=warning msg="csv in namespace with no operatorgroups" csv=packageserver id=R6j0m namespace=everest-olm phase=Failed time="2024-07-09T11:06:29Z" level=info msg="operatorgroup incorrect" csv=packageserver error="csv in namespace with no operatorgroups" id=R6j0m namespace=everest-olm phase=Failed I0709 11:06:29.087725 1 event.go:298] Event(v1.ObjectReference{Kind:"ClusterServiceVersion", Namespace:"everest-olm", Name:"packageserver", UID:"e8d796bf-0500-4ba8-8150-7c12282b328e", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"222873", FieldPath:""}): type: 'Warning' reason: 'NoOperatorGroup' csv in namespace with no operatorgroups time="2024-07-09T11:06:29Z" level=info msg="operatorgroup not found" csv=packageserver id=/U1vD namespace=everest-olm operatorgroup=olm-operators phase=Failed E0709 11:06:29.106256 1 queueinformer_operator.go:319] sync {"update" "everest-olm/packageserver"} failed: csv in namespace with no operatorgroups time="2024-07-09T11:06:29Z" level=warning msg="csv in namespace with no operatorgroups" csv=packageserver id=xkf4a namespace=everest-olm phase=Failed time="2024-07-09T11:06:29Z" level=info msg="operatorgroup incorrect" csv=packageserver error="csv in namespace with no operatorgroups" id=xkf4a namespace=everest-olm phase=Failed I0709 11:06:29.106478 1 event.go:298] Event(v1.ObjectReference{Kind:"ClusterServiceVersion", Namespace:"everest-olm", Name:"packageserver", UID:"e8d796bf-0500-4ba8-8150-7c12282b328e", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"222885", FieldPath:""}): type: 'Warning' reason: 'NoOperatorGroup' csv in namespace with no operatorgroups time="2024-07-09T11:06:29Z" level=info msg="operatorgroup not found" csv=packageserver id=9D+h8 namespace=everest-olm operatorgroup=olm-operators phase=Failed E0709 11:06:29.133323 1 queueinformer_operator.go:319] sync {"update" "everest-olm/packageserver"} failed: csv in namespace with no operatorgroups {"level":"info","ts":"2024-07-09T11:06:29Z","logger":"controllers.operatorcondition-generator","msg":"Cleaning up resources after CSV deletion","name":"packageserver","namespace":"everest-olm","selector":"olm.owner=packageserver,olm.owner.kind=ClusterServiceVersion,olm.owner.namespace=everest-olm"} time="2024-07-09T11:06:29Z" level=info msg="gcing children" csv=packageserver id=PmBVA namespace=everest-olm phase=Failed time="2024-07-09T11:06:29Z" level=info msg="gcing api service v1.packages.operators.coreos.com" csv=packageserver id=PmBVA namespace=everest-olm phase=Failed time="2024-07-09T11:06:29Z" level=info msg="InstallModes now support target namespaces. Transitioning to Pending..." csv=packageserver id=J0twy namespace=everest-olm phase=Failed I0709 11:06:29.341209 1 event.go:298] Event(v1.ObjectReference{Kind:"ClusterServiceVersion", Namespace:"everest-olm", Name:"packageserver", UID:"e8d796bf-0500-4ba8-8150-7c12282b328e", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"222885", FieldPath:""}): type: 'Normal' reason: 'RequirementsUnknown' InstallModes now support target namespaces time="2024-07-09T11:06:29Z" level=info msg="error updating ClusterServiceVersion status: Operation cannot be fulfilled on clusterserviceversions.operators.coreos.com \"packageserver\": StorageError: invalid object, Code: 4, Key: /registry/operators.coreos.com/clusterserviceversions/everest-olm/packageserver, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: e8d796bf-0500-4ba8-8150-7c12282b328e, UID in object meta: " csv=packageserver id=TbEpU namespace=everest-olm phase=Failed E0709 11:06:29.350346 1 queueinformer_operator.go:319] sync {"update" "everest-olm/packageserver"} failed: error updating ClusterServiceVersion status: Operation cannot be fulfilled on clusterserviceversions.operators.coreos.com "packageserver": StorageError: invalid object, Code: 4, Key: /registry/operators.coreos.com/clusterserviceversions/everest-olm/packageserver, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: e8d796bf-0500-4ba8-8150-7c12282b328e, UID in object meta: {"level":"error","ts":"2024-07-09T11:06:29Z","logger":"controllers.operatorcondition-generator","msg":"Removing finalizer","name":"packageserver","namespace":"everest-olm","error":"clusterserviceversions.operators.coreos.com \"packageserver\" not found","stacktrace":"github.com/operator-framework/operator-lifecycle-manager/pkg/controller/operators.(*OperatorConditionGeneratorReconciler).processFinalizer\n\t/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/pkg/controller/operators/operatorconditiongenerator_controller.go:284\ngithub.com/operator-framework/operator-lifecycle-manager/pkg/controller/operators.(*OperatorConditionGeneratorReconciler).Reconcile\n\t/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/pkg/controller/operators/operatorconditiongenerator_controller.go:103\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"} {"level":"error","ts":"2024-07-09T11:06:29Z","msg":"Reconciler error","controller":"clusterserviceversion","controllerGroup":"operators.coreos.com","controllerKind":"ClusterServiceVersion","ClusterServiceVersion":{"name":"packageserver","namespace":"everest-olm"},"namespace":"everest-olm","name":"packageserver","reconcileID":"29b7e7fc-a679-4869-9fe0-813235ab9e6f","error":"clusterserviceversions.operators.coreos.com \"packageserver\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"} time="2024-07-09T11:06:29Z" level=warning msg="error adding operatorgroup annotations" csv=packageserver error="Operation cannot be fulfilled on clusterserviceversions.operators.coreos.com \"packageserver\": the object has been modified; please apply your changes to the latest version and try again" id=0m7WS namespace=everest-olm opgroup=olm-operators phase= time="2024-07-09T11:06:29Z" level=info msg="operatorgroup incorrect" csv=packageserver error="Operation cannot be fulfilled on clusterserviceversions.operators.coreos.com \"packageserver\": the object has been modified; please apply your changes to the latest version and try again" id=0m7WS namespace=everest-olm phase= time="2024-07-09T11:06:29Z" level=info msg="not part of any operatorgroup, no annotations" csv=packageserver id=07Lxm namespace=everest-olm phase= E0709 11:06:29.556823 1 queueinformer_operator.go:319] sync {"update" "everest-olm/packageserver"} failed: Operation cannot be fulfilled on clusterserviceversions.operators.coreos.com "packageserver": the object has been modified; please apply your changes to the latest version and try again time="2024-07-09T11:06:29Z" level=info msg="operatorgroup incorrect" csv=packageserver error="" id=gj+dr namespace=everest-olm phase= time="2024-07-09T11:06:29Z" level=info msg="not part of any operatorgroup, no annotations" csv=packageserver id=/u6XW namespace=everest-olm phase= {"level":"error","ts":"2024-07-09T11:06:29Z","msg":"Reconciler error","controller":"operatorcondition","controllerGroup":"operators.coreos.com","controllerKind":"OperatorCondition","OperatorCondition":{"name":"packageserver","namespace":"everest-olm"},"namespace":"everest-olm","name":"packageserver","reconcileID":"4d3e5532-ba59-49f5-9722-b86de9b51be4","error":"Deployment.apps \"packageserver\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"} time="2024-07-09T11:06:29Z" level=info msg="scheduling ClusterServiceVersion for requirement verification" csv=packageserver id=4l5to namespace=everest-olm phase= I0709 11:06:29.579230 1 event.go:298] Event(v1.ObjectReference{Kind:"ClusterServiceVersion", Namespace:"everest-olm", Name:"packageserver", UID:"66d35921-db22-4506-b4d9-e55701a3b8a1", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"222913", FieldPath:""}): type: 'Normal' reason: 'RequirementsUnknown' requirements not yet checked {"level":"error","ts":"2024-07-09T11:06:29Z","msg":"Reconciler error","controller":"operatorcondition","controllerGroup":"operators.coreos.com","controllerKind":"OperatorCondition","OperatorCondition":{"name":"packageserver","namespace":"everest-olm"},"namespace":"everest-olm","name":"packageserver","reconcileID":"01963e77-1f63-43e1-8c63-9cb61250aa4b","error":"rolebindings.rbac.authorization.k8s.io \"packageserver\" already exists","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"} {"level":"error","ts":"2024-07-09T11:06:29Z","msg":"Reconciler error","controller":"operatorcondition","controllerGroup":"operators.coreos.com","controllerKind":"OperatorCondition","OperatorCondition":{"name":"packageserver","namespace":"everest-olm"},"namespace":"everest-olm","name":"packageserver","reconcileID":"e64fcc67-ae2e-42fe-85b8-e44e86d2d799","error":"Deployment.apps \"packageserver\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"} {"level":"error","ts":"2024-07-09T11:06:29Z","msg":"Reconciler error","controller":"operatorcondition","controllerGroup":"operators.coreos.com","controllerKind":"OperatorCondition","OperatorCondition":{"name":"packageserver","namespace":"everest-olm"},"namespace":"everest-olm","name":"packageserver","reconcileID":"7e645883-80c3-4c8d-b2f9-f101cae827dc","error":"Deployment.apps \"packageserver\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"} time="2024-07-09T11:06:29Z" level=error msg="update existing cluster role failed: nil" error="Operation cannot be fulfilled on clusterroles.rbac.authorization.k8s.io \"olm.og.olm-operators.view-82gdxvBtMmhgjcWZVmNGIauWAGlyiVPmTAV5sk\": the object has been modified; please apply your changes to the latest version and try again" {"level":"error","ts":"2024-07-09T11:06:29Z","msg":"Reconciler error","controller":"operatorcondition","controllerGroup":"operators.coreos.com","controllerKind":"OperatorCondition","OperatorCondition":{"name":"packageserver","namespace":"everest-olm"},"namespace":"everest-olm","name":"packageserver","reconcileID":"a48ce53b-b0f6-4e33-b164-1451ab06c793","error":"Deployment.apps \"packageserver\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"} time="2024-07-09T11:06:29Z" level=info msg="scheduling ClusterServiceVersion for install" csv=packageserver id=fecU9 namespace=everest-olm phase=Pending I0709 11:06:29.703895 1 event.go:298] Event(v1.ObjectReference{Kind:"ClusterServiceVersion", Namespace:"everest-olm", Name:"packageserver", UID:"66d35921-db22-4506-b4d9-e55701a3b8a1", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"222918", FieldPath:""}): type: 'Normal' reason: 'AllRequirementsMet' all requirements found, attempting install time="2024-07-09T11:06:29Z" level=info msg="scheduling ClusterServiceVersion for install" csv=packageserver id=oE1VR namespace=everest-olm phase=Pending I0709 11:06:29.724990 1 event.go:298] Event(v1.ObjectReference{Kind:"ClusterServiceVersion", Namespace:"everest-olm", Name:"packageserver", UID:"66d35921-db22-4506-b4d9-e55701a3b8a1", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"222918", FieldPath:""}): type: 'Normal' reason: 'AllRequirementsMet' all requirements found, attempting install {"level":"error","ts":"2024-07-09T11:06:29Z","msg":"Reconciler error","controller":"operatorcondition","controllerGroup":"operators.coreos.com","controllerKind":"OperatorCondition","OperatorCondition":{"name":"packageserver","namespace":"everest-olm"},"namespace":"everest-olm","name":"packageserver","reconcileID":"47ce4b56-b791-4257-b54a-48c0ad158d43","error":"Deployment.apps \"packageserver\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"} time="2024-07-09T11:06:29Z" level=info msg="error updating ClusterServiceVersion status: Operation cannot be fulfilled on clusterserviceversions.operators.coreos.com \"packageserver\": the object has been modified; please apply your changes to the latest version and try again" csv=packageserver id=93h0i namespace=everest-olm phase=Pending E0709 11:06:29.733122 1 queueinformer_operator.go:319] sync {"update" "everest-olm/packageserver"} failed: error updating ClusterServiceVersion status: Operation cannot be fulfilled on clusterserviceversions.operators.coreos.com "packageserver": the object has been modified; please apply your changes to the latest version and try again I0709 11:06:29.931982 1 event.go:298] Event(v1.ObjectReference{Kind:"ClusterServiceVersion", Namespace:"everest-olm", Name:"packageserver", UID:"66d35921-db22-4506-b4d9-e55701a3b8a1", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"222923", FieldPath:""}): type: 'Normal' reason: 'InstallSucceeded' waiting for install components to report healthy time="2024-07-09T11:06:29Z" level=info msg="install strategy successful" csv=packageserver id=1xC5l namespace=everest-olm phase=Installing strategy=deployment I0709 11:06:29.984727 1 event.go:298] Event(v1.ObjectReference{Kind:"ClusterServiceVersion", Namespace:"everest-olm", Name:"packageserver", UID:"66d35921-db22-4506-b4d9-e55701a3b8a1", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"222950", FieldPath:""}): type: 'Normal' reason: 'InstallWaiting' apiServices not installed time="2024-07-09T11:06:30Z" level=info msg="install strategy successful" csv=packageserver id=jUSII namespace=everest-olm phase=Installing strategy=deployment time="2024-07-09T11:06:30Z" level=info msg="install strategy successful" csv=packageserver id=EmA5l namespace=everest-olm phase=Installing strategy=deployment time="2024-07-09T11:06:30Z" level=info msg="install strategy successful" csv=packageserver id=fLR5U namespace=everest-olm phase=Installing strategy=deployment time="2024-07-09T11:06:30Z" level=info msg="install strategy successful" csv=packageserver id=Ii6Eu namespace=everest-olm phase=Installing strategy=deployment time="2024-07-09T11:06:30Z" level=info msg="install strategy successful" csv=packageserver id=l6QJ3 namespace=everest-olm phase=Installing strategy=deployment time="2024-07-09T11:06:30Z" level=info msg="install strategy successful" csv=packageserver id=GD27P namespace=everest-olm phase=Installing strategy=deployment time="2024-07-09T11:06:30Z" level=info msg="install strategy successful" csv=packageserver id=eE1kI namespace=everest-olm phase=Installing strategy=deployment time="2024-07-09T11:06:30Z" level=info msg="install strategy successful" csv=packageserver id=njvYn namespace=everest-olm phase=Installing strategy=deployment ``` > Those bottom few log lines repeat multiple times per second forever

packageserver

packageserver logs ``` time="2024-07-09T11:06:31Z" level=info msg="Using in-cluster kube client config" time="2024-07-09T11:06:31Z" level=info msg="Defaulting Interval to '12h0m0s'" I0709 11:06:31.330754 1 handler.go:275] Adding GroupVersion packages.operators.coreos.com v1 to ResourceManager time="2024-07-09T11:06:31Z" level=info msg="connection established. cluster-version: v1.30.2-eks-db838b0" time="2024-07-09T11:06:31Z" level=info msg="operator ready" time="2024-07-09T11:06:31Z" level=info msg="starting informers..." time="2024-07-09T11:06:31Z" level=info msg="informers started" time="2024-07-09T11:06:31Z" level=info msg="waiting for caches to sync..." time="2024-07-09T11:06:31Z" level=info msg="starting workers..." I0709 11:06:31.449997 1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController I0709 11:06:31.450126 1 shared_informer.go:311] Waiting for caches to sync for RequestHeaderAuthRequestController I0709 11:06:31.450019 1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0709 11:06:31.450146 1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0709 11:06:31.450169 1 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::apiserver.local.config/certificates/apiserver.crt::apiserver.local.config/certificates/apiserver.key" I0709 11:06:31.450028 1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController I0709 11:06:31.450399 1 shared_informer.go:311] Waiting for caches to sync for RequestHeaderAuthRequestController I0709 11:06:31.450058 1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0709 11:06:31.450412 1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0709 11:06:31.450478 1 secure_serving.go:213] Serving securely on [::]:5443 I0709 11:06:31.450508 1 tlsconfig.go:240] "Starting DynamicServingCertificateController" I0709 11:06:31.450063 1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0709 11:06:31.450575 1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0709 11:06:31.450097 1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0709 11:06:31.450603 1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0709 11:06:31.551235 1 shared_informer.go:318] Caches are synced for RequestHeaderAuthRequestController I0709 11:06:31.551254 1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0709 11:06:31.551274 1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0709 11:06:31.551300 1 shared_informer.go:318] Caches are synced for RequestHeaderAuthRequestController I0709 11:06:31.551363 1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0709 11:06:31.551235 1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file time="2024-07-09T11:06:41Z" level=info msg="connecting to source" action="sync catalogsource" address="everest-catalog.everest-olm.svc:50051" name=everest-catalog namespace=everest-olm W0709 11:06:41.414565 1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "everest-catalog.everest-olm.svc:50051", ServerName: "everest-catalog.everest-olm.svc:50051", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 172.20.204.21:50051: connect: connection refused" time="2024-07-09T11:06:41Z" level=info msg="updating PackageManifest based on CatalogSource changes: {everest-catalog everest-olm}" action="sync catalogsource" address="everest-catalog.everest-olm.svc:50051" name=everest-catalog namespace=everest-olm time="2024-07-09T11:06:41Z" level=warning msg="error getting bundle stream" action="refresh cache" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 172.20.204.21:50051: connect: connection refused\"" source="{everest-catalog everest-olm}" time="2024-07-09T11:06:42Z" level=info msg="updating PackageManifest based on CatalogSource changes: {everest-catalog everest-olm}" action="sync catalogsource" address="everest-catalog.everest-olm.svc:50051" name=everest-catalog namespace=everest-olm time="2024-07-09T11:06:42Z" level=warning msg="error getting bundle stream" action="refresh cache" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 172.20.204.21:50051: connect: connection refused\"" source="{everest-catalog everest-olm}" W0709 11:06:42.416792 1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "everest-catalog.everest-olm.svc:50051", ServerName: "everest-catalog.everest-olm.svc:50051", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 172.20.204.21:50051: connect: connection refused" W0709 11:06:43.993246 1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "everest-catalog.everest-olm.svc:50051", ServerName: "everest-catalog.everest-olm.svc:50051", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 172.20.204.21:50051: connect: connection refused" W0709 11:06:46.371165 1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "everest-catalog.everest-olm.svc:50051", ServerName: "everest-catalog.everest-olm.svc:50051", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 172.20.204.21:50051: connect: connection refused" W0709 11:06:50.450815 1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "everest-catalog.everest-olm.svc:50051", ServerName: "everest-catalog.everest-olm.svc:50051", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 172.20.204.21:50051: connect: connection refused" time="2024-07-09T11:06:55Z" level=info msg="updating PackageManifest based on CatalogSource changes: {everest-catalog everest-olm}" action="sync catalogsource" address="everest-catalog.everest-olm.svc:50051" name=everest-catalog namespace=everest-olm time="2024-07-09T11:06:55Z" level=warning msg="error getting bundle stream" action="refresh cache" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 172.20.204.21:50051: connect: connection refused\"" source="{everest-catalog everest-olm}" ```
fideloper commented 4 months ago

I got this working, we had to open up the security groups so the control plane could talk to the nodes over port 5443 (where as I had previously setup security groups for just intra-node communication on port 5443).

telderfts commented 2 months ago

This should be internal traffic within the EKS cluster so if we have to open security group ports then something isn't architected correctly on the Percona as the traffic shouldn't be routing like that. Percona should definitely take a look at this if that's the case.