Unable to fetch operator from OperatorSource on k8 cluster?

[rvsingh011]

Support Question

why I am unable to fetch operators from the operator source on k8 cluster?

What did you do? I added this as operatorSource to the cluster.

apiVersion: operators.coreos.com/v1
kind: OperatorSource
metadata:
  name: redhat-operators
  namespace: default
spec:
  displayName: Red Hat Operators
  endpoint: 'https://quay.io/cnr'
  publisher: Red Hat
  registryNamespace: redhat-operators
  type: appregistry

then, As per the documentation, I added a catalog source

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  labels:
    olm-visibility: hidden
    openshift-marketplace: 'true'
    opsrc-datastore: 'true'
    opsrc-owner-name: redhat-operators
    opsrc-owner-namespace: default
    opsrc-provider: redhat
  name: redhat-operators
  namespace: default
spec:
  address: 'redhat-operators.openshift-marketplace.svc:50051'
  displayName: Red Hat Operators
  icon:
    base64data: ''
    mediatype: ''
  publisher: Red Hat
  sourceType: grpc

What did you expect to see? I should see the operators, and doing kubectl get opsrc I should see the Status as Succeeded.

What did you see instead? Under which circumstances? I don't see anything under Status

Environment OLM version : 0.14.1

• Kubernetes version information:

  Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.5", GitCommit:"20c265fef0741dd71a66480e35bd69f18351daea", GitTreeState:"clean", BuildDate:"2019-10-15T19:16:51Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"darwin/amd64"}
  Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.2", GitCommit:"e6f1cd7", GitTreeState:"clean", BuildDate:"2020-04-20T06:30:19Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes cluster kind: minikube

Additional context Output of kubectl describe opsrc redhat-operators

Name:         redhat-operators
Namespace:    default
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"operators.coreos.com/v1","kind":"OperatorSource","metadata":{"annotations":{},"name":"redhat-operators","namespace":"defaul...
API Version:  operators.coreos.com/v1
Kind:         OperatorSource
Metadata:
  Creation Timestamp:  2020-05-19T16:28:33Z
  Generation:          1
  Managed Fields:
    API Version:  operators.coreos.com/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:displayName:
        f:endpoint:
        f:publisher:
        f:registryNamespace:
        f:type:
    Manager:         kubectl
    Operation:       Update
    Time:            2020-05-19T16:28:33Z
  Resource Version:  235166
  Self Link:         /apis/operators.coreos.com/v1/namespaces/default/operatorsources/redhat-operators
  UID:               c398bdfe-ba9b-4730-baf3-82cd7247dc4b
Spec:
  Display Name:        Red Hat Operators
  Endpoint:            https://quay.io/cnr
  Publisher:           Red Hat
  Registry Namespace:  redhat-operators
  Type:                appregistry
Events:                <none>```

[kevinrizza]

Hey @rvsingh011,

The marketplace operator (https://github.com/operator-framework/operator-marketplace) manages operator sources, not OLM. Depending on how you installed that component, that operator only looks for operator sources in the same namespace it is installed (for upstream, the default namespace is marketplace https://github.com/operator-framework/operator-marketplace/blob/master/deploy/upstream/08_operator.yaml#L5).

That being said, the marketplace should just create the catalog source for you -- can you point me to the documentation you are using? It seems like it is most likely out of date.

[rvsingh011]

Thanks @kevinrizza for such a quick reply. I have one more doubt which I have in the context of openshift cluster, I am not seeing the operator listed here: https://github.com/operator-framework/community-operators/tree/master/community-operators. I am using an openshift cluster, V4.3. All operator source seems to be configured.

Running the command kubectl get packagemanifest -n openshift-marketplace Does not list the operator codeready-toolchain-operator. Which is actually there in the community operator. All operator listed are:

NAME                                         CATALOG               AGE
ocean-operator                               Certified Operators   15d
cic-operator                                 Certified Operators   15d
cpx-cic-operator                             Certified Operators   15d
couchbase-enterprise-certified               Certified Operators   15d
rocketchat-operator-certified                Certified Operators   15d
rapidbiz-operator-certified                  Certified Operators   15d
datagrid                                     Red Hat Operators     15d
ibm-helm-api-operator-app                    Certified Operators   15d
oneagent-certified                           Certified Operators   15d
openshiftartifactoryha-operator              Certified Operators   15d
cockroachdb-certified                        Certified Operators   15d
openshiftansibleservicebroker                Red Hat Operators     15d
timemachine-operator                         Certified Operators   15d
storageos                                    Certified Operators   15d
kubemq-operator-marketplace                  Certified Operators   15d
traefikee-certified                          Certified Operators   15d
dv-operator                                  Red Hat Operators     15d
aqua-certified                               Certified Operators   15d
appdynamics-operator                         Certified Operators   15d
container-security-operator                  Red Hat Operators     15d
hazelcast-enterprise-certified               Certified Operators   15d
t8c-certified                                Certified Operators   15d
hpe-csi-operator                             Certified Operators   15d
elasticsearch-operator                       Red Hat Operators     15d
ibm-spectrum-scale-csi                       Certified Operators   15d
openunison-ocp-certified                     Certified Operators   15d
servicemeshoperator                          Red Hat Operators     15d
insightedge-enterprise-operator2             Certified Operators   15d
gpu-operator-certified                       Certified Operators   15d
nfd                                          Red Hat Operators     15d
ubix-operator                                Certified Operators   15d
runtime-component-operator-certified         Certified Operators   15d
sematext                                     Certified Operators   15d
ibm-spectrum-symphony-operator               Certified Operators   15d
amq-streams                                  Red Hat Operators     15d
nxiq-operator-certified                      Certified Operators   15d
atomicorp-helm-operator-certified            Certified Operators   15d
akka-cluster-operator-certified              Certified Operators   15d
ibm-auditlogging-operator-app                Certified Operators   15d
redhat-marketplace-operator                  Certified Operators   15d
traefikee-redhat-certified                   Certified Operators   15d
ibm-block-csi-operator                       Certified Operators   15d
hazelcast-jet-enterprise-operator            Certified Operators   15d
cih-operator-certified                       Certified Operators   15d
transadv-operator                            Certified Operators   15d
joget-dx-operator                            Certified Operators   15d
kubeturbo-certified                          Certified Operators   15d
cyberarmor-operator-certified                Certified Operators   15d
ivory-server-app                             Certified Operators   15d
eddi-operator-certified                      Certified Operators   15d
kiali-ossm                                   Red Hat Operators     15d
ocs-operator                                 Red Hat Operators     15d
cluster-logging                              Red Hat Operators     15d
ibm-metering-operator-app                    Certified Operators   15d
yugabyte-operator                            Certified Operators   15d
amq-online                                   Red Hat Operators     15d
triggermesh-operator                         Certified Operators   15d
ibm-mongodb-operator-app                     Certified Operators   15d
memql-certified                              Certified Operators   15d
k8s-triliovault                              Certified Operators   15d
kubevirt-hyperconverged                      Red Hat Operators     15d
here-service-operator-certified              Certified Operators   15d
neuvector-certified-operator                 Certified Operators   15d
fuse-online                                  Red Hat Operators     15d
cam-operator                                 Red Hat Operators     15d
xcrypt-operator                              Certified Operators   15d
twistlock-certified                          Certified Operators   15d
percona-server-mongodb-operator-certified    Certified Operators   15d
eap                                          Red Hat Operators     15d
ibm-helm-repo-operator-app                   Certified Operators   15d
splunk-certified                             Certified Operators   15d
amq-broker-lts                               Red Hat Operators     15d
cluster-kube-descheduler-operator            Red Hat Operators     15d
quay-operator                                Red Hat Operators     15d
kubeturbo-marketplace-certified              Certified Operators   15d
tidb-operator-certified                      Certified Operators   15d
f5-bigip-ctlr-operator                       Certified Operators   15d
amq-broker                                   Red Hat Operators     15d
ptp-operator                                 Red Hat Operators     15d
3scale-operator                              Red Hat Operators     15d
cortex-healthcare-hub-operator               Certified Operators   15d
open-liberty-certified                       Certified Operators   15d
sysdig-certified                             Certified Operators   15d
nuodb-ce-certified                           Certified Operators   15d
dotscience-operator                          Certified Operators   15d
ibm-monitoring-grafana-operator-app          Certified Operators   15d
cortex-fabric-operator                       Certified Operators   15d
apicast-operator                             Red Hat Operators     15d
metering-ocp                                 Red Hat Operators     15d
tigera-operator                              Certified Operators   15d
anchore-engine                               Certified Operators   15d
kubemq-operator                              Certified Operators   15d
serverless-operator                          Red Hat Operators     15d
ibm-platform-api-operator-app                Certified Operators   15d
nxrm-operator-certified                      Certified Operators   15d
uma-operator                                 Certified Operators   15d
planetscale-certified                        Certified Operators   15d
portworx-certified                           Certified Operators   15d
clusterresourceoverride                      Red Hat Operators     15d
sriov-network-operator                       Red Hat Operators     15d
open-enterprise-spinnaker                    Certified Operators   15d
citrix-adc-istio-ingress-gateway-operator    Certified Operators   15d
cortex-hub-operator                          Certified Operators   15d
fuse-apicurito                               Red Hat Operators     15d
local-storage-operator                       Red Hat Operators     15d
aqua-operator-certified                      Certified Operators   15d
infinibox-operator-certified                 Certified Operators   15d
kube-arangodb                                Certified Operators   15d
storageos-10tb                               Certified Operators   15d
performance-addon-operator                   Red Hat Operators     15d
gitlab-operator                              Certified Operators   15d
robin-operator                               Certified Operators   15d
synopsys-certified                           Certified Operators   15d
crunchy-postgres-operator                    Certified Operators   15d
appranix-cps                                 Certified Operators   15d
portshift-operator                           Certified Operators   15d
nginx-ingress-operator                       Certified Operators   15d
driverlessai-deployment-operator-certified   Certified Operators   15d
appsody-operator-certified                   Certified Operators   15d
codeready-workspaces                         Red Hat Operators     15d
kong                                         Certified Operators   15d
orca                                         Certified Operators   15d
presto-operator                              Certified Operators   15d
openshifttemplateservicebroker               Red Hat Operators     15d
citrix-cpx-istio-sidecar-injector-operator   Certified Operators   15d
instana-agent                                Certified Operators   15d
cert-manager-operator                        Certified Operators   15d
ibm-management-ingress-operator-app          Certified Operators   15d
perceptilabs-operator-package                Certified Operators   15d
couchdb-operator-certified                   Certified Operators   15d
jaeger-product                               Red Hat Operators     15d
openshift-pipelines-operator-rh              Red Hat Operators     15d
cortex-operator                              Certified Operators   15d
anaconda-team-edition                        Certified Operators   15d
cortex-certifai-operator                     Certified Operators   15d
zabbix-operator-certified                    Certified Operators   15d
businessautomation-operator                  Red Hat Operators     15d
amq7-interconnect-operator                   Red Hat Operators     15d
federatorai-certified                        Certified Operators   15d
mongodb-enterprise                           Certified Operators   15d
wavefront-operator                           Certified Operators   15d
newrelic-infrastructure                      Certified Operators   15d
storageos-1tb                                Certified Operators   15d
joget-openshift-operator                     Certified Operators   15d
amq7-cert-manager                            Red Hat Operators     15d
quay-bridge-operator                         Red Hat Operators     15d
alcide-kaudit-operator                       Certified Operators   15d
redis-enterprise-operator-cert               Certified Operators   15d
percona-xtradb-cluster-operator-certified    Certified Operators   15d
seldon-operator-certified                    Certified Operators   15d
datadog-operator-certified                   Certified Operators   15d```

[kevinrizza]

@rvsingh011

It seems like from that kubectl output that only certified and redhat operators are viewable by OLM for some reason. The operator source being in a good state just means that it was able to contact the external app-registry instance to get metadata associated with the operators, not that the local catalog was able to start and serve them. Can you take a look at the status of the catalog source community-operators in the openshift-marketplace namespace? And also take a look at the logs of the operator registry pod it is attempting to attach to -- it should also be in the openshift-marketplace namespace with a generated name that includes community-operators-

[rvsingh011]

@kevinrizza Yes, you are right, I see ann unhealthy status for the community pod.

  ContainersReady   True 
  PodScheduled      True 
Volumes:
  marketplace-trusted-ca:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      marketplace-trusted-ca
    Optional:  false
  default-token-r8jrf:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-r8jrf
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  beta.kubernetes.io/os=linux
Tolerations:     node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                    From                    Message
  ----     ------     ----                   ----                    -------
  Warning  Unhealthy  151m (x204 over 3h6m)  kubelet, 10.150.106.76  Readiness probe failed: timeout: failed to connect service "localhost:50051" within 1s

Is is something to do with the DNS and configuration of the openshift cluster?

[kevinrizza]

So, I actually think this is due to quay.io being unresponsive for most of the day yesterday. That error indicates your catalog registry pod was unable to connect to OLM's catalog operator, so my guess is the local registry just wasn't starting up at all (probably it couldn't download anything from quay properly and was crash looping). Can you try again today and see if it comes up successfully now?

[Kampe]

I'm seeing the same issues with the pod attempting to start and never becoming ready as the readycheck is hitting localhost:50051

OLM v0.15.1

The pods will just hang

olm           operatorhubio-catalog-kw7xd                 0/1     Running     6          6m35s

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[dinhxuanvu]

The issue is stale. Closed. Please feel free to reopen if needed.