0.22 release install failes with CSV "packageserver" failed to reach phase succeeded

[marandalucas]

Bug Report

What did you do? Basically, run the OLM installation in GKE cluster:

curl -L https://github.com/operator-framework/operator-lifecycle-manager/releases/download/v0.22.0/install.sh -o install.sh
chmod +x install.sh
./install.sh v0.22.0

What did you expect to see? A successful install. The packageserver always is in Installing state.

message: install timeout? why? the cluster has access to the internet and there are not limitations...

What did you see instead? Under which circumstances?

customresourcedefinition.apiextensions.k8s.io/catalogsources.operators.coreos.com created
customresourcedefinition.apiextensions.k8s.io/clusterserviceversions.operators.coreos.com created
customresourcedefinition.apiextensions.k8s.io/installplans.operators.coreos.com created
customresourcedefinition.apiextensions.k8s.io/olmconfigs.operators.coreos.com created
customresourcedefinition.apiextensions.k8s.io/operatorconditions.operators.coreos.com created
customresourcedefinition.apiextensions.k8s.io/operatorgroups.operators.coreos.com created
customresourcedefinition.apiextensions.k8s.io/operators.operators.coreos.com created
customresourcedefinition.apiextensions.k8s.io/subscriptions.operators.coreos.com created
customresourcedefinition.apiextensions.k8s.io/catalogsources.operators.coreos.com condition met
customresourcedefinition.apiextensions.k8s.io/clusterserviceversions.operators.coreos.com condition met
customresourcedefinition.apiextensions.k8s.io/installplans.operators.coreos.com condition met
customresourcedefinition.apiextensions.k8s.io/olmconfigs.operators.coreos.com condition met
customresourcedefinition.apiextensions.k8s.io/operatorconditions.operators.coreos.com condition met
customresourcedefinition.apiextensions.k8s.io/operatorgroups.operators.coreos.com condition met
customresourcedefinition.apiextensions.k8s.io/operators.operators.coreos.com condition met
customresourcedefinition.apiextensions.k8s.io/subscriptions.operators.coreos.com condition met
namespace/olm created
namespace/operators created
serviceaccount/olm-operator-serviceaccount created
clusterrole.rbac.authorization.k8s.io/system:controller:operator-lifecycle-manager created
clusterrolebinding.rbac.authorization.k8s.io/olm-operator-binding-olm created
olmconfig.operators.coreos.com/cluster created
deployment.apps/olm-operator created
deployment.apps/catalog-operator created
clusterrole.rbac.authorization.k8s.io/aggregate-olm-edit created
clusterrole.rbac.authorization.k8s.io/aggregate-olm-view created
operatorgroup.operators.coreos.com/global-operators created
operatorgroup.operators.coreos.com/olm-operators created
clusterserviceversion.operators.coreos.com/packageserver created
catalogsource.operators.coreos.com/operatorhubio-catalog created
Waiting for deployment "olm-operator" rollout to finish: 0 of 1 updated replicas are available...
deployment "olm-operator" successfully rolled out
Waiting for deployment "catalog-operator" rollout to finish: 0 of 1 updated replicas are available...
deployment "catalog-operator" successfully rolled out
Package server phase: Installing
CSV "packageserver" failed to reach phase succeeded

 kubectl get po -n olm
NAME                                                              READY   STATUS      RESTARTS   AGE
6e55037d724a762eb7277a8f1dc55a881cc6816c753283fdc85050591c9f7m6   0/1     Completed   0          140m
catalog-operator-65b5c5bbb8-jjkss                                 1/1     Running     0          91m
e0dad68ecd0743efc9dc2a213c6f80e8dc8eeedc34e1c07d91e31002far9l55   0/1     Completed   0          137m
olm-operator-66bdf4c9df-qzj52                                     1/1     Running     0          91m
operatorhubio-catalog-9t8g7                                       1/1     Running     0          90m
packageserver-f946679c4-d92th                                     1/1     Running     0          6m32s
packageserver-f946679c4-rdmwd                                     1/1     Running     0          6m32s

kubectl get csv -n olm
packageserver                      Package Server              0.21.2                                        Installing

k get csv packageserver -n olm -o yaml
apiVersion: operators.coreos.com/v1alpha1
kind: ClusterServiceVersion
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"operators.coreos.com/v1alpha1","kind":"ClusterServiceVersion","metadata":{"annotations":{},"labels":{"olm.version":"v0.21.2"},"name":"packageserver","namespace":"olm"},"spec":{"apiservicedefinitions":{"owned":[{"containerPort":5443,"deploymentName":"packageserver","description":"A PackageManifest is a resource generated from existing CatalogSources and their ConfigMaps","displayName":"PackageManifest","group":"packages.operators.coreos.com","kind":"PackageManifest","name":"packagemanifests","version":"v1"}]},"description":"Represents an Operator package that is available from a given CatalogSource which will resolve to a ClusterServiceVersion.","displayName":"Package Server","install":{"spec":{"clusterPermissions":[{"rules":[{"apiGroups":["authorization.k8s.io"],"resources":["subjectaccessreviews"],"verbs":["create","get"]},{"apiGroups":[""],"resources":["configmaps"],"verbs":["get","list","watch"]},{"apiGroups":["operators.coreos.com"],"resources":["catalogsources"],"verbs":["get","list","watch"]},{"apiGroups":["packages.operators.coreos.com"],"resources":["packagemanifests"],"verbs":["get","list"]}],"serviceAccountName":"olm-operator-serviceaccount"}],"deployments":[{"name":"packageserver","spec":{"replicas":2,"selector":{"matchLabels":{"app":"packageserver"}},"strategy":{"rollingUpdate":{"maxSurge":1,"maxUnavailable":1},"type":"RollingUpdate"},"template":{"metadata":{"labels":{"app":"packageserver"}},"spec":{"containers":[{"command":["/bin/package-server","-v=4","--secure-port","5443","--global-namespace","olm"],"image":"quay.io/operator-framework/olm@sha256:32db73274863b08cef237d02314a9d8c827ed2f33f0b00166dd3b055af63bb31","imagePullPolicy":"Always","livenessProbe":{"httpGet":{"path":"/healthz","port":5443,"scheme":"HTTPS"}},"name":"packageserver","ports":[{"containerPort":5443}],"readinessProbe":{"httpGet":{"path":"/healthz","port":5443,"scheme":"HTTPS"}},"resources":{"requests":{"cpu":"10m","memory":"50Mi"}},"securityContext":{"runAsUser":1000},"terminationMessagePolicy":"FallbackToLogsOnError","volumeMounts":[{"mountPath":"/tmp","name":"tmpfs"}]}],"nodeSelector":{"kubernetes.io/os":"linux"},"serviceAccountName":"olm-operator-serviceaccount","volumes":[{"emptyDir":{},"name":"tmpfs"}]}}}}]},"strategy":"deployment"},"installModes":[{"supported":true,"type":"OwnNamespace"},{"supported":true,"type":"SingleNamespace"},{"supported":true,"type":"MultiNamespace"},{"supported":true,"type":"AllNamespaces"}],"keywords":["packagemanifests","olm","packages"],"links":[{"name":"Package Server","url":"https://github.com/operator-framework/operator-lifecycle-manager/tree/master/pkg/package-server"}],"maintainers":[{"email":"openshift-operators@redhat.com","name":"Red Hat"}],"maturity":"alpha","minKubeVersion":"1.11.0","provider":{"name":"Red Hat"},"version":"v0.21.2"}}
    olm.operatorGroup: olm-operators
    olm.operatorNamespace: olm
    olm.targetNamespaces: olm
  creationTimestamp: "2022-11-03T14:57:32Z"
  generation: 2
  labels:
    olm.version: v0.21.2
  name: packageserver
  namespace: olm
  resourceVersion: "107001"
  uid: a50459a1-0406-4d80-8aeb-dd1a228840e4
spec:
  apiservicedefinitions:
    owned:
    - containerPort: 5443
      deploymentName: packageserver
      description: A PackageManifest is a resource generated from existing CatalogSources
        and their ConfigMaps
      displayName: PackageManifest
      group: packages.operators.coreos.com
      kind: PackageManifest
      name: packagemanifests
      version: v1
  cleanup:
    enabled: false
  customresourcedefinitions: {}
  description: Represents an Operator package that is available from a given CatalogSource
    which will resolve to a ClusterServiceVersion.
  displayName: Package Server
  install:
    spec:
      clusterPermissions:
      - rules:
        - apiGroups:
          - authorization.k8s.io
          resources:
          - subjectaccessreviews
          verbs:
          - create
          - get
        - apiGroups:
          - ""
          resources:
          - configmaps
          verbs:
          - get
          - list
          - watch
        - apiGroups:
          - operators.coreos.com
          resources:
          - catalogsources
          verbs:
          - get
          - list
          - watch
        - apiGroups:
          - packages.operators.coreos.com
          resources:
          - packagemanifests
          verbs:
          - get
          - list
        serviceAccountName: olm-operator-serviceaccount
      deployments:
      - name: packageserver
        spec:
          replicas: 2
          selector:
            matchLabels:
              app: packageserver
          strategy:
            rollingUpdate:
              maxSurge: 1
              maxUnavailable: 1
            type: RollingUpdate
          template:
            metadata:
              creationTimestamp: null
              labels:
                app: packageserver
            spec:
              containers:
              - command:
                - /bin/package-server
                - -v=4
                - --secure-port
                - "5443"
                - --global-namespace
                - olm
                image: quay.io/operator-framework/olm@sha256:32db73274863b08cef237d02314a9d8c827ed2f33f0b00166dd3b055af63bb31
                imagePullPolicy: Always
                livenessProbe:
                  httpGet:
                    path: /healthz
                    port: 5443
                    scheme: HTTPS
                name: packageserver
                ports:
                - containerPort: 5443
                  protocol: TCP
                readinessProbe:
                  httpGet:
                    path: /healthz
                    port: 5443
                    scheme: HTTPS
                resources:
                  requests:
                    cpu: 10m
                    memory: 50Mi
                securityContext:
                  runAsUser: 1000
                terminationMessagePolicy: FallbackToLogsOnError
                volumeMounts:
                - mountPath: /tmp
                  name: tmpfs
              nodeSelector:
                kubernetes.io/os: linux
              serviceAccountName: olm-operator-serviceaccount
              volumes:
              - emptyDir: {}
                name: tmpfs
    strategy: deployment
  installModes:
  - supported: true
    type: OwnNamespace
  - supported: true
    type: SingleNamespace
  - supported: true
    type: MultiNamespace
  - supported: true
    type: AllNamespaces
  keywords:
  - packagemanifests
  - olm
  - packages
  links:
  - name: Package Server
    url: https://github.com/operator-framework/operator-lifecycle-manager/tree/master/pkg/package-server
  maintainers:
  - email: openshift-operators@redhat.com
    name: Red Hat
  maturity: alpha
  minKubeVersion: 1.11.0
  provider:
    name: Red Hat
  version: 0.21.2
status:
  certsLastUpdated: "2022-11-03T14:57:33Z"
  certsRotateAt: "2024-11-01T14:57:33Z"
  cleanup: {}
  conditions:
  - lastTransitionTime: "2022-11-03T14:57:33Z"
    lastUpdateTime: "2022-11-03T14:57:33Z"
    message: requirements not yet checked
    phase: Pending
    reason: RequirementsUnknown
  - lastTransitionTime: "2022-11-03T14:57:33Z"
    lastUpdateTime: "2022-11-03T14:57:33Z"
    message: all requirements found, attempting install
    phase: InstallReady
    reason: AllRequirementsMet
  - lastTransitionTime: "2022-11-03T14:57:33Z"
    lastUpdateTime: "2022-11-03T14:57:33Z"
    message: waiting for install components to report healthy
    phase: Installing
    reason: InstallSucceeded
  - lastTransitionTime: "2022-11-03T14:57:33Z"
    lastUpdateTime: "2022-11-03T14:57:33Z"
    message: apiServices not installed
    phase: Installing
    reason: InstallWaiting
  - lastTransitionTime: "2022-11-03T15:02:32Z"
    lastUpdateTime: "2022-11-03T15:02:32Z"
    message: install timeout
    phase: Failed
    reason: InstallCheckFailed
  - lastTransitionTime: "2022-11-03T15:02:33Z"
    lastUpdateTime: "2022-11-03T15:02:33Z"
    message: apiServices not installed
    phase: Pending
    reason: NeedsReinstall
  - lastTransitionTime: "2022-11-03T15:02:33Z"
    lastUpdateTime: "2022-11-03T15:02:33Z"
    message: all requirements found, attempting install
    phase: InstallReady
    reason: AllRequirementsMet
  - lastTransitionTime: "2022-11-03T15:02:34Z"
    lastUpdateTime: "2022-11-03T15:02:34Z"
    message: waiting for install components to report healthy
    phase: Installing
    reason: InstallSucceeded
  - lastTransitionTime: "2022-11-03T15:02:34Z"
    lastUpdateTime: "2022-11-03T15:02:34Z"
    message: apiServices not installed
    phase: Installing
    reason: InstallWaiting
  lastTransitionTime: "2022-11-03T15:02:34Z"
  lastUpdateTime: "2022-11-03T15:02:34Z"
  message: apiServices not installed
  phase: Installing
  reason: InstallWaiting
  requirementStatus:
  - group: operators.coreos.com
    kind: ClusterServiceVersion
    message: CSV minKubeVersion (1.11.0) less than server version (v1.22.12-gke.2300)
    name: packageserver
    status: Present
    version: v1alpha1
  - group: apiregistration.k8s.io
    kind: APIService
    message: ""
    name: v1.packages.operators.coreos.com
    status: DeploymentFound
    version: v1
  - dependents:
    - group: rbac.authorization.k8s.io
      kind: PolicyRule
      message: cluster rule:{"verbs":["create","get"],"apiGroups":["authorization.k8s.io"],"resources":["subjectaccessreviews"]}
      status: Satisfied
      version: v1
    - group: rbac.authorization.k8s.io
      kind: PolicyRule
      message: cluster rule:{"verbs":["get","list","watch"],"apiGroups":[""],"resources":["configmaps"]}
      status: Satisfied
      version: v1
    - group: rbac.authorization.k8s.io
      kind: PolicyRule
      message: cluster rule:{"verbs":["get","list","watch"],"apiGroups":["operators.coreos.com"],"resources":["catalogsources"]}
      status: Satisfied
      version: v1
    - group: rbac.authorization.k8s.io
      kind: PolicyRule
      message: cluster rule:{"verbs":["get","list"],"apiGroups":["packages.operators.coreos.com"],"resources":["packagemanifests"]}
      status: Satisfied
      version: v1
    group: ""
    kind: ServiceAccount
    message: ""
    name: olm-operator-serviceaccount
    status: Present
    version: v1

Environment

• operator-lifecycle-manager version:
• v0.22.0

• Kubernetes version information: --cluster-version=1.22.12-gke.2300

  Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.12",

[exdx]

@marandalucas This is likely a transient error to do with the network. I'm not sure there are any plans to change how OLM installs package-server. I would remove the packageserver CSV and simply run the script again -- everything that installed successfully will be unchanged and packageserver will be reinstalled.