Operator installation stalls on CRC (CodeReady Containers)

andymiller96 commented 3 years ago

Environment

CRC for OpenShift 4.7.8

Fedora 34

Steps to recreate

Install CRC
Follow installation instructions in ploigos-software-factory-operator Readme

Operator installation in step 4 (below) stalls with "UpgradePending" status for operator under "Installed Operators" tab

oc apply -f - << EOF
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
name: ploigos-software-factory-operator
namespace: devsecops
spec:
channel: alpha
installPlanApproval: Automatic
name: ploigos-software-factory-operator
source: redhatgov-operators
sourceNamespace: openshift-marketplace
EOF

Notes

Installation via GUI succeeds
Nothing in event log
Adding Starting CSV: ploigos-software-factory-operator.v0.3.2 to subscription does not resolve issue

adamgoossens commented 3 years ago

Thanks, we'll take a look. I'll see if I can replicate it on a local CRC instance.

Meanwhile, are there any errors or unsatisfied conditions in the status fields for the Subscription or ClusterServiceVersion in the devsecops namespace?

adamgoossens commented 3 years ago

I think I know what the problem is here. I suspect there is no OperatorGroup present in the devsecops namespace and so OLM is not watching the namespace for installations.

To solve this problem there are three options:

Install PSFO using the OperatorHub in the GUI, because that process causes an OperatorGroup to be created and added to devsecops when you choose "Install to a specific namespace on the cluster" - this is why GUI installation works for you.
Install PSFO cluster-wide, as it will be installed into the openshift-operators namespace that has an OperatorGroup that targets all namespaces.
Create the OperatorGroup YAML ahead of time for the devsecops namespace.

The following OperatorGroup should work:

apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  namespace: devsecops
  name: devsecops-og
spec:
  targetNamespaces:
    - devsecops

We need to adjust the install documentation to include creating an OperatorGroup in the devsecops namespace.

andymiller96 commented 3 years ago

Thanks, we'll take a look. I'll see if I can replicate it on a local CRC instance.

Meanwhile, are there any errors or unsatisfied conditions in the status fields for the Subscription or ClusterServiceVersion in the devsecops namespace?

There don't appear to be.

$ oc describe subscriptions.operators.coreos.com ploigos-software-factory-operator 
Name:         ploigos-software-factory-operator
Namespace:    devsecops
Labels:       operators.coreos.com/ploigos-software-factory-operator.devsecops=
Annotations:  <none>
API Version:  operators.coreos.com/v1alpha1
Kind:         Subscription
Metadata:
  Creation Timestamp:  2021-05-12T00:19:10Z
  Generation:          1
  Managed Fields:
    API Version:  operators.coreos.com/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:catalogHealth:
        f:conditions:
        f:currentCSV:
        f:installPlanGeneration:
        f:installPlanRef:
          .:
          f:apiVersion:
          f:kind:
          f:name:
          f:namespace:
          f:resourceVersion:
          f:uid:
        f:installplan:
          .:
          f:apiVersion:
          f:kind:
          f:name:
          f:uuid:
        f:lastUpdated:
        f:state:
    Manager:      catalog
    Operation:    Update
    Time:         2021-05-12T00:19:10Z
    API Version:  operators.coreos.com/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:channel:
        f:installPlanApproval:
        f:name:
        f:source:
        f:sourceNamespace:
    Manager:      kubectl-client-side-apply
    Operation:    Update
    Time:         2021-05-12T00:19:10Z
    API Version:  operators.coreos.com/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          .:
          f:operators.coreos.com/ploigos-software-factory-operator.devsecops:
    Manager:         olm
    Operation:       Update
    Time:            2021-05-12T00:19:10Z
  Resource Version:  143262
  Self Link:         /apis/operators.coreos.com/v1alpha1/namespaces/devsecops/subscriptions/ploigos-software-factory-operator
  UID:               4b0b8d5d-05d8-4fec-9899-31f0fb51a2d5
Spec:
  Channel:                alpha
  Install Plan Approval:  Automatic
  Name:                   ploigos-software-factory-operator
  Source:                 redhatgov-operators
  Source Namespace:       openshift-marketplace
Status:
  Catalog Health:
    Catalog Source Ref:
      API Version:       operators.coreos.com/v1alpha1
      Kind:              CatalogSource
      Name:              certified-operators
      Namespace:         openshift-marketplace
      Resource Version:  141258
      UID:               a58e4cd9-a946-4a2e-b8b4-54f6fbec3d8c
    Healthy:             true
    Last Updated:        2021-05-12T00:19:10Z
    Catalog Source Ref:
      API Version:       operators.coreos.com/v1alpha1
      Kind:              CatalogSource
      Name:              community-operators
      Namespace:         openshift-marketplace
      Resource Version:  142629
      UID:               4117a726-400b-4604-ad71-c1b46965da03
    Healthy:             true
    Last Updated:        2021-05-12T00:19:10Z
    Catalog Source Ref:
      API Version:       operators.coreos.com/v1alpha1
      Kind:              CatalogSource
      Name:              redhat-marketplace
      Namespace:         openshift-marketplace
      Resource Version:  142283
      UID:               15e9c43b-1d79-4e03-9c40-d82dacf74d18
    Healthy:             true
    Last Updated:        2021-05-12T00:19:10Z
    Catalog Source Ref:
      API Version:       operators.coreos.com/v1alpha1
      Kind:              CatalogSource
      Name:              redhat-operators
      Namespace:         openshift-marketplace
      Resource Version:  142114
      UID:               0bc54fd0-12bc-426f-877e-7a7c595ff6bd
    Healthy:             true
    Last Updated:        2021-05-12T00:19:10Z
    Catalog Source Ref:
      API Version:       operators.coreos.com/v1alpha1
      Kind:              CatalogSource
      Name:              redhatgov-operators
      Namespace:         openshift-marketplace
      Resource Version:  52932
      UID:               fd1acd85-7a2b-49dc-8b23-9510d829eafa
    Healthy:             true
    Last Updated:        2021-05-12T00:19:10Z
  Conditions:
    Last Transition Time:   2021-05-12T00:19:10Z
    Message:                all available catalogsources are healthy
    Reason:                 AllCatalogSourcesHealthy
    Status:                 False
    Type:                   CatalogSourcesUnhealthy
    Last Transition Time:   2021-05-12T00:19:10Z
    Reason:                 Installing
    Status:                 True
    Type:                   InstallPlanPending
  Current CSV:              ploigos-software-factory-operator.v0.3.2
  Install Plan Generation:  1
  Install Plan Ref:
    API Version:       operators.coreos.com/v1alpha1
    Kind:              InstallPlan
    Name:              install-qd2pj
    Namespace:         devsecops
    Resource Version:  143258
    UID:               901735ba-bbc8-4fa4-b2db-3741555ed287
  Installplan:
    API Version:  operators.coreos.com/v1alpha1
    Kind:         InstallPlan
    Name:         install-qd2pj
    Uuid:         901735ba-bbc8-4fa4-b2db-3741555ed287
  Last Updated:   2021-05-12T00:19:10Z
  State:          UpgradePending
Events:           <none>

andymiller96 commented 3 years ago

I think I know what the problem is here. I suspect there is no OperatorGroup present in the devsecops namespace and so OLM is not watching the namespace for installations.

To solve this problem there are three options:

Install PSFO using the OperatorHub in the GUI, because that process causes an OperatorGroup to be created and added to devsecops when you choose "Install to a specific namespace on the cluster" - this is why GUI installation works for you.

Install PSFO cluster-wide, as it will be installed into the openshift-operators namespace that has an OperatorGroup that targets all namespaces.

Create the OperatorGroup YAML ahead of time for the devsecops namespace.

The following OperatorGroup should work:
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  namespace: devsecops
  name: devsecops-og
spec:
  targetNamespaces:
    - devsecops
We need to adjust the install documentation to include creating an OperatorGroup in the devsecops namespace.

Option 3 worked.

Option 1 was tested and working prior. I haven't tested option 2.

Since the oc cmd line method of installation was in the Readme, I had assumed that this issue was CRC-specific. IOW - I had assumed that the Readme installation instructions would've worked on a regular OCP cluster. Is this not the case?

adamgoossens commented 3 years ago

To be honest, I suspect the default instructions will not work on a regular OCP cluster. I could be wrong - OLM is still a little bit 'magic' to me - but I've never seen Operator installation work without a corresponding OperatorGroup.

Easy enough to test though, and a quick PR to the installation instructions should make the OperatorGroup requirement explicit.

adamgoossens commented 3 years ago

Yeah, I just attempted to follow our instructions on a OCP 4.7 cluster, with no OperatorGroup, and I'm stuck in UpgradePending.

I'll raise a PR to fix the README.

ploigos / ploigos-software-factory-operator