IBM / cloud-pak-deployer

Configuration-based installation of OpenShift and Cloud Pak for Data/Integration/Watson AIOps on various private and public cloud infrastructure providers. Deployment attempts to achieve the end-state defined in the configuration. If something fails along the way, you only need to restart the process to continue the deployment.
https://ibm.github.io/cloud-pak-deployer/
Apache License 2.0
131 stars 67 forks source link

Failure when installing operators #301

Closed triceam closed 1 year ago

triceam commented 1 year ago

Describe the bug I'm seeing failures when the deployer is installing operators/subscriptions. This is in an existing OCP 4.10 cluster, with the deployer running in a job inside the cluster (gitops configuration).

In the logs I see:


TASK [cp4d-subscriptions : Run apply-olm command to install cartridge subscriptions, logs are in /Data/cpd-status/log/cpd-instance-apply-olm-cartridge-sub.log] ***
Wednesday 18 January 2023 20:13:45 +0000 (0:00:00.263) 0:21:19.274 *****
FAILED - RETRYING: Run apply-olm command to install cartridge subscriptions, logs are in /Data/cpd-status/log/cpd-instance-apply-olm-cartridge-sub.log (2 retries left).
FAILED - RETRYING: Run apply-olm command to install cartridge subscriptions, logs are in /Data/cpd-status/log/cpd-instance-apply-olm-cartridge-sub.log (1 retries left).
changed: [localhost]
TASK [cp4d-subscriptions : Retrieving state of all operators in project ibm-common-services] ***
Wednesday 18 January 2023 20:26:53 +0000 (0:13:07.476) 0:34:26.750 *****
changed: [localhost]
TASK [cp4d-subscriptions : fail] ***********************************************
Wednesday 18 January 2023 20:26:53 +0000 (0:00:00.479) 0:34:27.229 *****
fatal: [localhost]: FAILED! => changed=false
msg: 'Creation of the subscriptions failed (possible due to OLM issue): [''ibm-common-service-operator,2023-01-18T20:13:57Z,ibm-common-service-operator.v3.19.0,AtLatestKnown'', ''ibm-namespace-scope-operator,2023-01-18T20:14:45Z,ibm-namespace-scope-operator.v1.13.0,AtLatestKnown'', ''operand-deployment-lifecycle-manager-app,2023-01-18T20:15:52Z,operand-deployment-lifecycle-manager.v1.17.0,AtLatestKnown'', ''cpd-operator,2023-01-18T20:18:00Z,cpd-platform-operator.v3.0.0,AtLatestKnown'', ''ibm-cpd-ccs-operator,2023-01-18T20:18:03Z,ibm-cpd-ccs.v2.0.0,AtLatestKnown'', ''ibm-db2u-operator,2023-01-18T20:18:05Z,db2u-operator.v2.0.0,AtLatestKnown'', ''redis-operator,2023-01-18T20:18:07Z,ibm-cloud-databases-redis.v1.5.1,AtLatestKnown'', ''ibm-dmc-operator-subscription,2023-01-18T20:18:10Z,ibm-databases-dmc.v1.5.0,AtLatestKnown'', ''ibm-dv-operator-catalog-subscription,2023-01-18T20:18:12Z,ibm-dv-operator.v1.8.0,AtLatestKnown'']'
PLAY RECAP *********************************************************************
localhost : ok=592 changed=99 unreachable=0 failed=1 skipped=294 rescued=0 ignored=0
Wednesday 18 January 2023 20:26:53 +0000 (0:00:00.032) 0:34:27.261 *****
===============================================================================
cp4d-catalog-source : Run apply-olm command to create catalog sources, logs are in /Data/cpd-status/log/apply-olm-create-catsrc.log - 932.01s
cp4d-subscriptions : Run apply-olm command to install cartridge subscriptions, logs are in /Data/cpd-status/log/cpd-instance-apply-olm-cartridge-sub.log - 787.48s
cp4d-case-save : Run apply-olm command to download case files, logs are in /Data/cpd-status/log/apply-olm-case-save.log -- 77.35s
cp4d-variables : Run list-components command --------------------------- 36.63s
cp4d-subscriptions : Generate preview script to install cartridge subscriptions, logs are in /Data/cpd-status/log/cpd-instance-apply-olm-cartridge-sub.log -- 35.37s
cp4d-catalog-source : Generate preview script to create catalog sources, logs are in /Data/cpd-status/log/apply-olm-create-catsrc.log -- 30.70s
cp4d-case-save : Generate preview script to download case files, logs are in /Data/cpd-status/log/apply-olm-case-save.log -- 26.23s
cp4d-ocp-kubelet-config : Retrieve /etc/kubernetes/kubelet.conf --------- 9.00s
openshift-download-client : Unpack OpenShift client from /Data/cpd-status/downloads/openshift-client-linux.tar.gz-4.8 --- 8.10s
cpd-cli-download : Unpack cpd-cli from /Data/cpd-status/downloads/cpd-cli-linux-amd64.tar.gz --- 8.00s
====================================================================================
Deployer FAILED. Check previous messages. If command line is not returned, press ^C.

It looks like the deployer is timing out when creating operator subscriptions. I saw the first FAILED - RETRYING: Run apply-olm command to install cartridge subscriptions, logs are in /Data/cpd-status/log/cpd-instance-apply-olm-cartridge-sub.log (2 retries left). message, and noticed one of the operator subscriptions hadn't completed successfully. When it was re-running, I deleted the "stuck" operator, and it proceeded to deploy correctly.

For this last failure, I do not see any stuck or failed deployments or events on the cluster, but it errored out nonetheless. Here's what I see for installed operators:

image

Expected behavior I expected the installation to proceed without any problems.

triceam commented 1 year ago

Note: That screenshot of installed operators does not contain all operators for what I expected to be installed.

Here's the config that I am attempting to deploy:

---
global_config:
  environment_name: cloud-pak-for-data
  cloud_platform: existing-ocp

openshift:
- name: cloud-pak-for-data
  ocp_version: 4.8
  cluster_name: cloud-pak-for-data
  domain_name: amt-data-fabric-0ed2b1307d0ce0e0627ae949f443cd72-0000.us-east.containers.appdomain.cloud
  openshift_storage:
  - storage_name: ocs-storage
    storage_type: ocs
    ocs_storage_label: ocs
    ocs_storage_size_gb: 500

cp4d:
- project: cpd-instance
  openshift_cluster_name: cloud-pak-for-data
  cp4d_version: 4.5.0
  sequential_install: True
  accept_licenses: true
  cartridges:

  - name: cp-foundation
    license_service:
      state: disabled
      threads_per_core: 2

  - name: lite

  - name: scheduler 
    state: removed 

#
# All tested cartridges. To install, change the "state" property to "installed". To uninstall, change the state
# to "removed" or comment out the entire cartridge. Make sure that the "-" and properties are aligned with the lite
# cartridge; the "-" is at position 3 and the property starts at position 5.
#

  - name: analyticsengine 
    size: small 
    state: removed 

  - name: bigsql
    state: removed 

  - name: ca
    size: small
    instances:
    - name: ca-instance
      metastore_ref: ca-metastore
    state: removed 

  - name: cde
    state: removed 

  - name: datagate
    state: removed 

  - name: datastage-ent-plus
    state: removed 
    # instances:
    #   - name: ds-instance
    #     # Optional settings
    #     description: "datastage ds-instance"
    #     size: medium
    #     storage_class: efs-nfs-client
    #     storage_size_gb: 60
    #     # Custom Scale options
    #     scale_px_runtime:
    #       replicas: 2
    #       cpu_request: 500m
    #       cpu_limit: 2
    #       memory_request: 2Gi
    #       memory_limit: 4Gi
    #     scale_px_compute:
    #       replicas: 2
    #       cpu_request: 1
    #       cpu_limit: 3
    #       memory_request: 4Gi
    #       memory_limit: 12Gi    

  - name: db2
    size: small
    instances:
    - name: ca-metastore
      metadata_size_gb: 20
      data_size_gb: 20
      backup_size_gb: 20  
      transactionlog_size_gb: 20
    state: removed 

  - name: db2u
    state: removed 

  - name: db2wh
    state: removed 

  - name: dmc
    state: removed 

  - name: dods
    size: small
    state: removed 

  - name: dp
    size: small
    state: removed 

  - name: dv
    size: small 
    instances:
    - name: data-virtualization
    state: installed 

  - name: hadoop
    size: small
    state: removed 

  - name: mdm
    size: small
    wkc_enabled: true
    state: removed 

  - name: openpages
    state: removed 

  - name: planning-analytics
    state: removed 

  - name: rstudio
    size: small
    state: removed 

  - name: spss
    state: removed 

  - name: voice-gateway
    replicas: 1
    state: removed 

  - name: watson-assistant
    size: small
    state: removed 

  - name: watson-discovery
    state: removed 

  - name: watson-ks
    size: small
    state: installed 

  - name: watson-openscale
    size: small
    state: removed 

  - name: watson-speech
    stt_size: xsmall
    tts_size: xsmall
    state: removed 

  - name: wkc
    size: small
    state: installed 

  - name: wml
    size: small
    state: installed 

  - name: wml-accelerator
    replicas: 1
    size: small
    state: removed 

  - name: wsl
    state: removed
triceam commented 1 year ago

I can restart the job, and it looks like it picks up where it last left off, but I don't see why it failed in the first place. I wouldn't expect this to fail in the first place.

After restarting, I see more failures:

TASK [cp4d-subscriptions : Show apply-olm command to create cartridge subscriptions] ***
Wednesday 18 January 2023 20:44:42 +0000 (0:00:00.112) 0:08:10.410 *****
ok: [localhost] =>
_apply_olm_command: |-
/opt/ansible/bin/apply-olm \
--release=4.5.0 \
--case_download=false \
--catsrc=false --sub=true --offline_dir=/Data/cpd-status/cp4d/case -v \
--preview=false \
--upgrade=false \
--components=cpfs,cpd_platform,dv,watson_ks,wkc,wml
TASK [cp4d-subscriptions : Starting background task to patch OLM artifacts in project ibm-common-services. Logs are in /Data/cpd-status/log/ibm-common-services-patch-olm.log] ***
Wednesday 18 January 2023 20:44:42 +0000 (0:00:00.031) 0:08:10.442 *****
changed: [localhost]
TASK [cp4d-subscriptions : Run apply-olm command to install cartridge subscriptions, logs are in /Data/cpd-status/log/cpd-instance-apply-olm-cartridge-sub.log] ***
Wednesday 18 January 2023 20:44:43 +0000 (0:00:00.235) 0:08:10.677 *****
FAILED - RETRYING: Run apply-olm command to install cartridge subscriptions, logs are in /Data/cpd-status/log/cpd-instance-apply-olm-cartridge-sub.log (2 retries left).
FAILED - RETRYING: Run apply-olm command to install cartridge subscriptions, logs are in /Data/cpd-status/log/cpd-instance-apply-olm-cartridge-sub.log (1 retries left).

but i see activity on the cluster:

image

and i still dont see any more failed operators:

image
triceam commented 1 year ago

I suspect this could be the issue I am running into: https://issues.redhat.com/projects/RHIBMCS/issues/RHIBMCS-147?filter=allopenissues

OpenShift 4.10 (the LTS release) now having a broken OLM functionality: https://bugzilla.redhat.com/show_bug.cgi?id=2101357

fketelaars commented 1 year ago

@triceam I also suspect this is caused by one of the OLM issues of OpenShift 4.10. Cloud Pak Deployer detects and remediates one of the issues where the CSV is created but not associated with the Subscription; it does this in the background while waiting for the operators to install. Unfortunately there are still race conditions and the remediation does not work 100% of the time, most-likely close to 95%. Re-running the deployer sometimes closes that gap to 100%.

If you can DM me the credentials of your cluster, I can take a look why deployer didn't manage to remediate.

There's also still a pending issue: #287 , which we didn't work on yet. From what I can see in the screen shots, this is not the issue you ran into.

You can view the $STATUS_DIR/log/ibm-common-services-patch-olm.log file and check if the OLM remediation eventually succeeded.

triceam commented 1 year ago

@fketelaars I increased the backoff limit so that the deploy job in the cluster automatically restarted... which it did 12 times, and it still is not able to deploy successfully when the combination of dv, wks, wkc, and wml are all installed at the same time. (Note: this configuration consistently fails for me (on 4 separate clusters now), but it works fine if I don't deploy any cartridges)

There are still no errors shown with installed operators:

image

However, the deploy job keeps failing with this message:

TASK [cp4d-subscriptions : Show apply-olm command to create cartridge subscriptions] ***
Thursday 19 January 2023 23:01:51 +0000 (0:00:00.262) 0:11:30.143 ******
ok: [localhost] =>
_apply_olm_command: |-
/opt/ansible/bin/apply-olm \
--release=4.5.0 \
--case_download=false \
--catsrc=false --sub=true --offline_dir=/Data/cpd-status/cp4d/case -v \
--preview=false \
--upgrade=false \
--components=cpfs,cpd_platform,dv,watson_ks,wkc,wml
TASK [cp4d-subscriptions : Starting background task to patch OLM artifacts in project ibm-common-services. Logs are in /Data/cpd-status/log/ibm-common-services-patch-olm.log] ***
Thursday 19 January 2023 23:01:51 +0000 (0:00:00.054) 0:11:30.197 ******
changed: [localhost]
TASK [cp4d-subscriptions : Run apply-olm command to install cartridge subscriptions, logs are in /Data/cpd-status/log/cpd-instance-apply-olm-cartridge-sub.log] ***
Thursday 19 January 2023 23:01:51 +0000 (0:00:00.419) 0:11:30.617 ******
FAILED - RETRYING: Run apply-olm command to install cartridge subscriptions, logs are in /Data/cpd-status/log/cpd-instance-apply-olm-cartridge-sub.log (2 retries left).
FAILED - RETRYING: Run apply-olm command to install cartridge subscriptions, logs are in /Data/cpd-status/log/cpd-instance-apply-olm-cartridge-sub.log (1 retries left).
changed: [localhost]
TASK [cp4d-subscriptions : Retrieving state of all operators in project ibm-common-services] ***
Thursday 19 January 2023 23:12:52 +0000 (0:11:00.919) 0:22:31.536 ******
changed: [localhost]
TASK [cp4d-subscriptions : fail] ***********************************************
Thursday 19 January 2023 23:12:53 +0000 (0:00:00.595) 0:22:32.132 ******
fatal: [localhost]: FAILED! => changed=false
msg: 'Creation of the subscriptions failed (possible due to OLM issue): [''ibm-common-service-operator,2023-01-19T18:19:01Z,ibm-common-service-operator.v3.19.0,AtLatestKnown'', ''ibm-namespace-scope-operator,2023-01-19T18:19:54Z,ibm-namespace-scope-operator.v1.13.0,AtLatestKnown'', ''operand-deployment-lifecycle-manager-app,2023-01-19T18:21:02Z,operand-deployment-lifecycle-manager.v1.17.0,AtLatestKnown'', ''cpd-operator,2023-01-19T18:23:30Z,cpd-platform-operator.v3.0.0,AtLatestKnown'', ''cloud-native-postgresql,2023-01-19T18:44:19Z,cloud-native-postgresql.v1.16.2,AtLatestKnown'', ''ibm-cpd-ccs-operator,2023-01-19T18:50:21Z,ibm-cpd-ccs.v2.0.0,AtLatestKnown'', ''ibm-dmc-operator-subscription,2023-01-19T19:12:41Z,ibm-databases-dmc.v1.5.0,AtLatestKnown'', ''ibm-dv-operator-catalog-subscription,2023-01-19T19:12:44Z,ibm-dv-operator.v1.8.0,AtLatestKnown'', ''ibm-db2u-operator,2023-01-19T19:13:02Z,db2u-operator.v2.0.0,AtLatestKnown'', ''redis-operator,2023-01-19T19:13:03Z,ibm-cloud-databases-redis.v1.5.1,AtLates...
''ibm-cpd-ae-operator,2023-01-19T19:13:07Z,ibm-cpd-ae.v2.0.0,AtLatestKnown'', ''ibm-db2aaservice-cp4d-operator,2023-01-19T19:13:09Z,ibm-db2aaservice-cp4d-operator.v2.0.0,AtLatestKnown'', ''ibm-fdb-operator,2023-01-19T19:13:09Z,ibm-opencontent-foundationdb.v1.0.5,AtLatestKnown'', ''ibm-cpd-datastage-operator,2023-01-19T19:13:10Z,ibm-cpd-datastage-operator.v2.0.0,AtLatestKnown'', ''ibm-cpd-iis-operator,2023-01-19T19:13:11Z,ibm-cpd-iis.v1.5.0,AtLatestKnown'', ''ibm-cpd-datarefinery-operator,2023-01-19T19:13:11Z,ibm-cpd-datarefinery.v2.0.0,AtLatestKnown'', ''manta-adl-operator,2023-01-19T19:13:12Z,manta-adl-operator.v1.3.8,AtLatestKnown'', ''ibm-cpd-wkc-operator-catalog-subscription,2023-01-19T19:13:13Z,ibm-cpd-wkc.v1.5.0,AtLatestKnown'', ''ibm-cpd-wml-operator,2023-01-19T19:13:14Z,ibm-cpd-wml-operator.v2.0.0,AtLatestKnown'', ''fdb-kubernetes-operator-v1.0-ibm-fdb-operator-catalog-openshift-marketplace,2023-01-19T19:14:21Z,fdb-kubernetes-operator.v1.0.5,AtLatestKnown'']'
PLAY RECAP *********************************************************************
localhost : ok=594 changed=76 unreachable=0 failed=1 skipped=299 rescued=0 ignored=0
Thursday 19 January 2023 23:12:53 +0000 (0:00:00.048) 0:22:32.180 ******
===============================================================================
cp4d-subscriptions : Run apply-olm command to install cartridge subscriptions, logs are in /Data/cpd-status/log/cpd-instance-apply-olm-cartridge-sub.log - 660.92s
cp4d-catalog-source : Run apply-olm command to create catalog sources, logs are in /Data/cpd-status/log/apply-olm-create-catsrc.log - 283.47s
cp4d-subscriptions : Generate preview script to install cartridge subscriptions, logs are in /Data/cpd-status/log/cpd-instance-apply-olm-cartridge-sub.log -- 57.74s
cp4d-variables : Run list-components command --------------------------- 54.04s
cp4d-catalog-source : Generate preview script to create catalog sources, logs are in /Data/cpd-status/log/apply-olm-create-catsrc.log -- 49.38s
cp4d-case-save : Generate preview script to download case files, logs are in /Data/cpd-status/log/apply-olm-case-save.log -- 41.69s
cp4d-case-save : Run shell script to download case files, logs are in /Data/cpd-status/log/apply-olm-case-save.log -- 38.83s
cpd-cli-download : Unpack cpd-cli from /Data/cpd-status/downloads/cpd-cli-linux-amd64.tar.gz -- 11.31s
openshift-download-client : Unpack OpenShift client from /Data/cpd-status/downloads/openshift-client-linux.tar.gz-4.8 --- 4.31s
cp4d-ocp-crio-config : Retrieve /etc/crio/crio.conf --------------------- 3.37s
====================================================================================
Deployer FAILED. Check previous messages. If command line is not returned, press ^C.
image

Here's what I see when I run cat $STATUS_DIR/log/ibm-common-services-patch-olm.log (older records are cut off): You'll see that the last messages say "Everything looks good", so I'm not sure what to do next to debug this situation.

[2023-01-19 19:09:32] ----------
[2023-01-19 19:09:32] Checking OLM artifacts in project ibm-common-services
[2023-01-19 19:09:32] ----------
[2023-01-19 19:09:32] Collecting OLM information
[2023-01-19 19:09:33] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 19:09:33] INFO: Everything looks good
[2023-01-19 19:09:33] ----------
[2023-01-19 19:09:33] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 19:09:33] ----------
[2023-01-19 19:10:33] ----------
[2023-01-19 19:10:33] Checking OLM artifacts in project ibm-common-services
[2023-01-19 19:10:33] ----------
[2023-01-19 19:10:33] Collecting OLM information
[2023-01-19 19:10:34] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 19:10:34] INFO: Everything looks good
[2023-01-19 19:10:34] ----------
[2023-01-19 19:10:34] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 19:10:34] ----------
[2023-01-19 19:11:34] ----------
[2023-01-19 19:11:34] Checking OLM artifacts in project ibm-common-services
[2023-01-19 19:11:34] ----------
[2023-01-19 19:11:34] Collecting OLM information
[2023-01-19 19:11:35] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 19:11:35] INFO: Everything looks good
[2023-01-19 19:11:35] ----------
[2023-01-19 19:11:35] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 19:11:35] ----------
[2023-01-19 19:12:35] ----------
[2023-01-19 19:12:35] Checking OLM artifacts in project ibm-common-services
[2023-01-19 19:12:35] ----------
[2023-01-19 19:12:35] Collecting OLM information
[2023-01-19 19:12:36] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 19:12:36] WARNING: Subscription ibm-db2u-operator does not have a valid CSV, will try to remediate.
[2023-01-19 19:12:36] WARNING: Subscription redis-operator does not have a valid CSV, will try to remediate.
[2023-01-19 19:12:36] WARNING: Subscription ibm-dmc-operator-subscription does not have a valid CSV, will try to remediate.
[2023-01-19 19:12:36] WARNING: Subscription ibm-dv-operator-catalog-subscription does not have a valid CSV, will try to remediate.
[2023-01-19 19:12:36] WARNING: Subscription ibm-cpd-ae-operator does not have a valid CSV, will try to remediate.
[2023-01-19 19:12:36] WARNING: Subscription ibm-fdb-operator does not have a valid CSV, will try to remediate.
[2023-01-19 19:12:36] WARNING: Subscription ibm-db2aaservice-cp4d-operator does not have a valid CSV, will try to remediate.
[2023-01-19 19:12:36] WARNING: Subscription ibm-cpd-datarefinery-operator does not have a valid CSV, will try to remediate.
[2023-01-19 19:12:36] WARNING: Subscription ibm-cpd-datastage-operator does not have a valid CSV, will try to remediate.
[2023-01-19 19:12:36] WARNING: Subscription ibm-cpd-iis-operator does not have a valid CSV, will try to remediate.
[2023-01-19 19:12:36] WARNING: Subscription ibm-cpd-wkc-operator-catalog-subscription does not have a valid CSV, will try to remediate.
[2023-01-19 19:12:36] WARNING: Subscription manta-adl-operator does not have a valid CSV, will try to remediate.
[2023-01-19 19:12:36] WARNING: Subscription ibm-cpd-wml-operator does not have a valid CSV, will try to remediate.
[2023-01-19 19:12:36] WARNING: Not all operators are correctly installed. All entries should have a CSV and have state AtLatestKnown
ibm-common-service-operator,2023-01-19T18:19:01Z,ibm-common-service-operator.v3.19.0,AtLatestKnown
ibm-namespace-scope-operator,2023-01-19T18:19:54Z,ibm-namespace-scope-operator.v1.13.0,AtLatestKnown
operand-deployment-lifecycle-manager-app,2023-01-19T18:21:02Z,operand-deployment-lifecycle-manager.v1.17.0,AtLatestKnown
cpd-operator,2023-01-19T18:23:30Z,cpd-platform-operator.v3.0.0,AtLatestKnown
cloud-native-postgresql,2023-01-19T18:44:19Z,cloud-native-postgresql.v1.16.2,AtLatestKnown
ibm-cpd-ccs-operator,2023-01-19T18:50:21Z,ibm-cpd-ccs.v2.0.0,AtLatestKnown
ibm-db2u-operator,2023-01-19T19:07:19Z,,
redis-operator,2023-01-19T19:07:20Z,,
ibm-dmc-operator-subscription,2023-01-19T19:07:21Z,,
ibm-dv-operator-catalog-subscription,2023-01-19T19:07:21Z,,
ibm-cpd-ae-operator,2023-01-19T19:07:24Z,,
ibm-fdb-operator,2023-01-19T19:07:26Z,,
ibm-db2aaservice-cp4d-operator,2023-01-19T19:07:26Z,,
ibm-cpd-datarefinery-operator,2023-01-19T19:07:27Z,,
ibm-cpd-datastage-operator,2023-01-19T19:07:27Z,,
ibm-cpd-iis-operator,2023-01-19T19:07:28Z,,
ibm-cpd-wkc-operator-catalog-subscription,2023-01-19T19:07:29Z,,
manta-adl-operator,2023-01-19T19:07:29Z,,
ibm-cpd-wml-operator,2023-01-19T19:07:31Z,,
[2023-01-19 19:12:36] START OF REMEDIATION. Diagnostics info in /Data/cpd-status/log/olm-diag-1674155555
[2023-01-19 19:12:36] DIAG: Collecting OLM operator logs to /Data/cpd-status/log/olm-diag-1674155555
[2023-01-19 19:12:37] DIAG: Exporting subscription ibm-db2u-operator to /Data/cpd-status/log/olm-diag-1674155555/sub-ibm-db2u-operator.yaml and deleting
[2023-01-19 19:12:37] DIAG: Exporting subscription redis-operator to /Data/cpd-status/log/olm-diag-1674155555/sub-redis-operator.yaml and deleting
[2023-01-19 19:12:38] DIAG: Exporting subscription ibm-dmc-operator-subscription to /Data/cpd-status/log/olm-diag-1674155555/sub-ibm-dmc-operator-subscription.yaml and deleting
[2023-01-19 19:12:38] DIAG: Exporting subscription ibm-dv-operator-catalog-subscription to /Data/cpd-status/log/olm-diag-1674155555/sub-ibm-dv-operator-catalog-subscription.yaml and deleting
[2023-01-19 19:12:38] DIAG: Exporting subscription ibm-cpd-ae-operator to /Data/cpd-status/log/olm-diag-1674155555/sub-ibm-cpd-ae-operator.yaml and deleting
[2023-01-19 19:12:39] DIAG: Exporting subscription ibm-fdb-operator to /Data/cpd-status/log/olm-diag-1674155555/sub-ibm-fdb-operator.yaml and deleting
[2023-01-19 19:12:39] DIAG: Exporting subscription ibm-db2aaservice-cp4d-operator to /Data/cpd-status/log/olm-diag-1674155555/sub-ibm-db2aaservice-cp4d-operator.yaml and deleting
[2023-01-19 19:12:40] DIAG: Exporting subscription ibm-cpd-datarefinery-operator to /Data/cpd-status/log/olm-diag-1674155555/sub-ibm-cpd-datarefinery-operator.yaml and deleting
[2023-01-19 19:12:40] DIAG: Exporting subscription ibm-cpd-datastage-operator to /Data/cpd-status/log/olm-diag-1674155555/sub-ibm-cpd-datastage-operator.yaml and deleting
[2023-01-19 19:12:41] DIAG: Exporting subscription ibm-cpd-iis-operator to /Data/cpd-status/log/olm-diag-1674155555/sub-ibm-cpd-iis-operator.yaml and deleting
[2023-01-19 19:12:41] DIAG: Exporting subscription ibm-cpd-wkc-operator-catalog-subscription to /Data/cpd-status/log/olm-diag-1674155555/sub-ibm-cpd-wkc-operator-catalog-subscription.yaml and deleting
[2023-01-19 19:12:41] DIAG: Exporting subscription manta-adl-operator to /Data/cpd-status/log/olm-diag-1674155555/sub-manta-adl-operator.yaml and deleting
[2023-01-19 19:12:42] DIAG: Exporting subscription ibm-cpd-wml-operator to /Data/cpd-status/log/olm-diag-1674155555/sub-ibm-cpd-wml-operator.yaml and deleting
[2023-01-19 19:12:42] DIAG: Orphaned CSV gitea-operator.v1.3.0 found, exporting to /Data/cpd-status/log/olm-diag-1674155555/csv-gitea-operator.v1.3.0.yaml
[2023-01-19 19:12:42] REMEDIATE: Deleting orphaned CSV gitea-operator.v1.3.0
[2023-01-19 19:12:42] DIAG: Orphaned CSV ibm-cpd-datastage-operator.v2.0.0 found, exporting to /Data/cpd-status/log/olm-diag-1674155555/csv-ibm-cpd-datastage-operator.v2.0.0.yaml
[2023-01-19 19:12:43] REMEDIATE: Deleting install plan install-fbxln, associated with CSV ibm-cpd-datastage-operator.v2.0.0
[2023-01-19 19:12:43] REMEDIATE: Deleting orphaned CSV ibm-cpd-datastage-operator.v2.0.0
[2023-01-19 19:12:43] DIAG: Orphaned CSV openshift-gitops-operator.v1.5.8 found, exporting to /Data/cpd-status/log/olm-diag-1674155555/csv-openshift-gitops-operator.v1.5.8.yaml
[2023-01-19 19:12:43] REMEDIATE: Deleting orphaned CSV openshift-gitops-operator.v1.5.8
[2023-01-19 19:12:44] DIAG: Orphaned CSV openshift-pipelines-operator-rh.v1.6.4 found, exporting to /Data/cpd-status/log/olm-diag-1674155555/csv-openshift-pipelines-operator-rh.v1.6.4.yaml
[2023-01-19 19:12:44] REMEDIATE: Deleting orphaned CSV openshift-pipelines-operator-rh.v1.6.4
[2023-01-19 19:12:44] DIAG: Orphaned CSV ibm-cpd-wkc.v1.5.0 found, exporting to /Data/cpd-status/log/olm-diag-1674155555/csv-ibm-cpd-wkc.v1.5.0.yaml
[2023-01-19 19:12:44] REMEDIATE: Deleting install plan install-fbxln, associated with CSV ibm-cpd-wkc.v1.5.0
[2023-01-19 19:12:44] REMEDIATE: Deleting orphaned CSV ibm-cpd-wkc.v1.5.0
[2023-01-19 19:12:45] DIAG: Orphaned CSV ibm-cpd-iis.v1.5.0 found, exporting to /Data/cpd-status/log/olm-diag-1674155555/csv-ibm-cpd-iis.v1.5.0.yaml
[2023-01-19 19:12:45] REMEDIATE: Deleting install plan install-fbxln, associated with CSV ibm-cpd-iis.v1.5.0
[2023-01-19 19:12:45] REMEDIATE: Deleting orphaned CSV ibm-cpd-iis.v1.5.0
[2023-01-19 19:12:45] DIAG: Orphaned CSV ibm-cpd-datarefinery.v2.0.0 found, exporting to /Data/cpd-status/log/olm-diag-1674155555/csv-ibm-cpd-datarefinery.v2.0.0.yaml
[2023-01-19 19:12:45] REMEDIATE: Deleting install plan install-fbxln, associated with CSV ibm-cpd-datarefinery.v2.0.0
[2023-01-19 19:12:46] REMEDIATE: Deleting orphaned CSV ibm-cpd-datarefinery.v2.0.0
[2023-01-19 19:12:46] DIAG: Orphaned CSV ibm-cpd-wml-operator.v2.0.0 found, exporting to /Data/cpd-status/log/olm-diag-1674155555/csv-ibm-cpd-wml-operator.v2.0.0.yaml
[2023-01-19 19:12:46] REMEDIATE: Deleting install plan install-fbxln, associated with CSV ibm-cpd-wml-operator.v2.0.0
[2023-01-19 19:12:46] REMEDIATE: Deleting orphaned CSV ibm-cpd-wml-operator.v2.0.0
[2023-01-19 19:12:46] DIAG: Orphaned CSV manta-adl-operator.v1.3.8 found, exporting to /Data/cpd-status/log/olm-diag-1674155555/csv-manta-adl-operator.v1.3.8.yaml
[2023-01-19 19:12:47] REMEDIATE: Deleting install plan install-fbxln, associated with CSV manta-adl-operator.v1.3.8
[2023-01-19 19:12:47] REMEDIATE: Deleting orphaned CSV manta-adl-operator.v1.3.8
[2023-01-19 19:12:47] REMEDIATE: Re-running script to create subscriptions /Data/cpd-status/cp4d/cpd-instance-apply-olm-cartridge-sub.sh, output in /Data/cpd-status/log/olm-diag-1674155555/cpd-instance-apply-olm-cartridge-sub.log
[2023-01-19 19:13:14] END OF REMEDIATION
[2023-01-19 19:13:14] ----------
[2023-01-19 19:13:14] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 19:13:14] ----------
[2023-01-19 19:14:14] ----------
[2023-01-19 19:14:14] Checking OLM artifacts in project ibm-common-services
[2023-01-19 19:14:14] ----------
[2023-01-19 19:14:14] Collecting OLM information
[2023-01-19 19:14:15] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 19:14:15] INFO: Everything looks good
[2023-01-19 19:14:15] ----------
[2023-01-19 19:14:15] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 19:14:15] ----------
[2023-01-19 19:15:15] ----------
[2023-01-19 19:15:15] Checking OLM artifacts in project ibm-common-services
[2023-01-19 19:15:15] ----------
[2023-01-19 19:15:15] Collecting OLM information
[2023-01-19 19:15:17] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 19:15:17] INFO: Everything looks good
[2023-01-19 19:15:17] ----------
[2023-01-19 19:15:17] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 19:15:17] ----------
[2023-01-19 19:16:17] ----------
[2023-01-19 19:16:17] Checking OLM artifacts in project ibm-common-services
[2023-01-19 19:16:17] ----------
[2023-01-19 19:16:17] Collecting OLM information
[2023-01-19 19:16:18] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 19:16:18] INFO: Everything looks good
[2023-01-19 19:16:18] ----------
[2023-01-19 19:16:18] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 19:16:18] ----------
[2023-01-19 19:17:18] ----------
[2023-01-19 19:17:18] Checking OLM artifacts in project ibm-common-services
[2023-01-19 19:17:18] ----------
[2023-01-19 19:17:18] Collecting OLM information
[2023-01-19 19:17:19] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 19:17:19] INFO: Everything looks good
[2023-01-19 19:17:19] ----------
[2023-01-19 19:17:19] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 19:17:19] ----------
[2023-01-19 19:18:19] ----------
[2023-01-19 19:18:19] Checking OLM artifacts in project ibm-common-services
[2023-01-19 19:18:19] ----------
[2023-01-19 19:18:19] Collecting OLM information
[2023-01-19 19:18:20] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 19:18:20] INFO: Everything looks good
[2023-01-19 19:18:20] ----------
[2023-01-19 19:18:20] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 19:18:20] ----------
[2023-01-19 19:32:53] ----------
[2023-01-19 19:32:53] Info: Checking access to project ibm-common-services
[2023-01-19 19:32:53] ----------
[2023-01-19 19:32:53] Checking OLM artifacts in project ibm-common-services
[2023-01-19 19:32:53] ----------
[2023-01-19 19:32:53] Collecting OLM information
[2023-01-19 19:32:54] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 19:32:54] INFO: Everything looks good
[2023-01-19 19:32:54] ----------
[2023-01-19 19:32:54] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 19:32:54] ----------
[2023-01-19 19:33:54] ----------
[2023-01-19 19:33:54] Checking OLM artifacts in project ibm-common-services
[2023-01-19 19:33:54] ----------
[2023-01-19 19:33:55] Collecting OLM information
[2023-01-19 19:33:56] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 19:33:56] INFO: Everything looks good
[2023-01-19 19:33:56] ----------
[2023-01-19 19:33:56] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 19:33:56] ----------
[2023-01-19 19:34:56] ----------
[2023-01-19 19:34:56] Checking OLM artifacts in project ibm-common-services
[2023-01-19 19:34:56] ----------
[2023-01-19 19:34:56] Collecting OLM information
[2023-01-19 19:34:57] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 19:34:57] INFO: Everything looks good
[2023-01-19 19:34:57] ----------
[2023-01-19 19:34:57] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 19:34:57] ----------
[2023-01-19 19:35:57] ----------
[2023-01-19 19:35:57] Checking OLM artifacts in project ibm-common-services
[2023-01-19 19:35:57] ----------
[2023-01-19 19:35:57] Collecting OLM information
[2023-01-19 19:35:58] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 19:35:58] INFO: Everything looks good
[2023-01-19 19:35:58] ----------
[2023-01-19 19:35:58] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 19:35:58] ----------
[2023-01-19 19:36:58] ----------
[2023-01-19 19:36:58] Checking OLM artifacts in project ibm-common-services
[2023-01-19 19:36:58] ----------
[2023-01-19 19:36:58] Collecting OLM information
[2023-01-19 19:36:59] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 19:36:59] INFO: Everything looks good
[2023-01-19 19:36:59] ----------
[2023-01-19 19:36:59] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 19:36:59] ----------
[2023-01-19 19:37:59] ----------
[2023-01-19 19:37:59] Checking OLM artifacts in project ibm-common-services
[2023-01-19 19:37:59] ----------
[2023-01-19 19:37:59] Collecting OLM information
[2023-01-19 19:38:00] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 19:38:00] INFO: Everything looks good
[2023-01-19 19:38:00] ----------
[2023-01-19 19:38:00] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 19:38:00] ----------
[2023-01-19 19:39:00] ----------
[2023-01-19 19:39:00] Checking OLM artifacts in project ibm-common-services
[2023-01-19 19:39:00] ----------
[2023-01-19 19:39:00] Collecting OLM information
[2023-01-19 19:39:01] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 19:39:01] INFO: Everything looks good
[2023-01-19 19:39:01] ----------
[2023-01-19 19:39:01] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 19:39:01] ----------
[2023-01-19 19:40:01] ----------
[2023-01-19 19:40:01] Checking OLM artifacts in project ibm-common-services
[2023-01-19 19:40:01] ----------
[2023-01-19 19:40:01] Collecting OLM information
[2023-01-19 19:40:02] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 19:40:02] INFO: Everything looks good
[2023-01-19 19:40:02] ----------
[2023-01-19 19:40:02] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 19:40:02] ----------
[2023-01-19 19:41:02] ----------
[2023-01-19 19:41:02] Checking OLM artifacts in project ibm-common-services
[2023-01-19 19:41:02] ----------
[2023-01-19 19:41:02] Collecting OLM information
[2023-01-19 19:41:04] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 19:41:04] INFO: Everything looks good
[2023-01-19 19:41:04] ----------
[2023-01-19 19:41:04] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 19:41:04] ----------
[2023-01-19 19:42:04] ----------
[2023-01-19 19:42:04] Checking OLM artifacts in project ibm-common-services
[2023-01-19 19:42:04] ----------
[2023-01-19 19:42:04] Collecting OLM information
[2023-01-19 19:42:05] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 19:42:05] INFO: Everything looks good
[2023-01-19 19:42:05] ----------
[2023-01-19 19:42:05] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 19:42:05] ----------
[2023-01-19 19:43:05] ----------
[2023-01-19 19:43:05] Checking OLM artifacts in project ibm-common-services
[2023-01-19 19:43:05] ----------
[2023-01-19 19:43:05] Collecting OLM information
[2023-01-19 19:43:06] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 19:43:06] INFO: Everything looks good
[2023-01-19 19:43:06] ----------
[2023-01-19 19:43:06] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 19:43:06] ----------
[2023-01-19 19:56:26] ----------
[2023-01-19 19:56:26] Info: Checking access to project ibm-common-services
[2023-01-19 19:56:26] ----------
[2023-01-19 19:56:26] Checking OLM artifacts in project ibm-common-services
[2023-01-19 19:56:26] ----------
[2023-01-19 19:56:26] Collecting OLM information
[2023-01-19 19:56:27] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 19:56:27] INFO: Everything looks good
[2023-01-19 19:56:27] ----------
[2023-01-19 19:56:27] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 19:56:27] ----------
[2023-01-19 19:57:27] ----------
[2023-01-19 19:57:27] Checking OLM artifacts in project ibm-common-services
[2023-01-19 19:57:27] ----------
[2023-01-19 19:57:27] Collecting OLM information
[2023-01-19 19:57:28] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 19:57:28] INFO: Everything looks good
[2023-01-19 19:57:28] ----------
[2023-01-19 19:57:28] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 19:57:28] ----------
[2023-01-19 19:58:28] ----------
[2023-01-19 19:58:28] Checking OLM artifacts in project ibm-common-services
[2023-01-19 19:58:28] ----------
[2023-01-19 19:58:28] Collecting OLM information
[2023-01-19 19:58:29] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 19:58:30] INFO: Everything looks good
[2023-01-19 19:58:30] ----------
[2023-01-19 19:58:30] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 19:58:30] ----------
[2023-01-19 19:59:30] ----------
[2023-01-19 19:59:30] Checking OLM artifacts in project ibm-common-services
[2023-01-19 19:59:30] ----------
[2023-01-19 19:59:30] Collecting OLM information
[2023-01-19 19:59:31] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 19:59:31] INFO: Everything looks good
[2023-01-19 19:59:31] ----------
[2023-01-19 19:59:31] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 19:59:31] ----------
[2023-01-19 20:00:31] ----------
[2023-01-19 20:00:31] Checking OLM artifacts in project ibm-common-services
[2023-01-19 20:00:31] ----------
[2023-01-19 20:00:31] Collecting OLM information
[2023-01-19 20:00:32] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 20:00:32] INFO: Everything looks good
[2023-01-19 20:00:32] ----------
[2023-01-19 20:00:32] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 20:00:32] ----------
[2023-01-19 20:01:32] ----------
[2023-01-19 20:01:32] Checking OLM artifacts in project ibm-common-services
[2023-01-19 20:01:32] ----------
[2023-01-19 20:01:32] Collecting OLM information
[2023-01-19 20:01:33] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 20:01:33] INFO: Everything looks good
[2023-01-19 20:01:33] ----------
[2023-01-19 20:01:33] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 20:01:33] ----------
[2023-01-19 20:02:33] ----------
[2023-01-19 20:02:33] Checking OLM artifacts in project ibm-common-services
[2023-01-19 20:02:33] ----------
[2023-01-19 20:02:33] Collecting OLM information
[2023-01-19 20:02:34] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 20:02:34] INFO: Everything looks good
[2023-01-19 20:02:34] ----------
[2023-01-19 20:02:34] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 20:02:34] ----------
[2023-01-19 20:03:34] ----------
[2023-01-19 20:03:34] Checking OLM artifacts in project ibm-common-services
[2023-01-19 20:03:34] ----------
[2023-01-19 20:03:34] Collecting OLM information
[2023-01-19 20:03:35] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 20:03:35] INFO: Everything looks good
[2023-01-19 20:03:35] ----------
[2023-01-19 20:03:35] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 20:03:35] ----------
[2023-01-19 20:04:35] ----------
[2023-01-19 20:04:35] Checking OLM artifacts in project ibm-common-services
[2023-01-19 20:04:35] ----------
[2023-01-19 20:04:35] Collecting OLM information
[2023-01-19 20:04:36] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 20:04:36] INFO: Everything looks good
[2023-01-19 20:04:36] ----------
[2023-01-19 20:04:36] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 20:04:36] ----------
[2023-01-19 20:05:36] ----------
[2023-01-19 20:05:36] Checking OLM artifacts in project ibm-common-services
[2023-01-19 20:05:36] ----------
[2023-01-19 20:05:36] Collecting OLM information
[2023-01-19 20:05:37] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 20:05:37] INFO: Everything looks good
[2023-01-19 20:05:37] ----------
[2023-01-19 20:05:37] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 20:05:37] ----------
[2023-01-19 20:06:37] ----------
[2023-01-19 20:06:37] Checking OLM artifacts in project ibm-common-services
[2023-01-19 20:06:37] ----------
[2023-01-19 20:06:37] Collecting OLM information
[2023-01-19 20:06:38] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 20:06:38] INFO: Everything looks good
[2023-01-19 20:06:38] ----------
[2023-01-19 20:06:38] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 20:06:38] ----------
[2023-01-19 20:24:57] ----------
[2023-01-19 20:24:58] Info: Checking access to project ibm-common-services
[2023-01-19 20:24:58] ----------
[2023-01-19 20:24:58] Checking OLM artifacts in project ibm-common-services
[2023-01-19 20:24:58] ----------
[2023-01-19 20:24:58] Collecting OLM information
[2023-01-19 20:24:59] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 20:24:59] INFO: Everything looks good
[2023-01-19 20:24:59] ----------
[2023-01-19 20:24:59] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 20:24:59] ----------
[2023-01-19 20:25:59] ----------
[2023-01-19 20:25:59] Checking OLM artifacts in project ibm-common-services
[2023-01-19 20:25:59] ----------
[2023-01-19 20:25:59] Collecting OLM information
[2023-01-19 20:26:00] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 20:26:00] INFO: Everything looks good
[2023-01-19 20:26:00] ----------
[2023-01-19 20:26:00] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 20:26:00] ----------
[2023-01-19 20:27:00] ----------
[2023-01-19 20:27:00] Checking OLM artifacts in project ibm-common-services
[2023-01-19 20:27:00] ----------
[2023-01-19 20:27:00] Collecting OLM information
[2023-01-19 20:27:01] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 20:27:01] INFO: Everything looks good
[2023-01-19 20:27:01] ----------
[2023-01-19 20:27:01] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 20:27:01] ----------
[2023-01-19 20:28:01] ----------
[2023-01-19 20:28:01] Checking OLM artifacts in project ibm-common-services
[2023-01-19 20:28:01] ----------
[2023-01-19 20:28:01] Collecting OLM information
[2023-01-19 20:28:02] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 20:28:02] INFO: Everything looks good
[2023-01-19 20:28:02] ----------
[2023-01-19 20:28:02] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 20:28:02] ----------
[2023-01-19 20:29:02] ----------
[2023-01-19 20:29:02] Checking OLM artifacts in project ibm-common-services
[2023-01-19 20:29:02] ----------
[2023-01-19 20:29:02] Collecting OLM information
[2023-01-19 20:29:03] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 20:29:03] INFO: Everything looks good
[2023-01-19 20:29:03] ----------
[2023-01-19 20:29:03] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 20:29:03] ----------
[2023-01-19 20:30:03] ----------
[2023-01-19 20:30:03] Checking OLM artifacts in project ibm-common-services
[2023-01-19 20:30:03] ----------
[2023-01-19 20:30:03] Collecting OLM information
[2023-01-19 20:30:04] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 20:30:04] INFO: Everything looks good
[2023-01-19 20:30:04] ----------
[2023-01-19 20:30:04] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 20:30:04] ----------
[2023-01-19 20:31:04] ----------
[2023-01-19 20:31:05] Checking OLM artifacts in project ibm-common-services
[2023-01-19 20:31:05] ----------
[2023-01-19 20:31:05] Collecting OLM information
[2023-01-19 20:31:05] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 20:31:05] INFO: Everything looks good
[2023-01-19 20:31:06] ----------
[2023-01-19 20:31:06] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 20:31:06] ----------
[2023-01-19 20:32:06] ----------
[2023-01-19 20:32:06] Checking OLM artifacts in project ibm-common-services
[2023-01-19 20:32:06] ----------
[2023-01-19 20:32:06] Collecting OLM information
[2023-01-19 20:32:07] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 20:32:07] INFO: Everything looks good
[2023-01-19 20:32:07] ----------
[2023-01-19 20:32:07] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 20:32:07] ----------
[2023-01-19 20:33:07] ----------
[2023-01-19 20:33:07] Checking OLM artifacts in project ibm-common-services
[2023-01-19 20:33:07] ----------
[2023-01-19 20:33:07] Collecting OLM information
[2023-01-19 20:33:08] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 20:33:08] INFO: Everything looks good
[2023-01-19 20:33:08] ----------
[2023-01-19 20:33:08] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 20:33:08] ----------
[2023-01-19 20:34:08] ----------
[2023-01-19 20:34:08] Checking OLM artifacts in project ibm-common-services
[2023-01-19 20:34:08] ----------
[2023-01-19 20:34:08] Collecting OLM information
[2023-01-19 20:34:09] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 20:34:09] INFO: Everything looks good
[2023-01-19 20:34:09] ----------
[2023-01-19 20:34:09] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 20:34:09] ----------
[2023-01-19 20:35:09] ----------
[2023-01-19 20:35:09] Checking OLM artifacts in project ibm-common-services
[2023-01-19 20:35:09] ----------
[2023-01-19 20:35:09] Collecting OLM information
[2023-01-19 20:35:10] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 20:35:10] INFO: Everything looks good
[2023-01-19 20:35:10] ----------
[2023-01-19 20:35:10] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 20:35:10] ----------
[2023-01-19 21:06:41] ----------
[2023-01-19 21:06:41] Info: Checking access to project ibm-common-services
[2023-01-19 21:06:42] ----------
[2023-01-19 21:06:42] Checking OLM artifacts in project ibm-common-services
[2023-01-19 21:06:42] ----------
[2023-01-19 21:06:42] Collecting OLM information
[2023-01-19 21:06:43] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 21:06:43] INFO: Everything looks good
[2023-01-19 21:06:43] ----------
[2023-01-19 21:06:43] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 21:06:43] ----------
[2023-01-19 21:07:43] ----------
[2023-01-19 21:07:43] Checking OLM artifacts in project ibm-common-services
[2023-01-19 21:07:43] ----------
[2023-01-19 21:07:43] Collecting OLM information
[2023-01-19 21:07:44] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 21:07:44] INFO: Everything looks good
[2023-01-19 21:07:44] ----------
[2023-01-19 21:07:44] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 21:07:44] ----------
[2023-01-19 21:08:44] ----------
[2023-01-19 21:08:44] Checking OLM artifacts in project ibm-common-services
[2023-01-19 21:08:44] ----------
[2023-01-19 21:08:44] Collecting OLM information
[2023-01-19 21:08:45] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 21:08:45] INFO: Everything looks good
[2023-01-19 21:08:45] ----------
[2023-01-19 21:08:45] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 21:08:45] ----------
[2023-01-19 21:09:45] ----------
[2023-01-19 21:09:45] Checking OLM artifacts in project ibm-common-services
[2023-01-19 21:09:45] ----------
[2023-01-19 21:09:45] Collecting OLM information
[2023-01-19 21:09:46] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 21:09:46] INFO: Everything looks good
[2023-01-19 21:09:46] ----------
[2023-01-19 21:09:46] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 21:09:46] ----------
[2023-01-19 21:10:46] ----------
[2023-01-19 21:10:46] Checking OLM artifacts in project ibm-common-services
[2023-01-19 21:10:46] ----------
[2023-01-19 21:10:46] Collecting OLM information
[2023-01-19 21:10:47] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 21:10:47] INFO: Everything looks good
[2023-01-19 21:10:47] ----------
[2023-01-19 21:10:47] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 21:10:47] ----------
[2023-01-19 21:11:47] ----------
[2023-01-19 21:11:47] Checking OLM artifacts in project ibm-common-services
[2023-01-19 21:11:47] ----------
[2023-01-19 21:11:47] Collecting OLM information
[2023-01-19 21:11:48] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 21:11:48] INFO: Everything looks good
[2023-01-19 21:11:48] ----------
[2023-01-19 21:11:48] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 21:11:48] ----------
[2023-01-19 21:12:48] ----------
[2023-01-19 21:12:48] Checking OLM artifacts in project ibm-common-services
[2023-01-19 21:12:48] ----------
[2023-01-19 21:12:48] Collecting OLM information
[2023-01-19 21:12:49] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 21:12:49] INFO: Everything looks good
[2023-01-19 21:12:49] ----------
[2023-01-19 21:12:49] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 21:12:49] ----------
[2023-01-19 21:13:49] ----------
[2023-01-19 21:13:49] Checking OLM artifacts in project ibm-common-services
[2023-01-19 21:13:49] ----------
[2023-01-19 21:13:49] Collecting OLM information
[2023-01-19 21:13:50] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 21:13:50] INFO: Everything looks good
[2023-01-19 21:13:50] ----------
[2023-01-19 21:13:50] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 21:13:50] ----------
[2023-01-19 21:14:50] ----------
[2023-01-19 21:14:50] Checking OLM artifacts in project ibm-common-services
[2023-01-19 21:14:50] ----------
[2023-01-19 21:14:50] Collecting OLM information
[2023-01-19 21:14:51] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 21:14:51] INFO: Everything looks good
[2023-01-19 21:14:51] ----------
[2023-01-19 21:14:51] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 21:14:51] ----------
[2023-01-19 21:15:52] ----------
[2023-01-19 21:15:52] Checking OLM artifacts in project ibm-common-services
[2023-01-19 21:15:52] ----------
[2023-01-19 21:15:52] Collecting OLM information
[2023-01-19 21:15:53] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 21:15:53] INFO: Everything looks good
[2023-01-19 21:15:53] ----------
[2023-01-19 21:15:53] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 21:15:53] ----------
[2023-01-19 21:16:53] ----------
[2023-01-19 21:16:53] Checking OLM artifacts in project ibm-common-services
[2023-01-19 21:16:53] ----------
[2023-01-19 21:16:53] Collecting OLM information
[2023-01-19 21:16:54] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 21:16:54] INFO: Everything looks good
[2023-01-19 21:16:54] ----------
[2023-01-19 21:16:54] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 21:16:54] ----------
[2023-01-19 21:29:22] ----------
[2023-01-19 21:29:22] Info: Checking access to project ibm-common-services
[2023-01-19 21:29:22] ----------
[2023-01-19 21:29:22] Checking OLM artifacts in project ibm-common-services
[2023-01-19 21:29:22] ----------
[2023-01-19 21:29:22] Collecting OLM information
[2023-01-19 21:29:23] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 21:29:23] INFO: Everything looks good
[2023-01-19 21:29:23] ----------
[2023-01-19 21:29:23] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 21:29:23] ----------
[2023-01-19 21:30:23] ----------
[2023-01-19 21:30:23] Checking OLM artifacts in project ibm-common-services
[2023-01-19 21:30:23] ----------
[2023-01-19 21:30:23] Collecting OLM information
[2023-01-19 21:30:24] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 21:30:24] INFO: Everything looks good
[2023-01-19 21:30:24] ----------
[2023-01-19 21:30:24] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 21:30:24] ----------
[2023-01-19 21:31:24] ----------
[2023-01-19 21:31:24] Checking OLM artifacts in project ibm-common-services
[2023-01-19 21:31:24] ----------
[2023-01-19 21:31:24] Collecting OLM information
[2023-01-19 21:31:25] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 21:31:25] INFO: Everything looks good
[2023-01-19 21:31:25] ----------
[2023-01-19 21:31:25] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 21:31:25] ----------
[2023-01-19 21:32:25] ----------
[2023-01-19 21:32:25] Checking OLM artifacts in project ibm-common-services
[2023-01-19 21:32:25] ----------
[2023-01-19 21:32:25] Collecting OLM information
[2023-01-19 21:32:26] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 21:32:26] INFO: Everything looks good
[2023-01-19 21:32:26] ----------
[2023-01-19 21:32:26] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 21:32:26] ----------
[2023-01-19 21:33:26] ----------
[2023-01-19 21:33:26] Checking OLM artifacts in project ibm-common-services
[2023-01-19 21:33:26] ----------
[2023-01-19 21:33:26] Collecting OLM information
[2023-01-19 21:33:27] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 21:33:27] INFO: Everything looks good
[2023-01-19 21:33:27] ----------
[2023-01-19 21:33:27] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 21:33:27] ----------
[2023-01-19 21:34:28] ----------
[2023-01-19 21:34:28] Checking OLM artifacts in project ibm-common-services
[2023-01-19 21:34:28] ----------
[2023-01-19 21:34:28] Collecting OLM information
[2023-01-19 21:34:28] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 21:34:29] INFO: Everything looks good
[2023-01-19 21:34:29] ----------
[2023-01-19 21:34:29] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 21:34:29] ----------
[2023-01-19 21:35:29] ----------
[2023-01-19 21:35:29] Checking OLM artifacts in project ibm-common-services
[2023-01-19 21:35:29] ----------
[2023-01-19 21:35:29] Collecting OLM information
[2023-01-19 21:35:30] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 21:35:30] INFO: Everything looks good
[2023-01-19 21:35:30] ----------
[2023-01-19 21:35:30] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 21:35:30] ----------
[2023-01-19 21:36:30] ----------
[2023-01-19 21:36:30] Checking OLM artifacts in project ibm-common-services
[2023-01-19 21:36:30] ----------
[2023-01-19 21:36:30] Collecting OLM information
[2023-01-19 21:36:31] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 21:36:31] INFO: Everything looks good
[2023-01-19 21:36:31] ----------
[2023-01-19 21:36:31] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 21:36:31] ----------
[2023-01-19 21:37:31] ----------
[2023-01-19 21:37:31] Checking OLM artifacts in project ibm-common-services
[2023-01-19 21:37:31] ----------
[2023-01-19 21:37:31] Collecting OLM information
[2023-01-19 21:37:32] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 21:37:32] INFO: Everything looks good
[2023-01-19 21:37:32] ----------
[2023-01-19 21:37:32] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 21:37:32] ----------
[2023-01-19 21:38:32] ----------
[2023-01-19 21:38:32] Checking OLM artifacts in project ibm-common-services
[2023-01-19 21:38:32] ----------
[2023-01-19 21:38:32] Collecting OLM information
[2023-01-19 21:38:33] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 21:38:33] INFO: Everything looks good
[2023-01-19 21:38:33] ----------
[2023-01-19 21:38:33] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 21:38:33] ----------
[2023-01-19 21:39:33] ----------
[2023-01-19 21:39:33] Checking OLM artifacts in project ibm-common-services
[2023-01-19 21:39:33] ----------
[2023-01-19 21:39:33] Collecting OLM information
[2023-01-19 21:39:34] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 21:39:34] INFO: Everything looks good
[2023-01-19 21:39:34] ----------
[2023-01-19 21:39:34] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 21:39:35] ----------
[2023-01-19 21:52:06] ----------
[2023-01-19 21:52:06] Info: Checking access to project ibm-common-services
[2023-01-19 21:52:06] ----------
[2023-01-19 21:52:06] Checking OLM artifacts in project ibm-common-services
[2023-01-19 21:52:06] ----------
[2023-01-19 21:52:06] Collecting OLM information
[2023-01-19 21:52:07] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 21:52:07] INFO: Everything looks good
[2023-01-19 21:52:07] ----------
[2023-01-19 21:52:07] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 21:52:07] ----------
[2023-01-19 21:53:07] ----------
[2023-01-19 21:53:07] Checking OLM artifacts in project ibm-common-services
[2023-01-19 21:53:07] ----------
[2023-01-19 21:53:07] Collecting OLM information
[2023-01-19 21:53:08] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 21:53:08] INFO: Everything looks good
[2023-01-19 21:53:08] ----------
[2023-01-19 21:53:08] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 21:53:08] ----------
[2023-01-19 21:54:08] ----------
[2023-01-19 21:54:08] Checking OLM artifacts in project ibm-common-services
[2023-01-19 21:54:08] ----------
[2023-01-19 21:54:08] Collecting OLM information
[2023-01-19 21:54:09] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 21:54:09] INFO: Everything looks good
[2023-01-19 21:54:09] ----------
[2023-01-19 21:54:09] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 21:54:09] ----------
[2023-01-19 21:55:09] ----------
[2023-01-19 21:55:09] Checking OLM artifacts in project ibm-common-services
[2023-01-19 21:55:09] ----------
[2023-01-19 21:55:09] Collecting OLM information
[2023-01-19 21:55:10] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 21:55:10] INFO: Everything looks good
[2023-01-19 21:55:10] ----------
[2023-01-19 21:55:10] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 21:55:10] ----------
[2023-01-19 21:56:10] ----------
[2023-01-19 21:56:10] Checking OLM artifacts in project ibm-common-services
[2023-01-19 21:56:10] ----------
[2023-01-19 21:56:10] Collecting OLM information
[2023-01-19 21:56:11] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 21:56:11] INFO: Everything looks good
[2023-01-19 21:56:11] ----------
[2023-01-19 21:56:11] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 21:56:11] ----------
[2023-01-19 21:57:11] ----------
[2023-01-19 21:57:11] Checking OLM artifacts in project ibm-common-services
[2023-01-19 21:57:11] ----------
[2023-01-19 21:57:11] Collecting OLM information
[2023-01-19 21:57:12] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 21:57:12] INFO: Everything looks good
[2023-01-19 21:57:12] ----------
[2023-01-19 21:57:12] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 21:57:12] ----------
[2023-01-19 21:58:12] ----------
[2023-01-19 21:58:12] Checking OLM artifacts in project ibm-common-services
[2023-01-19 21:58:12] ----------
[2023-01-19 21:58:12] Collecting OLM information
[2023-01-19 21:58:13] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 21:58:13] INFO: Everything looks good
[2023-01-19 21:58:13] ----------
[2023-01-19 21:58:14] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 21:58:14] ----------
[2023-01-19 21:59:14] ----------
[2023-01-19 21:59:14] Checking OLM artifacts in project ibm-common-services
[2023-01-19 21:59:14] ----------
[2023-01-19 21:59:14] Collecting OLM information
[2023-01-19 21:59:14] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 21:59:15] INFO: Everything looks good
[2023-01-19 21:59:15] ----------
[2023-01-19 21:59:15] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 21:59:15] ----------
[2023-01-19 22:00:15] ----------
[2023-01-19 22:00:15] Checking OLM artifacts in project ibm-common-services
[2023-01-19 22:00:15] ----------
[2023-01-19 22:00:15] Collecting OLM information
[2023-01-19 22:00:16] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 22:00:16] INFO: Everything looks good
[2023-01-19 22:00:16] ----------
[2023-01-19 22:00:16] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 22:00:16] ----------
[2023-01-19 22:01:16] ----------
[2023-01-19 22:01:16] Checking OLM artifacts in project ibm-common-services
[2023-01-19 22:01:16] ----------
[2023-01-19 22:01:16] Collecting OLM information
[2023-01-19 22:01:17] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 22:01:17] INFO: Everything looks good
[2023-01-19 22:01:17] ----------
[2023-01-19 22:01:17] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 22:01:17] ----------
[2023-01-19 22:02:17] ----------
[2023-01-19 22:02:17] Checking OLM artifacts in project ibm-common-services
[2023-01-19 22:02:17] ----------
[2023-01-19 22:02:17] Collecting OLM information
[2023-01-19 22:02:18] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 22:02:18] INFO: Everything looks good
[2023-01-19 22:02:18] ----------
[2023-01-19 22:02:18] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 22:02:18] ----------
[2023-01-19 22:14:57] ----------
[2023-01-19 22:14:57] Info: Checking access to project ibm-common-services
[2023-01-19 22:14:57] ----------
[2023-01-19 22:14:57] Checking OLM artifacts in project ibm-common-services
[2023-01-19 22:14:57] ----------
[2023-01-19 22:14:57] Collecting OLM information
[2023-01-19 22:14:58] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 22:14:59] INFO: Everything looks good
[2023-01-19 22:14:59] ----------
[2023-01-19 22:14:59] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 22:14:59] ----------
[2023-01-19 22:15:59] ----------
[2023-01-19 22:15:59] Checking OLM artifacts in project ibm-common-services
[2023-01-19 22:15:59] ----------
[2023-01-19 22:15:59] Collecting OLM information
[2023-01-19 22:16:00] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 22:16:00] INFO: Everything looks good
[2023-01-19 22:16:00] ----------
[2023-01-19 22:16:00] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 22:16:00] ----------
[2023-01-19 22:17:00] ----------
[2023-01-19 22:17:00] Checking OLM artifacts in project ibm-common-services
[2023-01-19 22:17:00] ----------
[2023-01-19 22:17:00] Collecting OLM information
[2023-01-19 22:17:01] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 22:17:01] INFO: Everything looks good
[2023-01-19 22:17:01] ----------
[2023-01-19 22:17:01] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 22:17:01] ----------
[2023-01-19 22:18:01] ----------
[2023-01-19 22:18:01] Checking OLM artifacts in project ibm-common-services
[2023-01-19 22:18:01] ----------
[2023-01-19 22:18:01] Collecting OLM information
[2023-01-19 22:18:02] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 22:18:02] INFO: Everything looks good
[2023-01-19 22:18:02] ----------
[2023-01-19 22:18:02] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 22:18:02] ----------
[2023-01-19 22:19:02] ----------
[2023-01-19 22:19:02] Checking OLM artifacts in project ibm-common-services
[2023-01-19 22:19:02] ----------
[2023-01-19 22:19:02] Collecting OLM information
[2023-01-19 22:19:03] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 22:19:03] INFO: Everything looks good
[2023-01-19 22:19:03] ----------
[2023-01-19 22:19:03] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 22:19:03] ----------
[2023-01-19 22:20:03] ----------
[2023-01-19 22:20:03] Checking OLM artifacts in project ibm-common-services
[2023-01-19 22:20:03] ----------
[2023-01-19 22:20:03] Collecting OLM information
[2023-01-19 22:20:04] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 22:20:04] INFO: Everything looks good
[2023-01-19 22:20:04] ----------
[2023-01-19 22:20:04] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 22:20:04] ----------
[2023-01-19 22:21:04] ----------
[2023-01-19 22:21:04] Checking OLM artifacts in project ibm-common-services
[2023-01-19 22:21:04] ----------
[2023-01-19 22:21:04] Collecting OLM information
[2023-01-19 22:21:05] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 22:21:05] INFO: Everything looks good
[2023-01-19 22:21:05] ----------
[2023-01-19 22:21:05] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 22:21:05] ----------
[2023-01-19 22:22:05] ----------
[2023-01-19 22:22:05] Checking OLM artifacts in project ibm-common-services
[2023-01-19 22:22:05] ----------
[2023-01-19 22:22:05] Collecting OLM information
[2023-01-19 22:22:06] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 22:22:06] INFO: Everything looks good
[2023-01-19 22:22:06] ----------
[2023-01-19 22:22:06] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 22:22:06] ----------
[2023-01-19 22:23:06] ----------
[2023-01-19 22:23:06] Checking OLM artifacts in project ibm-common-services
[2023-01-19 22:23:06] ----------
[2023-01-19 22:23:06] Collecting OLM information
[2023-01-19 22:23:07] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 22:23:07] INFO: Everything looks good
[2023-01-19 22:23:07] ----------
[2023-01-19 22:23:07] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 22:23:07] ----------
[2023-01-19 22:24:07] ----------
[2023-01-19 22:24:07] Checking OLM artifacts in project ibm-common-services
[2023-01-19 22:24:07] ----------
[2023-01-19 22:24:07] Collecting OLM information
[2023-01-19 22:24:08] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 22:24:08] INFO: Everything looks good
[2023-01-19 22:24:08] ----------
[2023-01-19 22:24:08] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 22:24:08] ----------
[2023-01-19 22:25:08] ----------
[2023-01-19 22:25:08] Checking OLM artifacts in project ibm-common-services
[2023-01-19 22:25:08] ----------
[2023-01-19 22:25:08] Collecting OLM information
[2023-01-19 22:25:09] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 22:25:09] INFO: Everything looks good
[2023-01-19 22:25:09] ----------
[2023-01-19 22:25:09] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 22:25:09] ----------
[2023-01-19 22:37:57] ----------
[2023-01-19 22:37:57] Info: Checking access to project ibm-common-services
[2023-01-19 22:37:57] ----------
[2023-01-19 22:37:57] Checking OLM artifacts in project ibm-common-services
[2023-01-19 22:37:57] ----------
[2023-01-19 22:37:57] Collecting OLM information
[2023-01-19 22:37:58] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 22:37:58] INFO: Everything looks good
[2023-01-19 22:37:58] ----------
[2023-01-19 22:37:58] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 22:37:58] ----------
[2023-01-19 22:38:58] ----------
[2023-01-19 22:38:58] Checking OLM artifacts in project ibm-common-services
[2023-01-19 22:38:58] ----------
[2023-01-19 22:38:58] Collecting OLM information
[2023-01-19 22:38:59] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 22:38:59] INFO: Everything looks good
[2023-01-19 22:38:59] ----------
[2023-01-19 22:38:59] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 22:38:59] ----------
[2023-01-19 22:39:59] ----------
[2023-01-19 22:39:59] Checking OLM artifacts in project ibm-common-services
[2023-01-19 22:39:59] ----------
[2023-01-19 22:39:59] Collecting OLM information
[2023-01-19 22:40:00] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 22:40:00] INFO: Everything looks good
[2023-01-19 22:40:00] ----------
[2023-01-19 22:40:00] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 22:40:00] ----------
[2023-01-19 22:41:00] ----------
[2023-01-19 22:41:00] Checking OLM artifacts in project ibm-common-services
[2023-01-19 22:41:00] ----------
[2023-01-19 22:41:00] Collecting OLM information
[2023-01-19 22:41:01] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 22:41:01] INFO: Everything looks good
[2023-01-19 22:41:01] ----------
[2023-01-19 22:41:01] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 22:41:01] ----------
[2023-01-19 22:42:01] ----------
[2023-01-19 22:42:01] Checking OLM artifacts in project ibm-common-services
[2023-01-19 22:42:01] ----------
[2023-01-19 22:42:01] Collecting OLM information
[2023-01-19 22:42:03] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 22:42:03] INFO: Everything looks good
[2023-01-19 22:42:03] ----------
[2023-01-19 22:42:03] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 22:42:03] ----------
[2023-01-19 22:43:03] ----------
[2023-01-19 22:43:03] Checking OLM artifacts in project ibm-common-services
[2023-01-19 22:43:03] ----------
[2023-01-19 22:43:03] Collecting OLM information
[2023-01-19 22:43:04] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 22:43:04] INFO: Everything looks good
[2023-01-19 22:43:04] ----------
[2023-01-19 22:43:04] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 22:43:04] ----------
[2023-01-19 22:44:04] ----------
[2023-01-19 22:44:04] Checking OLM artifacts in project ibm-common-services
[2023-01-19 22:44:04] ----------
[2023-01-19 22:44:04] Collecting OLM information
[2023-01-19 22:44:05] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 22:44:05] INFO: Everything looks good
[2023-01-19 22:44:05] ----------
[2023-01-19 22:44:05] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 22:44:05] ----------
[2023-01-19 22:45:05] ----------
[2023-01-19 22:45:05] Checking OLM artifacts in project ibm-common-services
[2023-01-19 22:45:05] ----------
[2023-01-19 22:45:05] Collecting OLM information
[2023-01-19 22:45:06] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 22:45:06] INFO: Everything looks good
[2023-01-19 22:45:06] ----------
[2023-01-19 22:45:06] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 22:45:06] ----------
[2023-01-19 22:46:06] ----------
[2023-01-19 22:46:06] Checking OLM artifacts in project ibm-common-services
[2023-01-19 22:46:06] ----------
[2023-01-19 22:46:06] Collecting OLM information
[2023-01-19 22:46:07] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 22:46:07] INFO: Everything looks good
[2023-01-19 22:46:07] ----------
[2023-01-19 22:46:07] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 22:46:07] ----------
[2023-01-19 22:47:07] ----------
[2023-01-19 22:47:07] Checking OLM artifacts in project ibm-common-services
[2023-01-19 22:47:07] ----------
[2023-01-19 22:47:07] Collecting OLM information
[2023-01-19 22:47:08] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 22:47:08] INFO: Everything looks good
[2023-01-19 22:47:08] ----------
[2023-01-19 22:47:08] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 22:47:08] ----------
[2023-01-19 22:48:08] ----------
[2023-01-19 22:48:08] Checking OLM artifacts in project ibm-common-services
[2023-01-19 22:48:08] ----------
[2023-01-19 22:48:08] Collecting OLM information
[2023-01-19 22:48:09] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 22:48:10] INFO: Everything looks good
[2023-01-19 22:48:10] ----------
[2023-01-19 22:48:10] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 22:48:10] ----------
[2023-01-19 23:01:51] ----------
[2023-01-19 23:01:51] Info: Checking access to project ibm-common-services
[2023-01-19 23:01:51] ----------
[2023-01-19 23:01:51] Checking OLM artifacts in project ibm-common-services
[2023-01-19 23:01:51] ----------
[2023-01-19 23:01:51] Collecting OLM information
[2023-01-19 23:01:52] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 23:01:52] INFO: Everything looks good
[2023-01-19 23:01:52] ----------
[2023-01-19 23:01:52] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 23:01:52] ----------
[2023-01-19 23:02:52] ----------
[2023-01-19 23:02:52] Checking OLM artifacts in project ibm-common-services
[2023-01-19 23:02:52] ----------
[2023-01-19 23:02:52] Collecting OLM information
[2023-01-19 23:02:53] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 23:02:53] INFO: Everything looks good
[2023-01-19 23:02:53] ----------
[2023-01-19 23:02:53] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 23:02:53] ----------
[2023-01-19 23:03:53] ----------
[2023-01-19 23:03:54] Checking OLM artifacts in project ibm-common-services
[2023-01-19 23:03:54] ----------
[2023-01-19 23:03:54] Collecting OLM information
[2023-01-19 23:03:55] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 23:03:55] INFO: Everything looks good
[2023-01-19 23:03:55] ----------
[2023-01-19 23:03:55] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 23:03:55] ----------
[2023-01-19 23:04:55] ----------
[2023-01-19 23:04:55] Checking OLM artifacts in project ibm-common-services
[2023-01-19 23:04:55] ----------
[2023-01-19 23:04:55] Collecting OLM information
[2023-01-19 23:04:56] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 23:04:56] INFO: Everything looks good
[2023-01-19 23:04:56] ----------
[2023-01-19 23:04:56] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 23:04:56] ----------
[2023-01-19 23:05:56] ----------
[2023-01-19 23:05:56] Checking OLM artifacts in project ibm-common-services
[2023-01-19 23:05:56] ----------
[2023-01-19 23:05:56] Collecting OLM information
[2023-01-19 23:05:57] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 23:05:57] INFO: Everything looks good
[2023-01-19 23:05:57] ----------
[2023-01-19 23:05:57] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 23:05:57] ----------
[2023-01-19 23:06:57] ----------
[2023-01-19 23:06:57] Checking OLM artifacts in project ibm-common-services
[2023-01-19 23:06:57] ----------
[2023-01-19 23:06:57] Collecting OLM information
[2023-01-19 23:06:58] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 23:06:58] INFO: Everything looks good
[2023-01-19 23:06:58] ----------
[2023-01-19 23:06:58] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 23:06:58] ----------
[2023-01-19 23:07:58] ----------
[2023-01-19 23:07:58] Checking OLM artifacts in project ibm-common-services
[2023-01-19 23:07:58] ----------
[2023-01-19 23:07:58] Collecting OLM information
[2023-01-19 23:07:59] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 23:07:59] INFO: Everything looks good
[2023-01-19 23:07:59] ----------
[2023-01-19 23:07:59] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 23:07:59] ----------
[2023-01-19 23:08:59] ----------
[2023-01-19 23:08:59] Checking OLM artifacts in project ibm-common-services
[2023-01-19 23:08:59] ----------
[2023-01-19 23:08:59] Collecting OLM information
[2023-01-19 23:09:00] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 23:09:00] INFO: Everything looks good
[2023-01-19 23:09:00] ----------
[2023-01-19 23:09:00] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 23:09:00] ----------
[2023-01-19 23:10:00] ----------
[2023-01-19 23:10:00] Checking OLM artifacts in project ibm-common-services
[2023-01-19 23:10:00] ----------
[2023-01-19 23:10:00] Collecting OLM information
[2023-01-19 23:10:01] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 23:10:02] INFO: Everything looks good
[2023-01-19 23:10:02] ----------
[2023-01-19 23:10:02] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 23:10:02] ----------
[2023-01-19 23:11:02] ----------
[2023-01-19 23:11:02] Checking OLM artifacts in project ibm-common-services
[2023-01-19 23:11:02] ----------
[2023-01-19 23:11:02] Collecting OLM information
[2023-01-19 23:11:03] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 23:11:03] INFO: Everything looks good
[2023-01-19 23:11:03] ----------
[2023-01-19 23:11:03] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 23:11:03] ----------
[2023-01-19 23:12:03] ----------
[2023-01-19 23:12:03] Checking OLM artifacts in project ibm-common-services
[2023-01-19 23:12:03] ----------
[2023-01-19 23:12:03] Collecting OLM information
[2023-01-19 23:12:04] INFO: Checking all subscriptions in OpenShift project ibm-common-services
[2023-01-19 23:12:04] INFO: Everything looks good
[2023-01-19 23:12:04] ----------
[2023-01-19 23:12:04] Finished checking OLM artifacts in project ibm-common-services, sleeping for 60 seconds
[2023-01-19 23:12:04] ----------
fketelaars commented 1 year ago

@triceam Now that you have selected the correct service (wkc) and installation went through, can this issue be closed? I will test the watson-ks installation as well, but this was successful as part of the CP4D 4.6.1 testing.

triceam commented 1 year ago

I'm still running into this same issue on a regular basis... I'm not sure how we should handle it, to be honest. The underlying issue is between Cloud Paks/cpd-cli and OpenShift olm, this is just a side effect of deploying using the deployer.

fketelaars commented 1 year ago

@triceam I know, the OLM issues are frustrating and I'm afraid we have to work through them one by one. I hit one #287 that I have not been able to reproduce since that time. Still waiting for this issue to occur again. If OLM gets stuck next time, can you give me access to your cluster so I can do some more research and try to remediate?

triceam commented 1 year ago

I saw a notification today... it looks like RedHat will be pushing a potential fix in 4.10.46. Let's hope that does the trick.

see:

triceam commented 1 year ago

OpenShift/OLM fix was released by RedHat. I have tested on a new ROKS cluster with workers at 4.10.50_1554, and it got past the operator subscriptions without any issue.

Issue: https://issues.redhat.com//browse/OCPBUGS-4128 Fixed in https://access.redhat.com/errata/RHBA-2022:9099