search

ibm-mas / cli

Maximo Application Suite command line interface
https://ibm-mas.github.io/cli/
Eclipse Public License 2.0
21 stars 16 forks source link

Missing mas-install pipeline with latest CLI v10.0.5 #1098

Closed rene-oromtz closed 1 week ago

rene-oromtz commented 1 week ago

CLI Version

IBM Maximo Application Suite Admin CLI v10.0.5

MAS CLI Input

20) Launch Install
If you are using storage classes that utilize 'WaitForFirstConsumer' binding mode choose 'No' at the prompt below
Wait for PVCs to bind? [y/n] y
✅️ OpenShift Pipelines Operator is installed and ready to use
✅️ Namespace is ready (mas-hcparft-pipelines)
✅️ MAS CLI image deployment test completed
✅️ Latest Tekton definitions are installed (v10.0.5)
✅️ PipelineRun for hcparft install submitted

Expected Behaviour

Pipelines to be installed

Actual Behavior

Missing mas-install pipeline. Only installed mas-update and mas-upgrade

oc get pipeline
NAME          AGE
mas-update    6m27s
mas-upgrade   6m27s

Following error is present on pipeline run: Error retrieving pipeline for pipelinerun mas-hcparft-pipelines/hcparft-install-240705-1803: error when getting Pipeline for PipelineRun hcparft-install-240705-1803: tekton client cannot get pipeline mas-install from local cluster: pipelines.tekton.dev "mas-install" not found

Installing with MASCLI v9.5.0 installs successfully the following pipelines:

NAME            AGE
mas-db2         78s
mas-install     78s
mas-ivt         78s
mas-rollback    78s
mas-uninstall   78s
mas-update      77s
mas-upgrade     77s
durera commented 1 week ago

Can you check in the mas.log file, I would assume there is an error in there that was generated when the pipelines were being installed, that we should have handled better and failed the "install tekton definitions" step here, I've not been able to reproduce this; everytime I run the install all the pipeline definitions are successfully installed.

durera commented 1 week ago

image

The mas.log should contain something like this:

2024-07-05 21:06:24,489   DEBUG    secret/pipeline-sls-entitlement created
2024-07-05 21:07:42,158   DEBUG    task.tekton.dev/mas-devops-appconnect created
2024-07-05 21:07:42,158   DEBUG    task.tekton.dev/mas-devops-arcgis created
2024-07-05 21:07:42,158   DEBUG    task.tekton.dev/mas-devops-cert-manager created
2024-07-05 21:07:42,158   DEBUG    task.tekton.dev/mas-devops-common-services created
2024-07-05 21:07:42,158   DEBUG    task.tekton.dev/mas-devops-cos-deprovision created
2024-07-05 21:07:42,158   DEBUG    task.tekton.dev/mas-devops-cos created
2024-07-05 21:07:42,158   DEBUG    task.tekton.dev/mas-devops-cp4d-service-update created
2024-07-05 21:07:42,158   DEBUG    task.tekton.dev/mas-devops-cp4d-service created
2024-07-05 21:07:42,158   DEBUG    task.tekton.dev/mas-devops-cp4d created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-db2 created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-eck created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-gencfg-workspace created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-grafana created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-ibm-catalogs created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-kafka-deprovision created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-kafka created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-launchivt created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-mongodb created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-must-gather created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-nvidia-gpu created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-ocp-deprovision created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-ocp-verify-all created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-ocp-verify created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-ocs created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/sls-registry-update created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-sls created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-suite-app-config created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-suite-app-install created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-suite-app-rollback created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-suite-app-uninstall created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-suite-app-upgrade created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-suite-app-verify created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-suite-certs created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-suite-config created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-suite-db2-setup-for-manage created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-suite-dns created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-suite-install created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-suite-rollback created
2024-07-05 21:07:42,160   DEBUG    task.tekton.dev/mas-devops-suite-uninstall created
2024-07-05 21:07:42,160   DEBUG    task.tekton.dev/mas-devops-suite-upgrade created
2024-07-05 21:07:42,160   DEBUG    task.tekton.dev/mas-devops-suite-verify created
2024-07-05 21:07:42,160   DEBUG    task.tekton.dev/mas-devops-turbonomic created
2024-07-05 21:07:42,160   DEBUG    task.tekton.dev/mas-devops-uds created
2024-07-05 21:07:42,160   DEBUG    task.tekton.dev/mas-devops-update-configmap created
2024-07-05 21:07:42,160   DEBUG    task.tekton.dev/mas-devops-wait-for-configmap created
2024-07-05 21:07:42,160   DEBUG    task.tekton.dev/mas-devops-wait-for-tekton created
2024-07-05 21:07:42,160   DEBUG    pipeline.tekton.dev/mas-db2 created
2024-07-05 21:07:42,160   DEBUG    pipeline.tekton.dev/mas-install created
2024-07-05 21:07:42,160   DEBUG    pipeline.tekton.dev/mas-ivt created
2024-07-05 21:07:42,160   DEBUG    pipeline.tekton.dev/mas-rollback created
2024-07-05 21:07:42,160   DEBUG    pipeline.tekton.dev/mas-uninstall created
2024-07-05 21:07:42,160   DEBUG    pipeline.tekton.dev/mas-update created
2024-07-05 21:07:42,160   DEBUG    pipeline.tekton.dev/mas-upgrade created

And I would hope that there would be a reason logged in there why only some of the definitions were installed.

rene-oromtz commented 1 week ago

Unfortunately I think I terminated the pod which had containing this mas.log.

I tried to recreate the issue on a different environment but without success, in this new environment, the pipelines where installed succesfully:

...
2024-07-05 21:55:50,660   DEBUG    pipeline.tekton.dev/mas-db2 created
2024-07-05 21:55:50,660   DEBUG    pipeline.tekton.dev/mas-install created
2024-07-05 21:55:50,660   DEBUG    pipeline.tekton.dev/mas-ivt created
2024-07-05 21:55:50,661   DEBUG    pipeline.tekton.dev/mas-rollback created
2024-07-05 21:55:50,661   DEBUG    pipeline.tekton.dev/mas-uninstall created
2024-07-05 21:55:50,661   DEBUG    pipeline.tekton.dev/mas-update created
2024-07-05 21:55:50,661   DEBUG    pipeline.tekton.dev/mas-upgrade created
2024-07-05 21:55:50,896   DEBUG    ---
apiVersion: tekton.dev/v1beta1

Only difference between one and the other is that the successful was a VPC on IBM Cloud and the failed cluster was behind a firewall. I'll try to see if I can relaunch on the other environment and validate if this is reproducible

durera commented 1 week ago

Thanks, you should get this error message logged:

    except KubectlCommandError as e:
        logger.warning(f"Error: Unable to install/update Tekton definitions: {e}")
        return False

but I can see that the error handling here is poor, so we will use this issue to track improving that ...

            with Halo(text=f'Installing latest Tekton definitions (v{self.version})', spinner=self.spinner) as h:
                updateTektonDefinitions(pipelinesNamespace, self.tektonDefsPath)
                h.stop_and_persist(symbol=self.successIcon, text=f"Latest Tekton definitions are installed (v{self.version})")

We're not event checking whether the Tekton definitions are successfully applied, I think it's because of a disconnect between the expectation that this function would throw an exception, but instead it catches the exception itself and just returns False.

We'll get that fixed next week so that, in case where this fails for whatever reason, it will fail earlier and prompt the user to check mas.log for the failure reason.

rene-oromtz commented 1 week ago

I just retried on the cluster were I faced this issue (I first had to clean all MAS references) and was unable to recreate this issue, I even retried with CLI v10.0.5:

2024-07-05 23:43:29,594   DEBUG    pipeline.tekton.dev/mas-db2 created
2024-07-05 23:43:29,595   DEBUG    pipeline.tekton.dev/mas-install created
2024-07-05 23:43:29,595   DEBUG    pipeline.tekton.dev/mas-ivt created
2024-07-05 23:43:29,595   DEBUG    pipeline.tekton.dev/mas-rollback created
2024-07-05 23:43:29,595   DEBUG    pipeline.tekton.dev/mas-uninstall created
2024-07-05 23:43:29,595   DEBUG    pipeline.tekton.dev/mas-update created
2024-07-05 23:43:29,595   DEBUG    pipeline.tekton.dev/mas-upgrade created
2024-07-05 23:43:29,818   DEBUG    ---

Seems weird that it failed on the first time but agree if you want to keep this issue open for tracking enhancement to error handling. Thanks for taking a look.

durera commented 1 week ago

This should be addressed by: https://github.com/ibm-mas/python-devops/commit/7294f17195778b6f1448049c2fda621e5df95493 and #1105