Failed installation ends with exit code 0

morenod commented 3 years ago

Need to check this installation, job is marked as sucess, but installation have failed.

Exit code is 0 somewhere where it should be 1

https://jenkins.scalelab.redhat.com/job/SCALE-CI-MS-ROSA/14/console

chentex commented 3 years ago

This error happens when for some reason the key and secret being used is not from the correct account, or it has been deleted.

This exactly is the block where we are ignoring the error code:

https://github.com/cloud-bulldozer/perfscale-managed-services/blob/main/osde2e-wrapper.py#L160:L180

My guess would be to add it there, but I don't know if this will affect other scenarios.

morenod commented 3 years ago

Another example of failed installation (_build_cluster function ends on ERROR) but job is success:

12:02:46 2021-03-02 11:02:46.817 ERROR osde2e-wrapper - _build_cluster: Failed to build cluster 2: mrnd-hv1-0002
12:02:46 2021-03-02 11:02:46.817 INFO osde2e-wrapper - _build_cluster: Attempting to load metadata json
12:02:46 2021-03-02 11:02:46.818 INFO osde2e-wrapper - _download_kubeconfig: Attempting to load metadata json
12:02:46 2021-03-02 11:02:46.818 INFO osde2e-wrapper - _download_kubeconfig: Downloading kubeconfig file for cluster 1j5slur1ld9iosfibnfk4vh92a5p6rhu on /var/jenkins/run-20210302-105101/mrnd-hv1-0002/1j5slur1ld9iosfibnfk4vh92a5p6rhu
12:02:46 2021-03-02 11:02:46.818 DEBUG osde2e-wrapper - _download_kubeconfig: ['/var/jenkins/run-20210302-105101/osde2e/out/osde2ectl', '--custom-config', 'cluster_account.yaml', 'get', '-k', '-i', '1j5slur1ld9iosfibnfk4vh92a5p6rhu', '--kube-config-path', '/var/jenkins/run-20210302-105101/mrnd-hv1-0002/1j5slur1ld9iosfibnfk4vh92a5p6rhu']
12:02:47 2021-03-02 11:02:46.977 ERROR osde2e-wrapper - _build_cluster: Failed to build cluster 1: mrnd-hv1-0001
12:02:47 2021-03-02 11:02:46.978 INFO osde2e-wrapper - _build_cluster: Attempting to load metadata json
12:02:47 2021-03-02 11:02:46.979 INFO osde2e-wrapper - _download_kubeconfig: Attempting to load metadata json
12:02:47 2021-03-02 11:02:46.979 INFO osde2e-wrapper - _download_kubeconfig: Downloading kubeconfig file for cluster 1j5sluo1gtlgckqjusdmfo5be5pd8jcn on /var/jenkins/run-20210302-105101/mrnd-hv1-0001/1j5sluo1gtlgckqjusdmfo5be5pd8jcn
12:02:47 2021-03-02 11:02:46.979 DEBUG osde2e-wrapper - _download_kubeconfig: ['/var/jenkins/run-20210302-105101/osde2e/out/osde2ectl', '--custom-config', 'cluster_account.yaml', 'get', '-k', '-i', '1j5sluo1gtlgckqjusdmfo5be5pd8jcn', '--kube-config-path', '/var/jenkins/run-20210302-105101/mrnd-hv1-0001/1j5sluo1gtlgckqjusdmfo5be5pd8jcn']
12:03:21 2021-03-02 11:03:21.502 INFO osde2e-wrapper - _watcher: Watcher exiting
12:03:21 INFO: Compress logs before downloading
12:03:24 INFO: Downloading /var/jenkins/run-20210302-105101 folder from 35.163.247.220 to be archieved as artifacts
12:03:27 receiving incremental file list
12:03:27 created directory /var/jenkins_home/workspace/SCALE-CI-MS-OSDE2E/artifacts
12:03:27 ./
12:03:27 installation.log.gz
12:03:27 uuid
12:03:27 mrnd-hv1-0001/
12:03:27 mrnd-hv1-0001/1j5sluo1gtlgckqjusdmfo5be5pd8jcn..metrics.prom
12:03:28 mrnd-hv1-0001/custom-prow-metadata.json
12:03:28 mrnd-hv1-0001/installation.log.gz
12:03:28 mrnd-hv1-0001/metadata.json
12:03:28 mrnd-hv1-0001/test_output.log
12:03:28 mrnd-hv1-0001/1j5sluo1gtlgckqjusdmfo5be5pd8jcn/
12:03:28 mrnd-hv1-0001/1j5sluo1gtlgckqjusdmfo5be5pd8jcn/mrnd-hv1-0001-kubeconfig.txt
12:03:28 mrnd-hv1-0001/install/
12:03:28 mrnd-hv1-0001/install/junit_beforesuite.xml
12:03:28 mrnd-hv1-0001/install/junit_j4lk8.xml
12:03:28 mrnd-hv1-0001/install/junit_logmetrics.xml
12:03:28 mrnd-hv1-0002/
12:03:28 mrnd-hv1-0002/1j5slur1ld9iosfibnfk4vh92a5p6rhu..metrics.prom
12:03:28 mrnd-hv1-0002/custom-prow-metadata.json
12:03:28 mrnd-hv1-0002/install-log.txt
12:03:28 mrnd-hv1-0002/installation.log.gz
12:03:28 mrnd-hv1-0002/metadata.json
12:03:28 mrnd-hv1-0002/test_output.log
12:03:28 mrnd-hv1-0002/uninstall-log.txt
12:03:28 mrnd-hv1-0002/1j5slur1ld9iosfibnfk4vh92a5p6rhu/
12:03:28 mrnd-hv1-0002/1j5slur1ld9iosfibnfk4vh92a5p6rhu/mrnd-hv1-0002-kubeconfig.txt
12:03:28 mrnd-hv1-0002/install/
12:03:28 mrnd-hv1-0002/install/junit_beforesuite.xml
12:03:28 mrnd-hv1-0002/install/junit_logmetrics.xml
12:03:28 mrnd-hv1-0002/install/junit_wb333.xml
12:03:28 
12:03:28 sent 590 bytes  received 312.06K bytes  69.48K bytes/sec
12:03:28 total size is 309.87K  speedup is 0.99
12:03:29 Archiving artifacts
12:03:29 [WS-CLEANUP] Deleting project workspace...
12:03:29 [WS-CLEANUP] Deferred wipeout is used...
12:03:29 [WS-CLEANUP] done
12:03:29 Finished: SUCCESS

cloud-bulldozer / perfscale-managed-services

Failed installation ends with exit code 0 #40