cyberark / kubernetes-conjur-deploy

Scripts for deploying DAP followers to Kubernetes and OpenShift given an existing DAP master cluster
Apache License 2.0
14 stars 15 forks source link

Jenkins tests for OC 4.3 sometimes hangs for namespace deletion #167

Closed diverdane closed 3 years ago

diverdane commented 3 years ago

Summary

The Jenkins pipeline test for OpenShift 4.3 sometimes hangs when the scripts try to clean up the Conjur namespace. The console logs for this failure look like this:

[2021-01-25T14:39:25.466Z] Logging in as cluster admin...
[2021-01-25T14:39:25.730Z] Login successful.
[2021-01-25T14:39:25.730Z] 
[2021-01-25T14:39:25.730Z] You have access to 63 projects, the list has been suppressed. You can list all projects with 'oc projects'
[2021-01-25T14:39:25.730Z] 
[2021-01-25T14:39:25.730Z] Using project "conjur-deploy-test-03eb13bd-9".
[2021-01-25T14:39:25.730Z] + set_namespace default
[2021-01-25T14:39:25.730Z] + [[ 1 != 1 ]]
[2021-01-25T14:39:25.730Z] ++ oc config current-context
[2021-01-25T14:39:25.730Z] + oc config set-context conjur-deploy-test-03eb13bd-9/api-openshift-43-itci-conjur-net:6443/kube:admin --namespace=default
[2021-01-25T14:39:25.992Z] + has_namespace conjur-deploy-test-03eb13bd-9
[2021-01-25T14:39:25.993Z] + oc get namespace conjur-deploy-test-03eb13bd-9
[2021-01-25T14:39:25.993Z] + true
[2021-01-25T14:39:25.993Z] + oc delete namespace conjur-deploy-test-03eb13bd-9
[2021-01-25T14:39:26.253Z] namespace "conjur-deploy-test-03eb13bd-9" deleted

           < = = = HANGING HERE = = = >

This does not happen for other versions of OpenShift.

This appears to be related to this OC 4.3 bug: https://bugzilla.redhat.com/show_bug.cgi?id=1798282 and can likely be fixed by deleting finalizers on Kubernetes services before deleting the corresponding namespace.

Steps to Reproduce

Look at an example failure log here: https://jenkins.conjur.net/blue/organizations/jenkins/cyberark--kubernetes-conjur-deploy/detail/master/658/pipeline/22

Expected Results

Tests should not hang for namespace deletion.

Actual Results (including error logs, if applicable)

See description above.

Reproducible

Version/Tag number

Latest master.

Environment setup

OpenShift 4.3.

Additional Information

This appears to be related to this OC 4.3 bug: https://bugzilla.redhat.com/show_bug.cgi?id=1798282 and can likely be fixed by deleting finalizers on Kubernetes services before deleting the corresponding namespace.