[BUG][DEPLOY] Failed to pull image quay.io/openshift/origin-oauth-proxy for management-ingress pod.

[giannisalinetti]

Describe the bug A clear and concise description of what the bug is. During deployment of ACM the managment-ingress pod cannot start with an ImagePullBackoff with an ErrImagePull for the image quay.io/openshift/origin-oauth-proxy.

To Reproduce Steps to reproduce the behavior:

1. Run the start.sh script
2. Wait for deployment of all pods to complete.
3. Check that all pods are running excpept correctly the managent-ingress.

Expected behavior All pods are running ore completed with no errors.

Screenshots

$ oc get pods -n open-cluster-management
NAME                                                              READY   STATUS             RESTARTS   AGE
acm-custom-registry-544ddf4f84-75997                              1/1     Running            0          12h
application-chart-1e0db-applicationui-7466647bb6-v6xp2            1/1     Running            0          21m
b204aeeeb5e8e89542b70ae24092621f8a5d905bd698700ca6b9d41de0kxqqf   0/1     Completed          0          32m
cert-manager-1098a-54c7cd774-mjd27                                1/1     Running            0          21m
cert-manager-webhook-c5a7a-cainjector-8484b449dd-zfcc6            1/1     Running            0          21m
cert-manager-webhook-f97bdb9fd-6jwhd                              1/1     Running            1          21m
configmap-watcher-f9ec5-8647dc6946-h2mzg                          1/1     Running            0          21m
console-chart-0f5e7-consoleapi-866c88ff58-pc5qx                   1/1     Running            0          19m
console-chart-0f5e7-consoleui-5df747b8ff-fj556                    1/1     Running            0          19m
console-header-856c64fc64-rbccd                                   1/1     Running            0          19m
etcd-cluster-4f4p6c6pwv                                           1/1     Running            0          21m
etcd-cluster-fhnhwdgxrq                                           1/1     Running            0          19m
etcd-cluster-zn5w9hnkh2                                           1/1     Running            0          21m
etcd-operator-558567f79d-fs9gn                                    3/3     Running            0          32m
grc-7fba5-grcui-697f87f79b-q98kk                                  1/1     Running            0          18m
grc-7fba5-grcuiapi-f55595748-ztwnh                                1/1     Running            0          18m
grc-7fba5-policy-propogator-8cb9ff78d-s8cl5                       1/1     Running            0          18m
hive-operator-fbcf446c5-bgl9n                                     1/1     Running            0          31m
kui-web-terminal-6fb99c7fbb-gm5mc                                 1/1     Running            0          21m
kui-web-terminal-75f6cb7f4d-69cw2                                 0/1     Pending            0          17m
management-ingress-48ccc-6648d5fbdf-rt78l                         1/2     ImagePullBackOff   0          17m
mcm-apiserver-648989b9f4-lkdvt                                    0/1     Pending            0          17m
mcm-apiserver-9fdff5f69-bhppr                                     1/1     Running            0          20m
mcm-controller-57955b6c66-28p7k                                   1/1     Running            0          21m
mcm-webhook-794f8d575-9vpst                                       1/1     Running            0          21m
multicluster-mongodb-0                                            1/1     Running            0          18m
multicluster-operators-application-6d7bf76cd8-jfscx               4/4     Running            3          31m
multicluster-operators-hub-subscription-65b48f6bb-zfjr9           1/1     Running            0          31m
multicluster-operators-standalone-subscription-8496f67d7-sqlkg    1/1     Running            0          31m
multiclusterhub-operator-5474547c66-shjfh                         1/1     Running            0          31m
multiclusterhub-repo-6cd66688cc-gvlwx                             1/1     Running            0          22m
rcm-controller-544b7cdd6b-v9n4c                                   1/1     Running            3          21m
search-operator-958dd6d67-m6nz5                                   1/1     Running            0          19m
search-prod-73f12-redisgraph-6cbcf6c958-s8j5x                     1/1     Running            0          17m
search-prod-73f12-search-aggregator-cbb6f96bc-78cxd               1/1     Running            0          17m
search-prod-73f12-search-api-5bd9d57669-wsbtg                     1/1     Running            0          17m
search-prod-73f12-search-collector-6f7cfffcdb-dg7mj               1/1     Running            0          17m
topology-33ea9-topology-56d466bfc9-hgtb8                          1/1     Running            0          19m
topology-33ea9-topologyapi-584d474477-bdlhr                       1/1     Running            0          19m

Desktop (please complete the following information):

• OS: [rhel]
• Browser [e.g. chrome, safari, firefox]
• Snapshot [1.0.0-SNAPSHOT-2020-05-04-17-43-49]

Additional context The problem looks related to the wrong SHA256 of the image. I used the following temporary workaround:

• Pull the image with podman or inspect it with skopeo skopeo inspect docker://quay.io/openshift/origin-oauth-proxy | grep Digest
• Grab the SHA256 hash and replace it in the Deployment resource of the management-ingress under spec.template.spec.container. oc edit deployment/managment-ingress-XXXXX
• Wait for the management-ingress pod to rollout again

[tpouyer]

The underlying origin-oauth-proxy image we used when building this shapshot has been removed from the repo quay.io/openshift/origin-oauth-proxy which is causing the breakage.

Going forward we will need to ensure that we mirror any origin-oauth-proxy image we use as part of a snapshot over to the quay.io/open-cluster-management/origin-oauth-proxy repo to ensure that it never goes away.

In the meantime we will try to figure out how best to proceed with "repairing" the broken snapshots. More than likely we will simply produce a new snapshot with an updated origin-oauth-proxy image and ensure that we mirror it into quay.io/open-cluster-management/origin-oauth-proxy repo.