search

openshift / hypershift

Hyperscale OpenShift - clusters with hosted control planes
https://hypershift-docs.netlify.app
Apache License 2.0
434 stars 319 forks source link

GA Operations: Investigation: MCS pod download leaks/extra restarts #516

Closed relyt0925 closed 2 years ago

relyt0925 commented 3 years ago

Replication pattern: Delete all ignition-server pods at once with kubectl delete pods -n NAMESPACE -l app=ignition-server

Tylers-MacBook-Pro:armada tylerlisowski$ kubectl get pods -n master-prtesting49-57190034
NAME                                                 READY   STATUS             RESTARTS   AGE
catalog-operator-754b6dc8f7-d82vd                    1/1     Running            4          11h
certified-operators-catalog-57d5b9775d-7zp59         1/1     Running            0          88m
certified-operators-catalog-rollout-27215316-tt7vf   0/1     Completed          0          88m
cloud-controller-manager-ddc49497f-4669r             1/1     Running            1          11h
cloud-controller-manager-ddc49497f-gpkts             1/1     Running            0          11h
cloud-controller-manager-ddc49497f-hlfsn             1/1     Running            0          11h
cluster-api-78697c5c98-4wj4f                         1/1     Running            0          11h
cluster-api-78697c5c98-ptsqf                         1/1     Running            0          11h
cluster-api-78697c5c98-v5t8h                         1/1     Running            0          11h
cluster-autoscaler-f6f5cb844-xxmsl                   1/1     Running            1          11h
cluster-health-7b8b9b944d-jb7nh                      0/1     CrashLoopBackOff   136        10h
cluster-policy-controller-5bdd77ccfc-dltbf           1/1     Running            0          11h
cluster-policy-controller-5bdd77ccfc-mkshk           1/1     Running            0          11h
cluster-policy-controller-5bdd77ccfc-wvsg2           1/1     Running            0          11h
cluster-version-operator-857bd8cc5c-xjn5r            1/1     Running            0          11h
community-operators-catalog-86c89bcd7d-r89mf         1/1     Running            0          88m
community-operators-catalog-rollout-27215316-t4psf   0/1     Completed          0          88m
control-plane-operator-f489d8c94-vs548               1/1     Running            0          11h
etcd-operator-f8c5c4f5-5cr8p                         4/4     Running            4          11h
etcd-operator-f8c5c4f5-p6qtq                         4/4     Running            4          11h
etcd-operator-f8c5c4f5-qwb5s                         4/4     Running            4          11h
etcd-prtesting49-57190034-6lc77s9fln                 2/2     Running            0          11h
etcd-prtesting49-57190034-7nx2w75jvd                 2/2     Running            0          11h
etcd-prtesting49-57190034-br8f2b7krk                 2/2     Running            0          11h
etcd-restore-operator-cert-creator-gz4zl             0/1     Completed          0          11h
hosted-cluster-config-operator-664f4b4fb9-tgfdf      1/1     Running            3          11h
ignition-server-5d6d87fb75-nb8bd                     0/1     Running            0          18s
ignition-server-5d6d87fb75-qbr5w                     0/1     Running            0          18s
ignition-server-5d6d87fb75-wh92h                     0/1     Running            0          18s
konnectivity-agent-84d4545d7-vljwj                   1/1     Running            0          11h
konnectivity-agent-84d4545d7-wdmfk                   1/1     Running            0          11h
konnectivity-agent-84d4545d7-znrl9                   1/1     Running            0          11h
konnectivity-server-778479cc58-p5dx4                 1/1     Running            0          11h
kube-apiserver-677b5667bc-4pd2j                      2/2     Running            0          11h
kube-apiserver-677b5667bc-588hh                      2/2     Running            0          11h
kube-apiserver-677b5667bc-9jdqk                      2/2     Running            0          11h
kube-controller-manager-6d4f6c5b8d-jxm29             1/1     Running            0          10h
kube-controller-manager-6d4f6c5b8d-sg9j9             1/1     Running            0          10h
kube-controller-manager-6d4f6c5b8d-ths5c             1/1     Running            0          10h
kube-scheduler-c97c77d6f-2rtnm                       1/1     Running            0          11h
kube-scheduler-c97c77d6f-f72w5                       1/1     Running            0          11h
kube-scheduler-c97c77d6f-fbhfw                       1/1     Running            0          11h
machine-config-server-1236201992                     0/1     Init:0/3           0          3s
machine-config-server-291020420                      0/1     Init:1/3           0          4s
machine-config-server-446334720                      1/1     Running            0          8s
manifests-bootstrapper                               0/1     Completed          3          11h
oauth-openshift-855b8c4bfc-7v4rp                     1/1     Running            0          11h
oauth-openshift-855b8c4bfc-cx7z4                     1/1     Running            0          11h
oauth-openshift-855b8c4bfc-kl57h                     1/1     Running            0          11h
olm-operator-56b5bc7747-ntmxz                        1/1     Running            4          11h
openshift-apiserver-8489b9bb65-rp7vg                 2/2     Running            0          11h
openshift-apiserver-8489b9bb65-tj62l                 2/2     Running            0          11h
openshift-apiserver-8489b9bb65-xtlhz                 2/2     Running            0          11h
openshift-controller-manager-7bcc868796-7vmtb        1/1     Running            0          11h
openshift-controller-manager-7bcc868796-lq4bw        1/1     Running            0          11h
openshift-controller-manager-7bcc868796-nr6xt        1/1     Running            0          11h
openshift-oauth-apiserver-f5c699955-dn75t            1/1     Running            3          11h
openshift-oauth-apiserver-f5c699955-jwvdg            1/1     Running            3          11h
openshift-oauth-apiserver-f5c699955-kqv68            1/1     Running            4          11h
packageserver-568ddbbdf9-25n7z                       1/1     Running            2          11h
packageserver-568ddbbdf9-cgc55                       1/1     Running            4          11h
rbac-sync-operator-84d4c6f768-mzppk                  1/1     Running            0          10h
redhat-marketplace-catalog-6f9bfbdd6f-hx7hn          1/1     Running            0          88m
redhat-marketplace-catalog-rollout-27215316-mfdpp    0/1     Completed          0          88m
redhat-operators-catalog-d57ff6f8c-2hv9m             1/1     Running            0          88m
redhat-operators-catalog-rollout-27215316-5pl66      0/1     Completed          0          88m
Tylers-MacBook-Pro:armada tylerlisowski$ kubectl get pods -n master-prtesting49-57190034
NAME                                                 READY   STATUS             RESTARTS   AGE
catalog-operator-754b6dc8f7-d82vd                    1/1     Running            4          11h
certified-operators-catalog-57d5b9775d-7zp59         1/1     Running            0          88m
certified-operators-catalog-rollout-27215316-tt7vf   0/1     Completed          0          88m
cloud-controller-manager-ddc49497f-4669r             1/1     Running            1          11h
cloud-controller-manager-ddc49497f-gpkts             1/1     Running            0          11h
cloud-controller-manager-ddc49497f-hlfsn             1/1     Running            0          11h
cluster-api-78697c5c98-4wj4f                         1/1     Running            0          11h
cluster-api-78697c5c98-ptsqf                         1/1     Running            0          11h
cluster-api-78697c5c98-v5t8h                         1/1     Running            0          11h
cluster-autoscaler-f6f5cb844-xxmsl                   1/1     Running            1          11h
cluster-health-7b8b9b944d-jb7nh                      0/1     CrashLoopBackOff   136        10h
cluster-policy-controller-5bdd77ccfc-dltbf           1/1     Running            0          11h
cluster-policy-controller-5bdd77ccfc-mkshk           1/1     Running            0          11h
cluster-policy-controller-5bdd77ccfc-wvsg2           1/1     Running            0          11h
cluster-version-operator-857bd8cc5c-xjn5r            1/1     Running            0          11h
community-operators-catalog-86c89bcd7d-r89mf         1/1     Running            0          88m
community-operators-catalog-rollout-27215316-t4psf   0/1     Completed          0          88m
control-plane-operator-f489d8c94-vs548               1/1     Running            0          11h
etcd-operator-f8c5c4f5-5cr8p                         4/4     Running            4          11h
etcd-operator-f8c5c4f5-p6qtq                         4/4     Running            4          11h
etcd-operator-f8c5c4f5-qwb5s                         4/4     Running            4          11h
etcd-prtesting49-57190034-6lc77s9fln                 2/2     Running            0          11h
etcd-prtesting49-57190034-7nx2w75jvd                 2/2     Running            0          11h
etcd-prtesting49-57190034-br8f2b7krk                 2/2     Running            0          11h
etcd-restore-operator-cert-creator-gz4zl             0/1     Completed          0          11h
hosted-cluster-config-operator-664f4b4fb9-tgfdf      1/1     Running            3          11h
ignition-server-5d6d87fb75-nb8bd                     0/1     Running            0          19s
ignition-server-5d6d87fb75-qbr5w                     0/1     Running            0          19s
ignition-server-5d6d87fb75-wh92h                     0/1     Running            0          19s
konnectivity-agent-84d4545d7-vljwj                   1/1     Running            0          11h
konnectivity-agent-84d4545d7-wdmfk                   1/1     Running            0          11h
konnectivity-agent-84d4545d7-znrl9                   1/1     Running            0          11h
konnectivity-server-778479cc58-p5dx4                 1/1     Running            0          11h
kube-apiserver-677b5667bc-4pd2j                      2/2     Running            0          11h
kube-apiserver-677b5667bc-588hh                      2/2     Running            0          11h
kube-apiserver-677b5667bc-9jdqk                      2/2     Running            0          11h
kube-controller-manager-6d4f6c5b8d-jxm29             1/1     Running            0          10h
kube-controller-manager-6d4f6c5b8d-sg9j9             1/1     Running            0          10h
kube-controller-manager-6d4f6c5b8d-ths5c             1/1     Running            0          10h
kube-scheduler-c97c77d6f-2rtnm                       1/1     Running            0          11h
kube-scheduler-c97c77d6f-f72w5                       1/1     Running            0          11h
kube-scheduler-c97c77d6f-fbhfw                       1/1     Running            0          11h
machine-config-server-1236201992                     0/1     Init:2/3           0          4s
machine-config-server-291020420                      0/1     Init:2/3           0          5s
machine-config-server-446334720                      1/1     Running            0          9s
manifests-bootstrapper                               0/1     Completed          3          11h
oauth-openshift-855b8c4bfc-7v4rp                     1/1     Running            0          11h
oauth-openshift-855b8c4bfc-cx7z4                     1/1     Running            0          11h
oauth-openshift-855b8c4bfc-kl57h                     1/1     Running            0          11h
olm-operator-56b5bc7747-ntmxz                        1/1     Running            4          11h
openshift-apiserver-8489b9bb65-rp7vg                 2/2     Running            0          11h
openshift-apiserver-8489b9bb65-tj62l                 2/2     Running            0          11h
openshift-apiserver-8489b9bb65-xtlhz                 2/2     Running            0          11h
openshift-controller-manager-7bcc868796-7vmtb        1/1     Running            0          11h
openshift-controller-manager-7bcc868796-lq4bw        1/1     Running            0          11h
openshift-controller-manager-7bcc868796-nr6xt        1/1     Running            0          11h
openshift-oauth-apiserver-f5c699955-dn75t            1/1     Running            3          11h
openshift-oauth-apiserver-f5c699955-jwvdg            1/1     Running            3          11h
openshift-oauth-apiserver-f5c699955-kqv68            1/1     Running            4          11h
packageserver-568ddbbdf9-25n7z                       1/1     Running            2          11h
packageserver-568ddbbdf9-cgc55                       1/1     Running            4          11h
rbac-sync-operator-84d4c6f768-mzppk                  1/1     Running            0          10h
redhat-marketplace-catalog-6f9bfbdd6f-hx7hn          1/1     Running            0          88m
redhat-marketplace-catalog-rollout-27215316-mfdpp    0/1     Completed          0          88m
redhat-operators-catalog-d57ff6f8c-2hv9m             1/1     Running            0          88m
redhat-operators-catalog-rollout-27215316-5pl66      0/1     Completed          0          88m
Tylers-MacBook-Pro:armada tylerlisowski$ kubectl get pods -n master-prtesting49-57190034
NAME                                                 READY   STATUS             RESTARTS   AGE
catalog-operator-754b6dc8f7-d82vd                    1/1     Running            4          11h
certified-operators-catalog-57d5b9775d-7zp59         1/1     Running            0          88m
certified-operators-catalog-rollout-27215316-tt7vf   0/1     Completed          0          88m
cloud-controller-manager-ddc49497f-4669r             1/1     Running            1          11h
cloud-controller-manager-ddc49497f-gpkts             1/1     Running            0          11h
cloud-controller-manager-ddc49497f-hlfsn             1/1     Running            0          11h
cluster-api-78697c5c98-4wj4f                         1/1     Running            0          11h
cluster-api-78697c5c98-ptsqf                         1/1     Running            0          11h
cluster-api-78697c5c98-v5t8h                         1/1     Running            0          11h
cluster-autoscaler-f6f5cb844-xxmsl                   1/1     Running            1          11h
cluster-health-7b8b9b944d-jb7nh                      0/1     CrashLoopBackOff   136        10h
cluster-policy-controller-5bdd77ccfc-dltbf           1/1     Running            0          11h
cluster-policy-controller-5bdd77ccfc-mkshk           1/1     Running            0          11h
cluster-policy-controller-5bdd77ccfc-wvsg2           1/1     Running            0          11h
cluster-version-operator-857bd8cc5c-xjn5r            1/1     Running            0          11h
community-operators-catalog-86c89bcd7d-r89mf         1/1     Running            0          88m
community-operators-catalog-rollout-27215316-t4psf   0/1     Completed          0          88m
control-plane-operator-f489d8c94-vs548               1/1     Running            0          11h
etcd-operator-f8c5c4f5-5cr8p                         4/4     Running            4          11h
etcd-operator-f8c5c4f5-p6qtq                         4/4     Running            4          11h
etcd-operator-f8c5c4f5-qwb5s                         4/4     Running            4          11h
etcd-prtesting49-57190034-6lc77s9fln                 2/2     Running            0          11h
etcd-prtesting49-57190034-7nx2w75jvd                 2/2     Running            0          11h
etcd-prtesting49-57190034-br8f2b7krk                 2/2     Running            0          11h
etcd-restore-operator-cert-creator-gz4zl             0/1     Completed          0          11h
hosted-cluster-config-operator-664f4b4fb9-tgfdf      1/1     Running            3          11h
ignition-server-5d6d87fb75-nb8bd                     0/1     Running            0          22s
ignition-server-5d6d87fb75-qbr5w                     0/1     Running            0          22s
ignition-server-5d6d87fb75-wh92h                     0/1     Running            0          22s
konnectivity-agent-84d4545d7-vljwj                   1/1     Running            0          11h
konnectivity-agent-84d4545d7-wdmfk                   1/1     Running            0          11h
konnectivity-agent-84d4545d7-znrl9                   1/1     Running            0          11h
konnectivity-server-778479cc58-p5dx4                 1/1     Running            0          11h
kube-apiserver-677b5667bc-4pd2j                      2/2     Running            0          11h
kube-apiserver-677b5667bc-588hh                      2/2     Running            0          11h
kube-apiserver-677b5667bc-9jdqk                      2/2     Running            0          11h
kube-controller-manager-6d4f6c5b8d-jxm29             1/1     Running            0          10h
kube-controller-manager-6d4f6c5b8d-sg9j9             1/1     Running            0          10h
kube-controller-manager-6d4f6c5b8d-ths5c             1/1     Running            0          10h
kube-scheduler-c97c77d6f-2rtnm                       1/1     Running            0          11h
kube-scheduler-c97c77d6f-f72w5                       1/1     Running            0          11h
kube-scheduler-c97c77d6f-fbhfw                       1/1     Running            0          11h
machine-config-server-1236201992                     1/1     Running            0          7s
machine-config-server-291020420                      0/1     PodInitializing    0          8s
machine-config-server-446334720                      1/1     Terminating        0          12s
manifests-bootstrapper                               0/1     Completed          3          11h
oauth-openshift-855b8c4bfc-7v4rp                     1/1     Running            0          11h
oauth-openshift-855b8c4bfc-cx7z4                     1/1     Running            0          11h
oauth-openshift-855b8c4bfc-kl57h                     1/1     Running            0          11h
olm-operator-56b5bc7747-ntmxz                        1/1     Running            4          11h
openshift-apiserver-8489b9bb65-rp7vg                 2/2     Running            0          11h
openshift-apiserver-8489b9bb65-tj62l                 2/2     Running            0          11h
openshift-apiserver-8489b9bb65-xtlhz                 2/2     Running            0          11h
openshift-controller-manager-7bcc868796-7vmtb        1/1     Running            0          11h
openshift-controller-manager-7bcc868796-lq4bw        1/1     Running            0          11h
openshift-controller-manager-7bcc868796-nr6xt        1/1     Running            0          11h
openshift-oauth-apiserver-f5c699955-dn75t            1/1     Running            3          11h
openshift-oauth-apiserver-f5c699955-jwvdg            1/1     Running            3          11h
openshift-oauth-apiserver-f5c699955-kqv68            1/1     Running            4          11h
packageserver-568ddbbdf9-25n7z                       1/1     Running            2          11h
packageserver-568ddbbdf9-cgc55                       1/1     Running            4          11h
rbac-sync-operator-84d4c6f768-mzppk                  1/1     Running            0          10h
redhat-marketplace-catalog-6f9bfbdd6f-hx7hn          1/1     Running            0          88m
redhat-marketplace-catalog-rollout-27215316-mfdpp    0/1     Completed          0          88m
redhat-operators-catalog-d57ff6f8c-2hv9m             1/1     Running            0          88m
redhat-operators-catalog-rollout-27215316-5pl66      0/1     Completed          0          88m
Tylers-MacBook-Pro:armada tylerlisowski$ kubectl get pods -n master-prtesting49-57190034
NAME                                                 READY   STATUS             RESTARTS   AGE
catalog-operator-754b6dc8f7-d82vd                    1/1     Running            4          11h
certified-operators-catalog-57d5b9775d-7zp59         1/1     Running            0          88m
certified-operators-catalog-rollout-27215316-tt7vf   0/1     Completed          0          88m
cloud-controller-manager-ddc49497f-4669r             1/1     Running            1          11h
cloud-controller-manager-ddc49497f-gpkts             1/1     Running            0          11h
cloud-controller-manager-ddc49497f-hlfsn             1/1     Running            0          11h
cluster-api-78697c5c98-4wj4f                         1/1     Running            0          11h
cluster-api-78697c5c98-ptsqf                         1/1     Running            0          11h
cluster-api-78697c5c98-v5t8h                         1/1     Running            0          11h
cluster-autoscaler-f6f5cb844-xxmsl                   1/1     Running            1          11h
cluster-health-7b8b9b944d-jb7nh                      0/1     CrashLoopBackOff   136        10h
cluster-policy-controller-5bdd77ccfc-dltbf           1/1     Running            0          11h
cluster-policy-controller-5bdd77ccfc-mkshk           1/1     Running            0          11h
cluster-policy-controller-5bdd77ccfc-wvsg2           1/1     Running            0          11h
cluster-version-operator-857bd8cc5c-xjn5r            1/1     Running            0          11h
community-operators-catalog-86c89bcd7d-r89mf         1/1     Running            0          88m
community-operators-catalog-rollout-27215316-t4psf   0/1     Completed          0          88m
control-plane-operator-f489d8c94-vs548               1/1     Running            0          11h
etcd-operator-f8c5c4f5-5cr8p                         4/4     Running            4          11h
etcd-operator-f8c5c4f5-p6qtq                         4/4     Running            4          11h
etcd-operator-f8c5c4f5-qwb5s                         4/4     Running            4          11h
etcd-prtesting49-57190034-6lc77s9fln                 2/2     Running            0          11h
etcd-prtesting49-57190034-7nx2w75jvd                 2/2     Running            0          11h
etcd-prtesting49-57190034-br8f2b7krk                 2/2     Running            0          11h
etcd-restore-operator-cert-creator-gz4zl             0/1     Completed          0          11h
hosted-cluster-config-operator-664f4b4fb9-tgfdf      1/1     Running            3          11h
ignition-server-5d6d87fb75-nb8bd                     0/1     Running            0          24s
ignition-server-5d6d87fb75-qbr5w                     0/1     Running            0          24s
ignition-server-5d6d87fb75-wh92h                     0/1     Running            0          24s
konnectivity-agent-84d4545d7-vljwj                   1/1     Running            0          11h
konnectivity-agent-84d4545d7-wdmfk                   1/1     Running            0          11h
konnectivity-agent-84d4545d7-znrl9                   1/1     Running            0          11h
konnectivity-server-778479cc58-p5dx4                 1/1     Running            0          11h
kube-apiserver-677b5667bc-4pd2j                      2/2     Running            0          11h
kube-apiserver-677b5667bc-588hh                      2/2     Running            0          11h
kube-apiserver-677b5667bc-9jdqk                      2/2     Running            0          11h
kube-controller-manager-6d4f6c5b8d-jxm29             1/1     Running            0          10h
kube-controller-manager-6d4f6c5b8d-sg9j9             1/1     Running            0          10h
kube-controller-manager-6d4f6c5b8d-ths5c             1/1     Running            0          10h
kube-scheduler-c97c77d6f-2rtnm                       1/1     Running            0          11h
kube-scheduler-c97c77d6f-f72w5                       1/1     Running            0          11h
kube-scheduler-c97c77d6f-fbhfw                       1/1     Running            0          11h
machine-config-server-1236201992                     1/1     Running            0          9s
machine-config-server-1463997948                     0/1     Init:0/3           0          2s
machine-config-server-291020420                      1/1     Running            0          10s
machine-config-server-446334720                      0/1     Terminating        0          14s
manifests-bootstrapper                               0/1     Completed          3          11h
oauth-openshift-855b8c4bfc-7v4rp                     1/1     Running            0          11h
oauth-openshift-855b8c4bfc-cx7z4                     1/1     Running            0          11h
oauth-openshift-855b8c4bfc-kl57h                     1/1     Running            0          11h
olm-operator-56b5bc7747-ntmxz                        1/1     Running            4          11h
openshift-apiserver-8489b9bb65-rp7vg                 2/2     Running            0          11h
openshift-apiserver-8489b9bb65-tj62l                 2/2     Running            0          11h
openshift-apiserver-8489b9bb65-xtlhz                 2/2     Running            0          11h
openshift-controller-manager-7bcc868796-7vmtb        1/1     Running            0          11h
openshift-controller-manager-7bcc868796-lq4bw        1/1     Running            0          11h
openshift-controller-manager-7bcc868796-nr6xt        1/1     Running            0          11h
openshift-oauth-apiserver-f5c699955-dn75t            1/1     Running            3          11h
openshift-oauth-apiserver-f5c699955-jwvdg            1/1     Running            3          11h
openshift-oauth-apiserver-f5c699955-kqv68            1/1     Running            4          11h
packageserver-568ddbbdf9-25n7z                       1/1     Running            2          11h
packageserver-568ddbbdf9-cgc55                       1/1     Running            4          11h
rbac-sync-operator-84d4c6f768-mzppk                  1/1     Running            0          10h
redhat-marketplace-catalog-6f9bfbdd6f-hx7hn          1/1     Running            0          88m
redhat-marketplace-catalog-rollout-27215316-mfdpp    0/1     Completed          0          88m
redhat-operators-catalog-d57ff6f8c-2hv9m             1/1     Running            0          88m
redhat-operators-catalog-rollout-27215316-5pl66      0/1     Completed          0          88m
Tylers-MacBook-Pro:armada tylerlisowski$ kubectl get pods -n master-prtesting49-57190034
NAME                                                 READY   STATUS             RESTARTS   AGE
catalog-operator-754b6dc8f7-d82vd                    1/1     Running            4          11h
certified-operators-catalog-57d5b9775d-7zp59         1/1     Running            0          88m
certified-operators-catalog-rollout-27215316-tt7vf   0/1     Completed          0          88m
cloud-controller-manager-ddc49497f-4669r             1/1     Running            1          11h
cloud-controller-manager-ddc49497f-gpkts             1/1     Running            0          11h
cloud-controller-manager-ddc49497f-hlfsn             1/1     Running            0          11h
cluster-api-78697c5c98-4wj4f                         1/1     Running            0          11h
cluster-api-78697c5c98-ptsqf                         1/1     Running            0          11h
cluster-api-78697c5c98-v5t8h                         1/1     Running            0          11h
cluster-autoscaler-f6f5cb844-xxmsl                   1/1     Running            1          11h
cluster-health-7b8b9b944d-jb7nh                      0/1     CrashLoopBackOff   136        10h
cluster-policy-controller-5bdd77ccfc-dltbf           1/1     Running            0          11h
cluster-policy-controller-5bdd77ccfc-mkshk           1/1     Running            0          11h
cluster-policy-controller-5bdd77ccfc-wvsg2           1/1     Running            0          11h
cluster-version-operator-857bd8cc5c-xjn5r            1/1     Running            0          11h
community-operators-catalog-86c89bcd7d-r89mf         1/1     Running            0          88m
community-operators-catalog-rollout-27215316-t4psf   0/1     Completed          0          88m
control-plane-operator-f489d8c94-vs548               1/1     Running            0          11h
etcd-operator-f8c5c4f5-5cr8p                         4/4     Running            4          11h
etcd-operator-f8c5c4f5-p6qtq                         4/4     Running            4          11h
etcd-operator-f8c5c4f5-qwb5s                         4/4     Running            4          11h
etcd-prtesting49-57190034-6lc77s9fln                 2/2     Running            0          11h
etcd-prtesting49-57190034-7nx2w75jvd                 2/2     Running            0          11h
etcd-prtesting49-57190034-br8f2b7krk                 2/2     Running            0          11h
etcd-restore-operator-cert-creator-gz4zl             0/1     Completed          0          11h
hosted-cluster-config-operator-664f4b4fb9-tgfdf      1/1     Running            3          11h
ignition-server-5d6d87fb75-nb8bd                     0/1     Running            0          26s
ignition-server-5d6d87fb75-qbr5w                     0/1     Running            0          26s
ignition-server-5d6d87fb75-wh92h                     0/1     Running            0          26s
konnectivity-agent-84d4545d7-vljwj                   1/1     Running            0          11h
konnectivity-agent-84d4545d7-wdmfk                   1/1     Running            0          11h
konnectivity-agent-84d4545d7-znrl9                   1/1     Running            0          11h
konnectivity-server-778479cc58-p5dx4                 1/1     Running            0          11h
kube-apiserver-677b5667bc-4pd2j                      2/2     Running            0          11h
kube-apiserver-677b5667bc-588hh                      2/2     Running            0          11h
kube-apiserver-677b5667bc-9jdqk                      2/2     Running            0          11h
kube-controller-manager-6d4f6c5b8d-jxm29             1/1     Running            0          10h
kube-controller-manager-6d4f6c5b8d-sg9j9             1/1     Running            0          10h
kube-controller-manager-6d4f6c5b8d-ths5c             1/1     Running            0          10h
kube-scheduler-c97c77d6f-2rtnm                       1/1     Running            0          11h
kube-scheduler-c97c77d6f-f72w5                       1/1     Running            0          11h
kube-scheduler-c97c77d6f-fbhfw                       1/1     Running            0          11h
machine-config-server-1236201992                     1/1     Terminating        0          11s
machine-config-server-1408474896                     0/1     Pending            0          0s
machine-config-server-1463997948                     0/1     Init:2/3           0          4s
machine-config-server-291020420                      1/1     Terminating        0          12s
machine-config-server-446334720                      0/1     Terminating        0          16s
manifests-bootstrapper                               0/1     Completed          3          11h
oauth-openshift-855b8c4bfc-7v4rp                     1/1     Running            0          11h
oauth-openshift-855b8c4bfc-cx7z4                     1/1     Running            0          11h
oauth-openshift-855b8c4bfc-kl57h                     1/1     Running            0          11h
olm-operator-56b5bc7747-ntmxz                        1/1     Running            4          11h
openshift-apiserver-8489b9bb65-rp7vg                 2/2     Running            0          11h
openshift-apiserver-8489b9bb65-tj62l                 2/2     Running            0          11h
openshift-apiserver-8489b9bb65-xtlhz                 2/2     Running            0          11h
openshift-controller-manager-7bcc868796-7vmtb        1/1     Running            0          11h
openshift-controller-manager-7bcc868796-lq4bw        1/1     Running            0          11h
openshift-controller-manager-7bcc868796-nr6xt        1/1     Running            0          11h
openshift-oauth-apiserver-f5c699955-dn75t            1/1     Running            3          11h
openshift-oauth-apiserver-f5c699955-jwvdg            1/1     Running            3          11h
openshift-oauth-apiserver-f5c699955-kqv68            1/1     Running            4          11h
packageserver-568ddbbdf9-25n7z                       1/1     Running            2          11h
packageserver-568ddbbdf9-cgc55                       1/1     Running            4          11h
rbac-sync-operator-84d4c6f768-mzppk                  1/1     Running            0          10h
redhat-marketplace-catalog-6f9bfbdd6f-hx7hn          1/1     Running            0          88m
redhat-marketplace-catalog-rollout-27215316-mfdpp    0/1     Completed          0          88m
redhat-operators-catalog-d57ff6f8c-2hv9m             1/1     Running            0          88m
redhat-operators-catalog-rollout-27215316-5pl66      0/1     Completed          0          88m
Tylers-MacBook-Pro:armada tylerlisowski$ kubectl get pods -n master-prtesting49-57190034
NAME                                                 READY   STATUS             RESTARTS   AGE
catalog-operator-754b6dc8f7-d82vd                    1/1     Running            4          11h
certified-operators-catalog-57d5b9775d-7zp59         1/1     Running            0          88m
certified-operators-catalog-rollout-27215316-tt7vf   0/1     Completed          0          88m
cloud-controller-manager-ddc49497f-4669r             1/1     Running            1          11h
cloud-controller-manager-ddc49497f-gpkts             1/1     Running            0          11h
cloud-controller-manager-ddc49497f-hlfsn             1/1     Running            0          11h
cluster-api-78697c5c98-4wj4f                         1/1     Running            0          11h
cluster-api-78697c5c98-ptsqf                         1/1     Running            0          11h
cluster-api-78697c5c98-v5t8h                         1/1     Running            0          11h
cluster-autoscaler-f6f5cb844-xxmsl                   1/1     Running            1          11h
cluster-health-7b8b9b944d-jb7nh                      0/1     CrashLoopBackOff   136        10h
cluster-policy-controller-5bdd77ccfc-dltbf           1/1     Running            0          11h
cluster-policy-controller-5bdd77ccfc-mkshk           1/1     Running            0          11h
cluster-policy-controller-5bdd77ccfc-wvsg2           1/1     Running            0          11h
cluster-version-operator-857bd8cc5c-xjn5r            1/1     Running            0          11h
community-operators-catalog-86c89bcd7d-r89mf         1/1     Running            0          88m
community-operators-catalog-rollout-27215316-t4psf   0/1     Completed          0          88m
control-plane-operator-f489d8c94-vs548               1/1     Running            0          11h
etcd-operator-f8c5c4f5-5cr8p                         4/4     Running            4          11h
etcd-operator-f8c5c4f5-p6qtq                         4/4     Running            4          11h
etcd-operator-f8c5c4f5-qwb5s                         4/4     Running            4          11h
etcd-prtesting49-57190034-6lc77s9fln                 2/2     Running            0          11h
etcd-prtesting49-57190034-7nx2w75jvd                 2/2     Running            0          11h
etcd-prtesting49-57190034-br8f2b7krk                 2/2     Running            0          11h
etcd-restore-operator-cert-creator-gz4zl             0/1     Completed          0          11h
hosted-cluster-config-operator-664f4b4fb9-tgfdf      1/1     Running            3          11h
ignition-server-5d6d87fb75-nb8bd                     0/1     Running            0          27s
ignition-server-5d6d87fb75-qbr5w                     0/1     Running            0          27s
ignition-server-5d6d87fb75-wh92h                     0/1     Running            0          27s
konnectivity-agent-84d4545d7-vljwj                   1/1     Running            0          11h
konnectivity-agent-84d4545d7-wdmfk                   1/1     Running            0          11h
konnectivity-agent-84d4545d7-znrl9                   1/1     Running            0          11h
konnectivity-server-778479cc58-p5dx4                 1/1     Running            0          11h
kube-apiserver-677b5667bc-4pd2j                      2/2     Running            0          11h
kube-apiserver-677b5667bc-588hh                      2/2     Running            0          11h
kube-apiserver-677b5667bc-9jdqk                      2/2     Running            0          11h
kube-controller-manager-6d4f6c5b8d-jxm29             1/1     Running            0          10h
kube-controller-manager-6d4f6c5b8d-sg9j9             1/1     Running            0          10h
kube-controller-manager-6d4f6c5b8d-ths5c             1/1     Running            0          10h
kube-scheduler-c97c77d6f-2rtnm                       1/1     Running            0          11h
kube-scheduler-c97c77d6f-f72w5                       1/1     Running            0          11h
kube-scheduler-c97c77d6f-fbhfw                       1/1     Running            0          11h
machine-config-server-1018368211                     0/1     Init:0/3           0          1s
machine-config-server-1236201992                     1/1     Terminating        0          12s
machine-config-server-1408474896                     0/1     Init:0/3           0          1s
machine-config-server-1463997948                     0/1     PodInitializing    0          5s
machine-config-server-291020420                      1/1     Terminating        0          13s
machine-config-server-446334720                      0/1     Terminating        0          17s
manifests-bootstrapper                               0/1     Completed          3          11h
oauth-openshift-855b8c4bfc-7v4rp                     1/1     Running            0          11h
oauth-openshift-855b8c4bfc-cx7z4                     1/1     Running            0          11h
oauth-openshift-855b8c4bfc-kl57h                     1/1     Running            0          11h
olm-operator-56b5bc7747-ntmxz                        1/1     Running            4          11h
openshift-apiserver-8489b9bb65-rp7vg                 2/2     Running            0          11h
openshift-apiserver-8489b9bb65-tj62l                 2/2     Running            0          11h
openshift-apiserver-8489b9bb65-xtlhz                 2/2     Running            0          11h
openshift-controller-manager-7bcc868796-7vmtb        1/1     Running            0          11h
openshift-controller-manager-7bcc868796-lq4bw        1/1     Running            0          11h
openshift-controller-manager-7bcc868796-nr6xt        1/1     Running            0          11h
openshift-oauth-apiserver-f5c699955-dn75t            1/1     Running            3          11h
openshift-oauth-apiserver-f5c699955-jwvdg            1/1     Running            3          11h
openshift-oauth-apiserver-f5c699955-kqv68            1/1     Running            4          11h
packageserver-568ddbbdf9-25n7z                       1/1     Running            2          11h
packageserver-568ddbbdf9-cgc55                       1/1     Running            4          11h
rbac-sync-operator-84d4c6f768-mzppk                  1/1     Running            0          10h
redhat-marketplace-catalog-6f9bfbdd6f-hx7hn          1/1     Running            0          88m
redhat-marketplace-catalog-rollout-27215316-mfdpp    0/1     Completed          0          88m
redhat-operators-catalog-d57ff6f8c-2hv9m             1/1     Running            0          88m
redhat-operators-catalog-rollout-27215316-5pl66      0/1     Completed          0          88m
Tylers-MacBook-Pro:armada tylerlisowski$ kubectl get pods -n master-prtesting49-57190034
NAME                                                 READY   STATUS             RESTARTS   AGE
catalog-operator-754b6dc8f7-d82vd                    1/1     Running            4          11h
certified-operators-catalog-57d5b9775d-7zp59         1/1     Running            0          88m
certified-operators-catalog-rollout-27215316-tt7vf   0/1     Completed          0          88m
cloud-controller-manager-ddc49497f-4669r             1/1     Running            1          11h
cloud-controller-manager-ddc49497f-gpkts             1/1     Running            0          11h
cloud-controller-manager-ddc49497f-hlfsn             1/1     Running            0          11h
cluster-api-78697c5c98-4wj4f                         1/1     Running            0          11h
cluster-api-78697c5c98-ptsqf                         1/1     Running            0          11h
cluster-api-78697c5c98-v5t8h                         1/1     Running            0          11h
cluster-autoscaler-f6f5cb844-xxmsl                   1/1     Running            1          11h
cluster-health-7b8b9b944d-jb7nh                      0/1     CrashLoopBackOff   136        10h
cluster-policy-controller-5bdd77ccfc-dltbf           1/1     Running            0          11h
cluster-policy-controller-5bdd77ccfc-mkshk           1/1     Running            0          11h
cluster-policy-controller-5bdd77ccfc-wvsg2           1/1     Running            0          11h
cluster-version-operator-857bd8cc5c-xjn5r            1/1     Running            0          11h
community-operators-catalog-86c89bcd7d-r89mf         1/1     Running            0          88m
community-operators-catalog-rollout-27215316-t4psf   0/1     Completed          0          88m
control-plane-operator-f489d8c94-vs548               1/1     Running            0          11h
etcd-operator-f8c5c4f5-5cr8p                         4/4     Running            4          11h
etcd-operator-f8c5c4f5-p6qtq                         4/4     Running            4          11h
etcd-operator-f8c5c4f5-qwb5s                         4/4     Running            4          11h
etcd-prtesting49-57190034-6lc77s9fln                 2/2     Running            0          11h
etcd-prtesting49-57190034-7nx2w75jvd                 2/2     Running            0          11h
etcd-prtesting49-57190034-br8f2b7krk                 2/2     Running            0          11h
etcd-restore-operator-cert-creator-gz4zl             0/1     Completed          0          11h
hosted-cluster-config-operator-664f4b4fb9-tgfdf      1/1     Running            3          11h
ignition-server-5d6d87fb75-nb8bd                     0/1     Running            0          29s
ignition-server-5d6d87fb75-qbr5w                     0/1     Running            0          29s
ignition-server-5d6d87fb75-wh92h                     0/1     Running            0          29s
konnectivity-agent-84d4545d7-vljwj                   1/1     Running            0          11h
konnectivity-agent-84d4545d7-wdmfk                   1/1     Running            0          11h
konnectivity-agent-84d4545d7-znrl9                   1/1     Running            0          11h
konnectivity-server-778479cc58-p5dx4                 1/1     Running            0          11h
kube-apiserver-677b5667bc-4pd2j                      2/2     Running            0          11h
kube-apiserver-677b5667bc-588hh                      2/2     Running            0          11h
kube-apiserver-677b5667bc-9jdqk                      2/2     Running            0          11h
kube-controller-manager-6d4f6c5b8d-jxm29             1/1     Running            0          10h
kube-controller-manager-6d4f6c5b8d-sg9j9             1/1     Running            0          10h
kube-controller-manager-6d4f6c5b8d-ths5c             1/1     Running            0          10h
kube-scheduler-c97c77d6f-2rtnm                       1/1     Running            0          11h
kube-scheduler-c97c77d6f-f72w5                       1/1     Running            0          11h
kube-scheduler-c97c77d6f-fbhfw                       1/1     Running            0          11h
machine-config-server-1018368211                     0/1     Init:1/3           0          3s
machine-config-server-1236201992                     0/1     Terminating        0          14s
machine-config-server-1408474896                     0/1     Init:1/3           0          3s
machine-config-server-1463997948                     1/1     Running            0          7s
machine-config-server-446334720                      0/1     Terminating        0          19s
manifests-bootstrapper                               0/1     Completed          3          11h
oauth-openshift-855b8c4bfc-7v4rp                     1/1     Running            0          11h
oauth-openshift-855b8c4bfc-cx7z4                     1/1     Running            0          11h
oauth-openshift-855b8c4bfc-kl57h                     1/1     Running            0          11h
olm-operator-56b5bc7747-ntmxz                        1/1     Running            4          11h
openshift-apiserver-8489b9bb65-rp7vg                 2/2     Running            0          11h
openshift-apiserver-8489b9bb65-tj62l                 2/2     Running            0          11h
openshift-apiserver-8489b9bb65-xtlhz                 2/2     Running            0          11h
openshift-controller-manager-7bcc868796-7vmtb        1/1     Running            0          11h
openshift-controller-manager-7bcc868796-lq4bw        1/1     Running            0          11h
openshift-controller-manager-7bcc868796-nr6xt        1/1     Running            0          11h
openshift-oauth-apiserver-f5c699955-dn75t            1/1     Running            3          11h
openshift-oauth-apiserver-f5c699955-jwvdg            1/1     Running            3          11h
openshift-oauth-apiserver-f5c699955-kqv68            1/1     Running            4          11h
packageserver-568ddbbdf9-25n7z                       1/1     Running            2          11h
packageserver-568ddbbdf9-cgc55                       1/1     Running            4          11h
rbac-sync-operator-84d4c6f768-mzppk                  1/1     Running            0          10h
redhat-marketplace-catalog-6f9bfbdd6f-hx7hn          1/1     Running            0          88m
redhat-marketplace-catalog-rollout-27215316-mfdpp    0/1     Completed          0          88m
redhat-operators-catalog-d57ff6f8c-2hv9m             1/1     Running            0          88m
redhat-operators-catalog-rollout-27215316-5pl66      0/1     Completed          0          88m
Tylers-MacBook-Pro:armada tylerlisowski$ kubectl get pods -n master-prtesting49-57190034
NAME                                                 READY   STATUS             RESTARTS   AGE
catalog-operator-754b6dc8f7-d82vd                    1/1     Running            4          11h
certified-operators-catalog-57d5b9775d-7zp59         1/1     Running            0          88m
certified-operators-catalog-rollout-27215316-tt7vf   0/1     Completed          0          89m
cloud-controller-manager-ddc49497f-4669r             1/1     Running            1          11h
cloud-controller-manager-ddc49497f-gpkts             1/1     Running            0          11h
cloud-controller-manager-ddc49497f-hlfsn             1/1     Running            0          11h
cluster-api-78697c5c98-4wj4f                         1/1     Running            0          11h
cluster-api-78697c5c98-ptsqf                         1/1     Running            0          11h
cluster-api-78697c5c98-v5t8h                         1/1     Running            0          11h
cluster-autoscaler-f6f5cb844-xxmsl                   1/1     Running            1          11h
cluster-health-7b8b9b944d-jb7nh                      0/1     CrashLoopBackOff   136        10h
cluster-policy-controller-5bdd77ccfc-dltbf           1/1     Running            0          11h
cluster-policy-controller-5bdd77ccfc-mkshk           1/1     Running            0          11h
cluster-policy-controller-5bdd77ccfc-wvsg2           1/1     Running            0          11h
cluster-version-operator-857bd8cc5c-xjn5r            1/1     Running            0          11h
community-operators-catalog-86c89bcd7d-r89mf         1/1     Running            0          88m
community-operators-catalog-rollout-27215316-t4psf   0/1     Completed          0          89m
control-plane-operator-f489d8c94-vs548               1/1     Running            0          11h
etcd-operator-f8c5c4f5-5cr8p                         4/4     Running            4          11h
etcd-operator-f8c5c4f5-p6qtq                         4/4     Running            4          11h
etcd-operator-f8c5c4f5-qwb5s                         4/4     Running            4          11h
etcd-prtesting49-57190034-6lc77s9fln                 2/2     Running            0          11h
etcd-prtesting49-57190034-7nx2w75jvd                 2/2     Running            0          11h
etcd-prtesting49-57190034-br8f2b7krk                 2/2     Running            0          11h
etcd-restore-operator-cert-creator-gz4zl             0/1     Completed          0          11h
hosted-cluster-config-operator-664f4b4fb9-tgfdf      1/1     Running            3          11h
ignition-server-5d6d87fb75-nb8bd                     0/1     Running            0          31s
ignition-server-5d6d87fb75-qbr5w                     0/1     Running            0          31s
ignition-server-5d6d87fb75-wh92h                     0/1     Running            0          31s
konnectivity-agent-84d4545d7-vljwj                   1/1     Running            0          11h
konnectivity-agent-84d4545d7-wdmfk                   1/1     Running            0          11h
konnectivity-agent-84d4545d7-znrl9                   1/1     Running            0          11h
konnectivity-server-778479cc58-p5dx4                 1/1     Running            0          11h
kube-apiserver-677b5667bc-4pd2j                      2/2     Running            0          11h
kube-apiserver-677b5667bc-588hh                      2/2     Running            0          11h
kube-apiserver-677b5667bc-9jdqk                      2/2     Running            0          11h
kube-controller-manager-6d4f6c5b8d-jxm29             1/1     Running            0          10h
kube-controller-manager-6d4f6c5b8d-sg9j9             1/1     Running            0          10h
kube-controller-manager-6d4f6c5b8d-ths5c             1/1     Running            0          10h
kube-scheduler-c97c77d6f-2rtnm                       1/1     Running            0          11h
kube-scheduler-c97c77d6f-f72w5                       1/1     Running            0          11h
kube-scheduler-c97c77d6f-fbhfw                       1/1     Running            0          11h
machine-config-server-1018368211                     0/1     Init:2/3           0          5s
machine-config-server-1408474896                     0/1     PodInitializing    0          5s
machine-config-server-1463997948                     1/1     Running            0          9s
manifests-bootstrapper                               0/1     Completed          3          11h
oauth-openshift-855b8c4bfc-7v4rp                     1/1     Running            0          11h
oauth-openshift-855b8c4bfc-cx7z4                     1/1     Running            0          11h
oauth-openshift-855b8c4bfc-kl57h                     1/1     Running            0          11h
olm-operator-56b5bc7747-ntmxz                        1/1     Running            4          11h
openshift-apiserver-8489b9bb65-rp7vg                 2/2     Running            0          11h
openshift-apiserver-8489b9bb65-tj62l                 2/2     Running            0          11h
openshift-apiserver-8489b9bb65-xtlhz                 2/2     Running            0          11h
openshift-controller-manager-7bcc868796-7vmtb        1/1     Running            0          11h
openshift-controller-manager-7bcc868796-lq4bw        1/1     Running            0          11h
openshift-controller-manager-7bcc868796-nr6xt        1/1     Running            0          11h
openshift-oauth-apiserver-f5c699955-dn75t            1/1     Running            3          11h
openshift-oauth-apiserver-f5c699955-jwvdg            1/1     Running            3          11h
openshift-oauth-apiserver-f5c699955-kqv68            1/1     Running            4          11h
packageserver-568ddbbdf9-25n7z                       1/1     Running            2          11h
packageserver-568ddbbdf9-cgc55                       1/1     Running            4          11h
rbac-sync-operator-84d4c6f768-mzppk                  1/1     Running            0          10h
redhat-marketplace-catalog-6f9bfbdd6f-hx7hn          1/1     Running            0          88m
redhat-marketplace-catalog-rollout-27215316-mfdpp    0/1     Completed          0          89m
redhat-operators-catalog-d57ff6f8c-2hv9m             1/1     Running            0          88m
redhat-operators-catalog-rollout-27215316-5pl66      0/1     Completed          0          89m
Tylers-MacBook-Pro:armada tylerlisowski$ kubectl get pods -n master-prtesting49-57190034
NAME                                                 READY   STATUS             RESTARTS   AGE
catalog-operator-754b6dc8f7-d82vd                    1/1     Running            4          11h
certified-operators-catalog-57d5b9775d-7zp59         1/1     Running            0          89m
certified-operators-catalog-rollout-27215316-tt7vf   0/1     Completed          0          89m
cloud-controller-manager-ddc49497f-4669r             1/1     Running            1          11h
cloud-controller-manager-ddc49497f-gpkts             1/1     Running            0          11h
cloud-controller-manager-ddc49497f-hlfsn             1/1     Running            0          11h
cluster-api-78697c5c98-4wj4f                         1/1     Running            0          11h
cluster-api-78697c5c98-ptsqf                         1/1     Running            0          11h
cluster-api-78697c5c98-v5t8h                         1/1     Running            0          11h
cluster-autoscaler-f6f5cb844-xxmsl                   1/1     Running            1          11h
cluster-health-7b8b9b944d-jb7nh                      0/1     CrashLoopBackOff   136        10h
cluster-policy-controller-5bdd77ccfc-dltbf           1/1     Running            0          11h
cluster-policy-controller-5bdd77ccfc-mkshk           1/1     Running            0          11h
cluster-policy-controller-5bdd77ccfc-wvsg2           1/1     Running            0          11h
cluster-version-operator-857bd8cc5c-xjn5r            1/1     Running            0          11h
community-operators-catalog-86c89bcd7d-r89mf         1/1     Running            0          89m
community-operators-catalog-rollout-27215316-t4psf   0/1     Completed          0          89m
control-plane-operator-f489d8c94-vs548               1/1     Running            0          11h
etcd-operator-f8c5c4f5-5cr8p                         4/4     Running            4          11h
etcd-operator-f8c5c4f5-p6qtq                         4/4     Running            4          11h
etcd-operator-f8c5c4f5-qwb5s                         4/4     Running            4          11h
etcd-prtesting49-57190034-6lc77s9fln                 2/2     Running            0          11h
etcd-prtesting49-57190034-7nx2w75jvd                 2/2     Running            0          11h
etcd-prtesting49-57190034-br8f2b7krk                 2/2     Running            0          11h
etcd-restore-operator-cert-creator-gz4zl             0/1     Completed          0          11h
hosted-cluster-config-operator-664f4b4fb9-tgfdf      1/1     Running            3          11h
ignition-server-5d6d87fb75-nb8bd                     0/1     Running            0          33s
ignition-server-5d6d87fb75-qbr5w                     0/1     Running            0          33s
ignition-server-5d6d87fb75-wh92h                     0/1     Running            0          33s
konnectivity-agent-84d4545d7-vljwj                   1/1     Running            0          11h
konnectivity-agent-84d4545d7-wdmfk                   1/1     Running            0          11h
konnectivity-agent-84d4545d7-znrl9                   1/1     Running            0          11h
konnectivity-server-778479cc58-p5dx4                 1/1     Running            0          11h
kube-apiserver-677b5667bc-4pd2j                      2/2     Running            0          11h
kube-apiserver-677b5667bc-588hh                      2/2     Running            0          11h
kube-apiserver-677b5667bc-9jdqk                      2/2     Running            0          11h
kube-controller-manager-6d4f6c5b8d-jxm29             1/1     Running            0          10h
kube-controller-manager-6d4f6c5b8d-sg9j9             1/1     Running            0          10h
kube-controller-manager-6d4f6c5b8d-ths5c             1/1     Running            0          10h
kube-scheduler-c97c77d6f-2rtnm                       1/1     Running            0          11h
kube-scheduler-c97c77d6f-f72w5                       1/1     Running            0          11h
kube-scheduler-c97c77d6f-fbhfw                       1/1     Running            0          11h
machine-config-server-1018368211                     1/1     Running            0          7s
machine-config-server-1408474896                     1/1     Running            0          7s
machine-config-server-1463997948                     1/1     Terminating        0          11s
manifests-bootstrapper                               0/1     Completed          3          11h
oauth-openshift-855b8c4bfc-7v4rp                     1/1     Running            0          11h
oauth-openshift-855b8c4bfc-cx7z4                     1/1     Running            0          11h
oauth-openshift-855b8c4bfc-kl57h                     1/1     Running            0          11h
olm-operator-56b5bc7747-ntmxz                        1/1     Running            4          11h
openshift-apiserver-8489b9bb65-rp7vg                 2/2     Running            0          11h
openshift-apiserver-8489b9bb65-tj62l                 2/2     Running            0          11h
openshift-apiserver-8489b9bb65-xtlhz                 2/2     Running            0          11h
openshift-controller-manager-7bcc868796-7vmtb        1/1     Running            0          11h
openshift-controller-manager-7bcc868796-lq4bw        1/1     Running            0          11h
openshift-controller-manager-7bcc868796-nr6xt        1/1     Running            0          11h
openshift-oauth-apiserver-f5c699955-dn75t            1/1     Running            3          11h
openshift-oauth-apiserver-f5c699955-jwvdg            1/1     Running            3          11h
openshift-oauth-apiserver-f5c699955-kqv68            1/1     Running            4          11h
packageserver-568ddbbdf9-25n7z                       1/1     Running            2          11h
packageserver-568ddbbdf9-cgc55                       1/1     Running            4          11h
rbac-sync-operator-84d4c6f768-mzppk                  1/1     Running            0          10h
redhat-marketplace-catalog-6f9bfbdd6f-hx7hn          1/1     Running            0          89m
redhat-marketplace-catalog-rollout-27215316-mfdpp    0/1     Completed          0          89m
redhat-operators-catalog-d57ff6f8c-2hv9m             1/1     Running            0          89m
redhat-operators-catalog-rollout-27215316-5pl66      0/1     Completed          0          89m
Tylers-MacBook-Pro:armada tylerlisowski$ kubectl get pods -n master-prtesting49-57190034
NAME                                                 READY   STATUS             RESTARTS   AGE
catalog-operator-754b6dc8f7-d82vd                    1/1     Running            4          11h
certified-operators-catalog-57d5b9775d-7zp59         1/1     Running            0          89m
certified-operators-catalog-rollout-27215316-tt7vf   0/1     Completed          0          89m
cloud-controller-manager-ddc49497f-4669r             1/1     Running            1          11h
cloud-controller-manager-ddc49497f-gpkts             1/1     Running            0          11h
cloud-controller-manager-ddc49497f-hlfsn             1/1     Running            0          11h
cluster-api-78697c5c98-4wj4f                         1/1     Running            0          11h
cluster-api-78697c5c98-ptsqf                         1/1     Running            0          11h
cluster-api-78697c5c98-v5t8h                         1/1     Running            0          11h
cluster-autoscaler-f6f5cb844-xxmsl                   1/1     Running            1          11h
cluster-health-7b8b9b944d-jb7nh                      0/1     CrashLoopBackOff   136        10h
cluster-policy-controller-5bdd77ccfc-dltbf           1/1     Running            0          11h
cluster-policy-controller-5bdd77ccfc-mkshk           1/1     Running            0          11h
cluster-policy-controller-5bdd77ccfc-wvsg2           1/1     Running            0          11h
cluster-version-operator-857bd8cc5c-xjn5r            1/1     Running            0          11h
community-operators-catalog-86c89bcd7d-r89mf         1/1     Running            0          89m
community-operators-catalog-rollout-27215316-t4psf   0/1     Completed          0          89m
control-plane-operator-f489d8c94-vs548               1/1     Running            0          11h
etcd-operator-f8c5c4f5-5cr8p                         4/4     Running            4          11h
etcd-operator-f8c5c4f5-p6qtq                         4/4     Running            4          11h
etcd-operator-f8c5c4f5-qwb5s                         4/4     Running            4          11h
etcd-prtesting49-57190034-6lc77s9fln                 2/2     Running            0          11h
etcd-prtesting49-57190034-7nx2w75jvd                 2/2     Running            0          11h
etcd-prtesting49-57190034-br8f2b7krk                 2/2     Running            0          11h
etcd-restore-operator-cert-creator-gz4zl             0/1     Completed          0          11h
hosted-cluster-config-operator-664f4b4fb9-tgfdf      1/1     Running            3          11h
ignition-server-5d6d87fb75-nb8bd                     0/1     Running            0          34s
ignition-server-5d6d87fb75-qbr5w                     0/1     Running            0          34s
ignition-server-5d6d87fb75-wh92h                     0/1     Running            0          34s
konnectivity-agent-84d4545d7-vljwj                   1/1     Running            0          11h
konnectivity-agent-84d4545d7-wdmfk                   1/1     Running            0          11h
konnectivity-agent-84d4545d7-znrl9                   1/1     Running            0          11h
konnectivity-server-778479cc58-p5dx4                 1/1     Running            0          11h
kube-apiserver-677b5667bc-4pd2j                      2/2     Running            0          11h
kube-apiserver-677b5667bc-588hh                      2/2     Running            0          11h
kube-apiserver-677b5667bc-9jdqk                      2/2     Running            0          11h
kube-controller-manager-6d4f6c5b8d-jxm29             1/1     Running            0          10h
kube-controller-manager-6d4f6c5b8d-sg9j9             1/1     Running            0          10h
kube-controller-manager-6d4f6c5b8d-ths5c             1/1     Running            0          10h
kube-scheduler-c97c77d6f-2rtnm                       1/1     Running            0          11h
kube-scheduler-c97c77d6f-f72w5                       1/1     Running            0          11h
kube-scheduler-c97c77d6f-fbhfw                       1/1     Running            0          11h
machine-config-server-1018368211                     1/1     Running            0          8s
machine-config-server-1408474896                     1/1     Running            0          8s
machine-config-server-1463997948                     0/1     Terminating        0          12s
manifests-bootstrapper                               0/1     Completed          3          11h
oauth-openshift-855b8c4bfc-7v4rp                     1/1     Running            0          11h
oauth-openshift-855b8c4bfc-cx7z4                     1/1     Running            0          11h
oauth-openshift-855b8c4bfc-kl57h                     1/1     Running            0          11h
olm-operator-56b5bc7747-ntmxz                        1/1     Running            4          11h
openshift-apiserver-8489b9bb65-rp7vg                 2/2     Running            0          11h
openshift-apiserver-8489b9bb65-tj62l                 2/2     Running            0          11h
openshift-apiserver-8489b9bb65-xtlhz                 2/2     Running            0          11h
openshift-controller-manager-7bcc868796-7vmtb        1/1     Running            0          11h
openshift-controller-manager-7bcc868796-lq4bw        1/1     Running            0          11h
openshift-controller-manager-7bcc868796-nr6xt        1/1     Running            0          11h
openshift-oauth-apiserver-f5c699955-dn75t            1/1     Running            3          11h
openshift-oauth-apiserver-f5c699955-jwvdg            1/1     Running            3          11h
openshift-oauth-apiserver-f5c699955-kqv68            1/1     Running            4          11h
packageserver-568ddbbdf9-25n7z                       1/1     Running            2          11h
packageserver-568ddbbdf9-cgc55                       1/1     Running            4          11h
rbac-sync-operator-84d4c6f768-mzppk                  1/1     Running            0          10h
redhat-marketplace-catalog-6f9bfbdd6f-hx7hn          1/1     Running            0          89m
redhat-marketplace-catalog-rollout-27215316-mfdpp    0/1     Completed          0          89m
redhat-operators-catalog-d57ff6f8c-2hv9m             1/1     Running            0          89m
redhat-operators-catalog-rollout-27215316-5pl66      0/1     Completed          0          89m
Tylers-MacBook-Pro:armada tylerlisowski$ kubectl get pods -n master-prtesting49-57190034
NAME                                                 READY   STATUS             RESTARTS   AGE
catalog-operator-754b6dc8f7-d82vd                    1/1     Running            4          11h
certified-operators-catalog-57d5b9775d-7zp59         1/1     Running            0          89m
certified-operators-catalog-rollout-27215316-tt7vf   0/1     Completed          0          89m
cloud-controller-manager-ddc49497f-4669r             1/1     Running            1          11h
cloud-controller-manager-ddc49497f-gpkts             1/1     Running            0          11h
cloud-controller-manager-ddc49497f-hlfsn             1/1     Running            0          11h
cluster-api-78697c5c98-4wj4f                         1/1     Running            0          11h
cluster-api-78697c5c98-ptsqf                         1/1     Running            0          11h
cluster-api-78697c5c98-v5t8h                         1/1     Running            0          11h
cluster-autoscaler-f6f5cb844-xxmsl                   1/1     Running            1          11h
cluster-health-7b8b9b944d-jb7nh                      0/1     CrashLoopBackOff   136        10h
cluster-policy-controller-5bdd77ccfc-dltbf           1/1     Running            0          11h
cluster-policy-controller-5bdd77ccfc-mkshk           1/1     Running            0          11h
cluster-policy-controller-5bdd77ccfc-wvsg2           1/1     Running            0          11h
cluster-version-operator-857bd8cc5c-xjn5r            1/1     Running            0          11h
community-operators-catalog-86c89bcd7d-r89mf         1/1     Running            0          89m
community-operators-catalog-rollout-27215316-t4psf   0/1     Completed          0          89m
control-plane-operator-f489d8c94-vs548               1/1     Running            0          11h
etcd-operator-f8c5c4f5-5cr8p                         4/4     Running            4          11h
etcd-operator-f8c5c4f5-p6qtq                         4/4     Running            4          11h
etcd-operator-f8c5c4f5-qwb5s                         4/4     Running            4          11h
etcd-prtesting49-57190034-6lc77s9fln                 2/2     Running            0          11h
etcd-prtesting49-57190034-7nx2w75jvd                 2/2     Running            0          11h
etcd-prtesting49-57190034-br8f2b7krk                 2/2     Running            0          11h
etcd-restore-operator-cert-creator-gz4zl             0/1     Completed          0          11h
hosted-cluster-config-operator-664f4b4fb9-tgfdf      1/1     Running            3          11h
ignition-server-5d6d87fb75-nb8bd                     0/1     Running            0          36s
ignition-server-5d6d87fb75-qbr5w                     0/1     Running            0          36s
ignition-server-5d6d87fb75-wh92h                     0/1     Running            0          36s
konnectivity-agent-84d4545d7-vljwj                   1/1     Running            0          11h
konnectivity-agent-84d4545d7-wdmfk                   1/1     Running            0          11h
konnectivity-agent-84d4545d7-znrl9                   1/1     Running            0          11h
konnectivity-server-778479cc58-p5dx4                 1/1     Running            0          11h
kube-apiserver-677b5667bc-4pd2j                      2/2     Running            0          11h
kube-apiserver-677b5667bc-588hh                      2/2     Running            0          11h
kube-apiserver-677b5667bc-9jdqk                      2/2     Running            0          11h
kube-controller-manager-6d4f6c5b8d-jxm29             1/1     Running            0          10h
kube-controller-manager-6d4f6c5b8d-sg9j9             1/1     Running            0          10h
kube-controller-manager-6d4f6c5b8d-ths5c             1/1     Running            0          10h
kube-scheduler-c97c77d6f-2rtnm                       1/1     Running            0          11h
kube-scheduler-c97c77d6f-f72w5                       1/1     Running            0          11h
kube-scheduler-c97c77d6f-fbhfw                       1/1     Running            0          11h
machine-config-server-1018368211                     1/1     Running            0          10s
machine-config-server-1408474896                     1/1     Running            0          10s
machine-config-server-1463997948                     0/1     Terminating        0          14s
manifests-bootstrapper                               0/1     Completed          3          11h
oauth-openshift-855b8c4bfc-7v4rp                     1/1     Running            0          11h
oauth-openshift-855b8c4bfc-cx7z4                     1/1     Running            0          11h
oauth-openshift-855b8c4bfc-kl57h                     1/1     Running            0          11h
olm-operator-56b5bc7747-ntmxz                        1/1     Running            4          11h
openshift-apiserver-8489b9bb65-rp7vg                 2/2     Running            0          11h
openshift-apiserver-8489b9bb65-tj62l                 2/2     Running            0          11h
openshift-apiserver-8489b9bb65-xtlhz                 2/2     Running            0          11h
openshift-controller-manager-7bcc868796-7vmtb        1/1     Running            0          11h
openshift-controller-manager-7bcc868796-lq4bw        1/1     Running            0          11h
openshift-controller-manager-7bcc868796-nr6xt        1/1     Running            0          11h
openshift-oauth-apiserver-f5c699955-dn75t            1/1     Running            3          11h
openshift-oauth-apiserver-f5c699955-jwvdg            1/1     Running            3          11h
openshift-oauth-apiserver-f5c699955-kqv68            1/1     Running            4          11h
packageserver-568ddbbdf9-25n7z                       1/1     Running            2          11h
packageserver-568ddbbdf9-cgc55                       1/1     Running            4          11h
rbac-sync-operator-84d4c6f768-mzppk                  1/1     Running            0          10h
redhat-marketplace-catalog-6f9bfbdd6f-hx7hn          1/1     Running            0          89m
redhat-marketplace-catalog-rollout-27215316-mfdpp    0/1     Completed          0          89m
redhat-operators-catalog-d57ff6f8c-2hv9m             1/1     Running            0          89m
redhat-operators-catalog-rollout-27215316-5pl66      0/1     Completed          0          89m
Tylers-MacBook-Pro:armada tylerlisowski$ kubectl get pods -n master-prtesting49-57190034
NAME                                                 READY   STATUS             RESTARTS   AGE
catalog-operator-754b6dc8f7-d82vd                    1/1     Running            4          11h
certified-operators-catalog-57d5b9775d-7zp59         1/1     Running            0          89m
certified-operators-catalog-rollout-27215316-tt7vf   0/1     Completed          0          89m
cloud-controller-manager-ddc49497f-4669r             1/1     Running            1          11h
cloud-controller-manager-ddc49497f-gpkts             1/1     Running            0          11h
cloud-controller-manager-ddc49497f-hlfsn             1/1     Running            0          11h
cluster-api-78697c5c98-4wj4f                         1/1     Running            0          11h
cluster-api-78697c5c98-ptsqf                         1/1     Running            0          11h
cluster-api-78697c5c98-v5t8h                         1/1     Running            0          11h
cluster-autoscaler-f6f5cb844-xxmsl                   1/1     Running            1          11h
cluster-health-7b8b9b944d-jb7nh                      0/1     CrashLoopBackOff   136        10h
cluster-policy-controller-5bdd77ccfc-dltbf           1/1     Running            0          11h
cluster-policy-controller-5bdd77ccfc-mkshk           1/1     Running            0          11h
cluster-policy-controller-5bdd77ccfc-wvsg2           1/1     Running            0          11h
cluster-version-operator-857bd8cc5c-xjn5r            1/1     Running            0          11h
community-operators-catalog-86c89bcd7d-r89mf         1/1     Running            0          89m
community-operators-catalog-rollout-27215316-t4psf   0/1     Completed          0          89m
control-plane-operator-f489d8c94-vs548               1/1     Running            0          11h
etcd-operator-f8c5c4f5-5cr8p                         4/4     Running            4          11h
etcd-operator-f8c5c4f5-p6qtq                         4/4     Running            4          11h
etcd-operator-f8c5c4f5-qwb5s                         4/4     Running            4          11h
etcd-prtesting49-57190034-6lc77s9fln                 2/2     Running            0          11h
etcd-prtesting49-57190034-7nx2w75jvd                 2/2     Running            0          11h
etcd-prtesting49-57190034-br8f2b7krk                 2/2     Running            0          11h
etcd-restore-operator-cert-creator-gz4zl             0/1     Completed          0          11h
hosted-cluster-config-operator-664f4b4fb9-tgfdf      1/1     Running            3          11h
ignition-server-5d6d87fb75-nb8bd                     0/1     Running            0          37s
ignition-server-5d6d87fb75-qbr5w                     0/1     Running            0          37s
ignition-server-5d6d87fb75-wh92h                     0/1     Running            0          37s
konnectivity-agent-84d4545d7-vljwj                   1/1     Running            0          11h
konnectivity-agent-84d4545d7-wdmfk                   1/1     Running            0          11h
konnectivity-agent-84d4545d7-znrl9                   1/1     Running            0          11h
konnectivity-server-778479cc58-p5dx4                 1/1     Running            0          11h
kube-apiserver-677b5667bc-4pd2j                      2/2     Running            0          11h
kube-apiserver-677b5667bc-588hh                      2/2     Running            0          11h
kube-apiserver-677b5667bc-9jdqk                      2/2     Running            0          11h
kube-controller-manager-6d4f6c5b8d-jxm29             1/1     Running            0          10h
kube-controller-manager-6d4f6c5b8d-sg9j9             1/1     Running            0          10h
kube-controller-manager-6d4f6c5b8d-ths5c             1/1     Running            0          10h
kube-scheduler-c97c77d6f-2rtnm                       1/1     Running            0          11h
kube-scheduler-c97c77d6f-f72w5                       1/1     Running            0          11h
kube-scheduler-c97c77d6f-fbhfw                       1/1     Running            0          11h
machine-config-server-1018368211                     1/1     Terminating        0          11s
machine-config-server-1408474896                     1/1     Terminating        0          11s
machine-config-server-1463997948                     0/1     Terminating        0          15s
manifests-bootstrapper                               0/1     Completed          3          11h
oauth-openshift-855b8c4bfc-7v4rp                     1/1     Running            0          11h
oauth-openshift-855b8c4bfc-cx7z4                     1/1     Running            0          11h
oauth-openshift-855b8c4bfc-kl57h                     1/1     Running            0          11h
olm-operator-56b5bc7747-ntmxz                        1/1     Running            4          11h
openshift-apiserver-8489b9bb65-rp7vg                 2/2     Running            0          11h
openshift-apiserver-8489b9bb65-tj62l                 2/2     Running            0          11h
openshift-apiserver-8489b9bb65-xtlhz                 2/2     Running            0          11h
openshift-controller-manager-7bcc868796-7vmtb        1/1     Running            0          11h
openshift-controller-manager-7bcc868796-lq4bw        1/1     Running            0          11h
openshift-controller-manager-7bcc868796-nr6xt        1/1     Running            0          11h
openshift-oauth-apiserver-f5c699955-dn75t            1/1     Running            3          11h
openshift-oauth-apiserver-f5c699955-jwvdg            1/1     Running            3          11h
openshift-oauth-apiserver-f5c699955-kqv68            1/1     Running            4          11h
packageserver-568ddbbdf9-25n7z                       1/1     Running            2          11h
packageserver-568ddbbdf9-cgc55                       1/1     Running            4          11h
rbac-sync-operator-84d4c6f768-mzppk                  1/1     Running            0          10h
redhat-marketplace-catalog-6f9bfbdd6f-hx7hn          1/1     Running            0          89m
redhat-marketplace-catalog-rollout-27215316-mfdpp    0/1     Completed          0          89m
redhat-operators-catalog-d57ff6f8c-2hv9m             1/1     Running            0          89m
redhat-operators-catalog-rollout-27215316-5pl66      0/1     Completed          0          89m
Tylers-MacBook-Pro:armada tylerlisowski$ kubectl get pods -n master-prtesting49-57190034
NAME                                                 READY   STATUS             RESTARTS   AGE
catalog-operator-754b6dc8f7-d82vd                    1/1     Running            4          11h
certified-operators-catalog-57d5b9775d-7zp59         1/1     Running            0          89m
certified-operators-catalog-rollout-27215316-tt7vf   0/1     Completed          0          89m
cloud-controller-manager-ddc49497f-4669r             1/1     Running            1          11h
cloud-controller-manager-ddc49497f-gpkts             1/1     Running            0          11h
cloud-controller-manager-ddc49497f-hlfsn             1/1     Running            0          11h
cluster-api-78697c5c98-4wj4f                         1/1     Running            0          11h
cluster-api-78697c5c98-ptsqf                         1/1     Running            0          11h
cluster-api-78697c5c98-v5t8h                         1/1     Running            0          11h
cluster-autoscaler-f6f5cb844-xxmsl                   1/1     Running            1          11h
cluster-health-7b8b9b944d-jb7nh                      0/1     CrashLoopBackOff   136        10h
cluster-policy-controller-5bdd77ccfc-dltbf           1/1     Running            0          11h
cluster-policy-controller-5bdd77ccfc-mkshk           1/1     Running            0          11h
cluster-policy-controller-5bdd77ccfc-wvsg2           1/1     Running            0          11h
cluster-version-operator-857bd8cc5c-xjn5r            1/1     Running            0          11h
community-operators-catalog-86c89bcd7d-r89mf         1/1     Running            0          89m
community-operators-catalog-rollout-27215316-t4psf   0/1     Completed          0          89m
control-plane-operator-f489d8c94-vs548               1/1     Running            0          11h
etcd-operator-f8c5c4f5-5cr8p                         4/4     Running            4          11h
etcd-operator-f8c5c4f5-p6qtq                         4/4     Running            4          11h
etcd-operator-f8c5c4f5-qwb5s                         4/4     Running            4          11h
etcd-prtesting49-57190034-6lc77s9fln                 2/2     Running            0          11h
etcd-prtesting49-57190034-7nx2w75jvd                 2/2     Running            0          11h
etcd-prtesting49-57190034-br8f2b7krk                 2/2     Running            0          11h
etcd-restore-operator-cert-creator-gz4zl             0/1     Completed          0          11h
hosted-cluster-config-operator-664f4b4fb9-tgfdf      1/1     Running            3          11h
ignition-server-5d6d87fb75-nb8bd                     0/1     Running            0          40s
ignition-server-5d6d87fb75-qbr5w                     0/1     Running            0          40s
ignition-server-5d6d87fb75-wh92h                     0/1     Running            0          40s
konnectivity-agent-84d4545d7-vljwj                   1/1     Running            0          11h
konnectivity-agent-84d4545d7-wdmfk                   1/1     Running            0          11h
konnectivity-agent-84d4545d7-znrl9                   1/1     Running            0          11h
konnectivity-server-778479cc58-p5dx4                 1/1     Running            0          11h
kube-apiserver-677b5667bc-4pd2j                      2/2     Running            0          11h
kube-apiserver-677b5667bc-588hh                      2/2     Running            0          11h
kube-apiserver-677b5667bc-9jdqk                      2/2     Running            0          11h
kube-controller-manager-6d4f6c5b8d-jxm29             1/1     Running            0          10h
kube-controller-manager-6d4f6c5b8d-sg9j9             1/1     Running            0          10h
kube-controller-manager-6d4f6c5b8d-ths5c             1/1     Running            0          10h
kube-scheduler-c97c77d6f-2rtnm                       1/1     Running            0          11h
kube-scheduler-c97c77d6f-f72w5                       1/1     Running            0          11h
kube-scheduler-c97c77d6f-fbhfw                       1/1     Running            0          11h
machine-config-server-1018368211                     0/1     Terminating        0          14s
machine-config-server-1408474896                     0/1     Terminating        0          14s
machine-config-server-1463997948                     0/1     Terminating        0          18s
manifests-bootstrapper                               0/1     Completed          3          11h
oauth-openshift-855b8c4bfc-7v4rp                     1/1     Running            0          11h
oauth-openshift-855b8c4bfc-cx7z4                     1/1     Running            0          11h
oauth-openshift-855b8c4bfc-kl57h                     1/1     Running            0          11h
olm-operator-56b5bc7747-ntmxz                        1/1     Running            4          11h
openshift-apiserver-8489b9bb65-rp7vg                 2/2     Running            0          11h
openshift-apiserver-8489b9bb65-tj62l                 2/2     Running            0          11h
openshift-apiserver-8489b9bb65-xtlhz                 2/2     Running            0          11h
openshift-controller-manager-7bcc868796-7vmtb        1/1     Running            0          11h
openshift-controller-manager-7bcc868796-lq4bw        1/1     Running            0          11h
openshift-controller-manager-7bcc868796-nr6xt        1/1     Running            0          11h
openshift-oauth-apiserver-f5c699955-dn75t            1/1     Running            3          11h
openshift-oauth-apiserver-f5c699955-jwvdg            1/1     Running            3          11h
openshift-oauth-apiserver-f5c699955-kqv68            1/1     Running            4          11h
packageserver-568ddbbdf9-25n7z                       1/1     Running            2          11h
packageserver-568ddbbdf9-cgc55                       1/1     Running            4          11h
rbac-sync-operator-84d4c6f768-mzppk                  1/1     Running            0          10h
redhat-marketplace-catalog-6f9bfbdd6f-hx7hn          1/1     Running            0          89m
redhat-marketplace-catalog-rollout-27215316-mfdpp    0/1     Completed          0          89m
redhat-operators-catalog-d57ff6f8c-2hv9m             1/1     Running            0          89m
redhat-operators-catalog-rollout-27215316-5pl66      0/1     Completed          0          89m
Tylers-MacBook-Pro:armada tylerlisowski$ kubectl get pods -n master-prtesting49-57190034
NAME                                                 READY   STATUS             RESTARTS   AGE
catalog-operator-754b6dc8f7-d82vd                    1/1     Running            4          11h
certified-operators-catalog-57d5b9775d-7zp59         1/1     Running            0          89m
certified-operators-catalog-rollout-27215316-tt7vf   0/1     Completed          0          89m
cloud-controller-manager-ddc49497f-4669r             1/1     Running            1          11h
cloud-controller-manager-ddc49497f-gpkts             1/1     Running            0          11h
cloud-controller-manager-ddc49497f-hlfsn             1/1     Running            0          11h
cluster-api-78697c5c98-4wj4f                         1/1     Running            0          11h
cluster-api-78697c5c98-ptsqf                         1/1     Running            0          11h
cluster-api-78697c5c98-v5t8h                         1/1     Running            0          11h
cluster-autoscaler-f6f5cb844-xxmsl                   1/1     Running            1          11h
cluster-health-7b8b9b944d-jb7nh                      0/1     CrashLoopBackOff   136        10h
cluster-policy-controller-5bdd77ccfc-dltbf           1/1     Running            0          11h
cluster-policy-controller-5bdd77ccfc-mkshk           1/1     Running            0          11h
cluster-policy-controller-5bdd77ccfc-wvsg2           1/1     Running            0          11h
cluster-version-operator-857bd8cc5c-xjn5r            1/1     Running            0          11h
community-operators-catalog-86c89bcd7d-r89mf         1/1     Running            0          89m
community-operators-catalog-rollout-27215316-t4psf   0/1     Completed          0          89m
control-plane-operator-f489d8c94-vs548               1/1     Running            0          11h
etcd-operator-f8c5c4f5-5cr8p                         4/4     Running            4          11h
etcd-operator-f8c5c4f5-p6qtq                         4/4     Running            4          11h
etcd-operator-f8c5c4f5-qwb5s                         4/4     Running            4          11h
etcd-prtesting49-57190034-6lc77s9fln                 2/2     Running            0          11h
etcd-prtesting49-57190034-7nx2w75jvd                 2/2     Running            0          11h
etcd-prtesting49-57190034-br8f2b7krk                 2/2     Running            0          11h
etcd-restore-operator-cert-creator-gz4zl             0/1     Completed          0          11h
hosted-cluster-config-operator-664f4b4fb9-tgfdf      1/1     Running            3          11h
ignition-server-5d6d87fb75-nb8bd                     0/1     Running            0          41s
ignition-server-5d6d87fb75-qbr5w                     0/1     Running            0          41s
ignition-server-5d6d87fb75-wh92h                     0/1     Running            0          41s
konnectivity-agent-84d4545d7-vljwj                   1/1     Running            0          11h
konnectivity-agent-84d4545d7-wdmfk                   1/1     Running            0          11h
konnectivity-agent-84d4545d7-znrl9                   1/1     Running            0          11h
konnectivity-server-778479cc58-p5dx4                 1/1     Running            0          11h
kube-apiserver-677b5667bc-4pd2j                      2/2     Running            0          11h
kube-apiserver-677b5667bc-588hh                      2/2     Running            0          11h
kube-apiserver-677b5667bc-9jdqk                      2/2     Running            0          11h
kube-controller-manager-6d4f6c5b8d-jxm29             1/1     Running            0          10h
kube-controller-manager-6d4f6c5b8d-sg9j9             1/1     Running            0          10h
kube-controller-manager-6d4f6c5b8d-ths5c             1/1     Running            0          10h
kube-scheduler-c97c77d6f-2rtnm                       1/1     Running            0          11h
kube-scheduler-c97c77d6f-f72w5                       1/1     Running            0          11h
kube-scheduler-c97c77d6f-fbhfw                       1/1     Running            0          11h
machine-config-server-1018368211                     0/1     Terminating        0          15s
machine-config-server-1408474896                     0/1     Terminating        0          15s
manifests-bootstrapper                               0/1     Completed          3          11h
oauth-openshift-855b8c4bfc-7v4rp                     1/1     Running            0          11h
oauth-openshift-855b8c4bfc-cx7z4                     1/1     Running            0          11h
oauth-openshift-855b8c4bfc-kl57h                     1/1     Running            0          11h
olm-operator-56b5bc7747-ntmxz                        1/1     Running            4          11h
openshift-apiserver-8489b9bb65-rp7vg                 2/2     Running            0          11h
openshift-apiserver-8489b9bb65-tj62l                 2/2     Running            0          11h
openshift-apiserver-8489b9bb65-xtlhz                 2/2     Running            0          11h
openshift-controller-manager-7bcc868796-7vmtb        1/1     Running            0          11h
openshift-controller-manager-7bcc868796-lq4bw        1/1     Running            0          11h
openshift-controller-manager-7bcc868796-nr6xt        1/1     Running            0          11h
openshift-oauth-apiserver-f5c699955-dn75t            1/1     Running            3          11h
openshift-oauth-apiserver-f5c699955-jwvdg            1/1     Running            3          11h
openshift-oauth-apiserver-f5c699955-kqv68            1/1     Running            4          11h
packageserver-568ddbbdf9-25n7z                       1/1     Running            2          11h
packageserver-568ddbbdf9-cgc55                       1/1     Running            4          11h
rbac-sync-operator-84d4c6f768-mzppk                  1/1     Running            0          10h
redhat-marketplace-catalog-6f9bfbdd6f-hx7hn          1/1     Running            0          89m
redhat-marketplace-catalog-rollout-27215316-mfdpp    0/1     Completed          0          89m
redhat-operators-catalog-d57ff6f8c-2hv9m             1/1     Running            0          89m
redhat-operators-catalog-rollout-27215316-5pl66      0/1     Completed          0          89m
Tylers-MacBook-Pro:armada tylerlisowski$ kubectl get pods -n master-prtesting49-57190034
NAME                                                 READY   STATUS             RESTARTS   AGE
catalog-operator-754b6dc8f7-d82vd                    1/1     Running            4          11h
certified-operators-catalog-57d5b9775d-7zp59         1/1     Running            0          89m
certified-operators-catalog-rollout-27215316-tt7vf   0/1     Completed          0          89m
cloud-controller-manager-ddc49497f-4669r             1/1     Running            1          11h
cloud-controller-manager-ddc49497f-gpkts             1/1     Running            0          11h
cloud-controller-manager-ddc49497f-hlfsn             1/1     Running            0          11h
cluster-api-78697c5c98-4wj4f                         1/1     Running            0          11h
cluster-api-78697c5c98-ptsqf                         1/1     Running            0          11h
cluster-api-78697c5c98-v5t8h                         1/1     Running            0          11h
cluster-autoscaler-f6f5cb844-xxmsl                   1/1     Running            1          11h
cluster-health-7b8b9b944d-jb7nh                      0/1     CrashLoopBackOff   136        10h
cluster-policy-controller-5bdd77ccfc-dltbf           1/1     Running            0          11h
cluster-policy-controller-5bdd77ccfc-mkshk           1/1     Running            0          11h
cluster-policy-controller-5bdd77ccfc-wvsg2           1/1     Running            0          11h
cluster-version-operator-857bd8cc5c-xjn5r            1/1     Running            0          11h
community-operators-catalog-86c89bcd7d-r89mf         1/1     Running            0          89m
community-operators-catalog-rollout-27215316-t4psf   0/1     Completed          0          89m
control-plane-operator-f489d8c94-vs548               1/1     Running            0          11h
etcd-operator-f8c5c4f5-5cr8p                         4/4     Running            4          11h
etcd-operator-f8c5c4f5-p6qtq                         4/4     Running            4          11h
etcd-operator-f8c5c4f5-qwb5s                         4/4     Running            4          11h
etcd-prtesting49-57190034-6lc77s9fln                 2/2     Running            0          11h
etcd-prtesting49-57190034-7nx2w75jvd                 2/2     Running            0          11h
etcd-prtesting49-57190034-br8f2b7krk                 2/2     Running            0          11h
etcd-restore-operator-cert-creator-gz4zl             0/1     Completed          0          11h
hosted-cluster-config-operator-664f4b4fb9-tgfdf      1/1     Running            3          11h
ignition-server-5d6d87fb75-nb8bd                     0/1     Running            0          43s
ignition-server-5d6d87fb75-qbr5w                     0/1     Running            0          43s
ignition-server-5d6d87fb75-wh92h                     0/1     Running            0          43s
konnectivity-agent-84d4545d7-vljwj                   1/1     Running            0          11h
konnectivity-agent-84d4545d7-wdmfk                   1/1     Running            0          11h
konnectivity-agent-84d4545d7-znrl9                   1/1     Running            0          11h
konnectivity-server-778479cc58-p5dx4                 1/1     Running            0          11h
kube-apiserver-677b5667bc-4pd2j                      2/2     Running            0          11h
kube-apiserver-677b5667bc-588hh                      2/2     Running            0          11h
kube-apiserver-677b5667bc-9jdqk                      2/2     Running            0          11h
kube-controller-manager-6d4f6c5b8d-jxm29             1/1     Running            0          10h
kube-controller-manager-6d4f6c5b8d-sg9j9             1/1     Running            0          10h
kube-controller-manager-6d4f6c5b8d-ths5c             1/1     Running            0          10h
kube-scheduler-c97c77d6f-2rtnm                       1/1     Running            0          11h
kube-scheduler-c97c77d6f-f72w5                       1/1     Running            0          11h
kube-scheduler-c97c77d6f-fbhfw                       1/1     Running            0          11h
machine-config-server-1018368211                     0/1     Terminating        0          17s
machine-config-server-1408474896                     0/1     Terminating        0          17s
manifests-bootstrapper                               0/1     Completed          3          11h
oauth-openshift-855b8c4bfc-7v4rp                     1/1     Running            0          11h
oauth-openshift-855b8c4bfc-cx7z4                     1/1     Running            0          11h
oauth-openshift-855b8c4bfc-kl57h                     1/1     Running            0          11h
olm-operator-56b5bc7747-ntmxz                        1/1     Running            4          11h
openshift-apiserver-8489b9bb65-rp7vg                 2/2     Running            0          11h
openshift-apiserver-8489b9bb65-tj62l                 2/2     Running            0          11h
openshift-apiserver-8489b9bb65-xtlhz                 2/2     Running            0          11h
openshift-controller-manager-7bcc868796-7vmtb        1/1     Running            0          11h
openshift-controller-manager-7bcc868796-lq4bw        1/1     Running            0          11h
openshift-controller-manager-7bcc868796-nr6xt        1/1     Running            0          11h
openshift-oauth-apiserver-f5c699955-dn75t            1/1     Running            3          11h
openshift-oauth-apiserver-f5c699955-jwvdg            1/1     Running            3          11h
openshift-oauth-apiserver-f5c699955-kqv68            1/1     Running            4          11h
packageserver-568ddbbdf9-25n7z                       1/1     Running            2          11h
packageserver-568ddbbdf9-cgc55                       1/1     Running            4          11h
rbac-sync-operator-84d4c6f768-mzppk                  1/1     Running            0          10h
redhat-marketplace-catalog-6f9bfbdd6f-hx7hn          1/1     Running            0          89m
redhat-marketplace-catalog-rollout-27215316-mfdpp    0/1     Completed          0          89m
redhat-operators-catalog-d57ff6f8c-2hv9m             1/1     Running            0          89m
redhat-operators-catalog-rollout-27215316-5pl66      0/1     Completed          0          89m
Tylers-MacBook-Pro:armada tylerlisowski$ kubectl get pods -n master-prtesting49-57190034
NAME                                                 READY   STATUS             RESTARTS   AGE
catalog-operator-754b6dc8f7-d82vd                    1/1     Running            4          11h
certified-operators-catalog-57d5b9775d-7zp59         1/1     Running            0          89m
certified-operators-catalog-rollout-27215316-tt7vf   0/1     Completed          0          89m
cloud-controller-manager-ddc49497f-4669r             1/1     Running            1          11h
cloud-controller-manager-ddc49497f-gpkts             1/1     Running            0          11h
cloud-controller-manager-ddc49497f-hlfsn             1/1     Running            0          11h
cluster-api-78697c5c98-4wj4f                         1/1     Running            0          11h
cluster-api-78697c5c98-ptsqf                         1/1     Running            0          11h
cluster-api-78697c5c98-v5t8h                         1/1     Running            0          11h
cluster-autoscaler-f6f5cb844-xxmsl                   1/1     Running            1          11h
cluster-health-7b8b9b944d-jb7nh                      0/1     CrashLoopBackOff   136        10h
cluster-policy-controller-5bdd77ccfc-dltbf           1/1     Running            0          11h
cluster-policy-controller-5bdd77ccfc-mkshk           1/1     Running            0          11h
cluster-policy-controller-5bdd77ccfc-wvsg2           1/1     Running            0          11h
cluster-version-operator-857bd8cc5c-xjn5r            1/1     Running            0          11h
community-operators-catalog-86c89bcd7d-r89mf         1/1     Running            0          89m
community-operators-catalog-rollout-27215316-t4psf   0/1     Completed          0          89m
control-plane-operator-f489d8c94-vs548               1/1     Running            0          11h
etcd-operator-f8c5c4f5-5cr8p                         4/4     Running            4          11h
etcd-operator-f8c5c4f5-p6qtq                         4/4     Running            4          11h
etcd-operator-f8c5c4f5-qwb5s                         4/4     Running            4          11h
etcd-prtesting49-57190034-6lc77s9fln                 2/2     Running            0          11h
etcd-prtesting49-57190034-7nx2w75jvd                 2/2     Running            0          11h
etcd-prtesting49-57190034-br8f2b7krk                 2/2     Running            0          11h
etcd-restore-operator-cert-creator-gz4zl             0/1     Completed          0          11h
hosted-cluster-config-operator-664f4b4fb9-tgfdf      1/1     Running            3          11h
ignition-server-5d6d87fb75-nb8bd                     0/1     Running            0          44s
ignition-server-5d6d87fb75-qbr5w                     0/1     Running            0          44s
ignition-server-5d6d87fb75-wh92h                     0/1     Running            0          44s
konnectivity-agent-84d4545d7-vljwj                   1/1     Running            0          11h
konnectivity-agent-84d4545d7-wdmfk                   1/1     Running            0          11h
konnectivity-agent-84d4545d7-znrl9                   1/1     Running            0          11h
konnectivity-server-778479cc58-p5dx4                 1/1     Running            0          11h
kube-apiserver-677b5667bc-4pd2j                      2/2     Running            0          11h
kube-apiserver-677b5667bc-588hh                      2/2     Running            0          11h
kube-apiserver-677b5667bc-9jdqk                      2/2     Running            0          11h
kube-controller-manager-6d4f6c5b8d-jxm29             1/1     Running            0          10h
kube-controller-manager-6d4f6c5b8d-sg9j9             1/1     Running            0          10h
kube-controller-manager-6d4f6c5b8d-ths5c             1/1     Running            0          10h
kube-scheduler-c97c77d6f-2rtnm                       1/1     Running            0          11h
kube-scheduler-c97c77d6f-f72w5                       1/1     Running            0          11h
kube-scheduler-c97c77d6f-fbhfw                       1/1     Running            0          11h
machine-config-server-1018368211                     0/1     Terminating        0          18s
machine-config-server-1408474896                     0/1     Terminating        0          18s
manifests-bootstrapper                               0/1     Completed          3          11h
oauth-openshift-855b8c4bfc-7v4rp                     1/1     Running            0          11h
oauth-openshift-855b8c4bfc-cx7z4                     1/1     Running            0          11h
oauth-openshift-855b8c4bfc-kl57h                     1/1     Running            0          11h
olm-operator-56b5bc7747-ntmxz                        1/1     Running            4          11h
openshift-apiserver-8489b9bb65-rp7vg                 2/2     Running            0          11h
openshift-apiserver-8489b9bb65-tj62l                 2/2     Running            0          11h
openshift-apiserver-8489b9bb65-xtlhz                 2/2     Running            0          11h
openshift-controller-manager-7bcc868796-7vmtb        1/1     Running            0          11h
openshift-controller-manager-7bcc868796-lq4bw        1/1     Running            0          11h
openshift-controller-manager-7bcc868796-nr6xt        1/1     Running            0          11h
openshift-oauth-apiserver-f5c699955-dn75t            1/1     Running            3          11h
openshift-oauth-apiserver-f5c699955-jwvdg            1/1     Running            3          11h
openshift-oauth-apiserver-f5c699955-kqv68            1/1     Running            4          11h
packageserver-568ddbbdf9-25n7z                       1/1     Running            2          11h
packageserver-568ddbbdf9-cgc55                       1/1     Running            4          11h
rbac-sync-operator-84d4c6f768-mzppk                  1/1     Running            0          10h
redhat-marketplace-catalog-6f9bfbdd6f-hx7hn          1/1     Running            0          89m
redhat-marketplace-catalog-rollout-27215316-mfdpp    0/1     Completed          0          89m
redhat-operators-catalog-d57ff6f8c-2hv9m             1/1     Running            0          89m
redhat-operators-catalog-rollout-27215316-5pl66      0/1     Completed          0          89m
Tylers-MacBook-Pro:armada tylerlisowski$ kubectl get pods -n master-prtesting49-57190034
NAME                                                 READY   STATUS             RESTARTS   AGE
catalog-operator-754b6dc8f7-d82vd                    1/1     Running            4          11h
certified-operators-catalog-57d5b9775d-7zp59         1/1     Running            0          89m
certified-operators-catalog-rollout-27215316-tt7vf   0/1     Completed          0          89m
cloud-controller-manager-ddc49497f-4669r             1/1     Running            1          11h
cloud-controller-manager-ddc49497f-gpkts             1/1     Running            0          11h
cloud-controller-manager-ddc49497f-hlfsn             1/1     Running            0          11h
cluster-api-78697c5c98-4wj4f                         1/1     Running            0          11h
cluster-api-78697c5c98-ptsqf                         1/1     Running            0          11h
cluster-api-78697c5c98-v5t8h                         1/1     Running            0          11h
cluster-autoscaler-f6f5cb844-xxmsl                   1/1     Running            1          11h
cluster-health-7b8b9b944d-jb7nh                      0/1     CrashLoopBackOff   136        10h
cluster-policy-controller-5bdd77ccfc-dltbf           1/1     Running            0          11h
cluster-policy-controller-5bdd77ccfc-mkshk           1/1     Running            0          11h
cluster-policy-controller-5bdd77ccfc-wvsg2           1/1     Running            0          11h
cluster-version-operator-857bd8cc5c-xjn5r            1/1     Running            0          11h
community-operators-catalog-86c89bcd7d-r89mf         1/1     Running            0          89m
community-operators-catalog-rollout-27215316-t4psf   0/1     Completed          0          89m
control-plane-operator-f489d8c94-vs548               1/1     Running            0          11h
etcd-operator-f8c5c4f5-5cr8p                         4/4     Running            4          11h
etcd-operator-f8c5c4f5-p6qtq                         4/4     Running            4          11h
etcd-operator-f8c5c4f5-qwb5s                         4/4     Running            4          11h
etcd-prtesting49-57190034-6lc77s9fln                 2/2     Running            0          11h
etcd-prtesting49-57190034-7nx2w75jvd                 2/2     Running            0          11h
etcd-prtesting49-57190034-br8f2b7krk                 2/2     Running            0          11h
etcd-restore-operator-cert-creator-gz4zl             0/1     Completed          0          11h
hosted-cluster-config-operator-664f4b4fb9-tgfdf      1/1     Running            3          11h
ignition-server-5d6d87fb75-nb8bd                     0/1     Running            0          46s
ignition-server-5d6d87fb75-qbr5w                     0/1     Running            0          46s
ignition-server-5d6d87fb75-wh92h                     0/1     Running            0          46s
konnectivity-agent-84d4545d7-vljwj                   1/1     Running            0          11h
konnectivity-agent-84d4545d7-wdmfk                   1/1     Running            0          11h
konnectivity-agent-84d4545d7-znrl9                   1/1     Running            0          11h
konnectivity-server-778479cc58-p5dx4                 1/1     Running            0          11h
kube-apiserver-677b5667bc-4pd2j                      2/2     Running            0          11h
kube-apiserver-677b5667bc-588hh                      2/2     Running            0          11h
kube-apiserver-677b5667bc-9jdqk                      2/2     Running            0          11h
kube-controller-manager-6d4f6c5b8d-jxm29             1/1     Running            0          10h
kube-controller-manager-6d4f6c5b8d-sg9j9             1/1     Running            0          10h
kube-controller-manager-6d4f6c5b8d-ths5c             1/1     Running            0          10h
kube-scheduler-c97c77d6f-2rtnm                       1/1     Running            0          11h
kube-scheduler-c97c77d6f-f72w5                       1/1     Running            0          11h
kube-scheduler-c97c77d6f-fbhfw                       1/1     Running            0          11h
machine-config-server-1018368211                     0/1     Terminating        0          20s
machine-config-server-1408474896                     0/1     Terminating        0          20s
manifests-bootstrapper                               0/1     Completed          3          11h
oauth-openshift-855b8c4bfc-7v4rp                     1/1     Running            0          11h
oauth-openshift-855b8c4bfc-cx7z4                     1/1     Running            0          11h
oauth-openshift-855b8c4bfc-kl57h                     1/1     Running            0          11h
olm-operator-56b5bc7747-ntmxz                        1/1     Running            4          11h
openshift-apiserver-8489b9bb65-rp7vg                 2/2     Running            0          11h
openshift-apiserver-8489b9bb65-tj62l                 2/2     Running            0          11h
openshift-apiserver-8489b9bb65-xtlhz                 2/2     Running            0          11h
openshift-controller-manager-7bcc868796-7vmtb        1/1     Running            0          11h
openshift-controller-manager-7bcc868796-lq4bw        1/1     Running            0          11h
openshift-controller-manager-7bcc868796-nr6xt        1/1     Running            0          11h
openshift-oauth-apiserver-f5c699955-dn75t            1/1     Running            3          11h
openshift-oauth-apiserver-f5c699955-jwvdg            1/1     Running            3          11h
openshift-oauth-apiserver-f5c699955-kqv68            1/1     Running            4          11h
packageserver-568ddbbdf9-25n7z                       1/1     Running            2          11h
packageserver-568ddbbdf9-cgc55                       1/1     Running            4          11h
rbac-sync-operator-84d4c6f768-mzppk                  1/1     Running            0          10h
redhat-marketplace-catalog-6f9bfbdd6f-hx7hn          1/1     Running            0          89m
redhat-marketplace-catalog-rollout-27215316-mfdpp    0/1     Completed          0          89m
redhat-operators-catalog-d57ff6f8c-2hv9m             1/1     Running            0          89m
redhat-operators-catalog-rollout-27215316-5pl66      0/1     Completed          0          89m
Tylers-MacBook-Pro:armada tylerlisowski$ kubectl get pods -n master-prtesting49-57190034
NAME                                                 READY   STATUS             RESTARTS   AGE
catalog-operator-754b6dc8f7-d82vd                    1/1     Running            4          11h
certified-operators-catalog-57d5b9775d-7zp59         1/1     Running            0          89m
certified-operators-catalog-rollout-27215316-tt7vf   0/1     Completed          0          89m
cloud-controller-manager-ddc49497f-4669r             1/1     Running            1          11h
cloud-controller-manager-ddc49497f-gpkts             1/1     Running            0          11h
cloud-controller-manager-ddc49497f-hlfsn             1/1     Running            0          11h
cluster-api-78697c5c98-4wj4f                         1/1     Running            0          11h
cluster-api-78697c5c98-ptsqf                         1/1     Running            0          11h
cluster-api-78697c5c98-v5t8h                         1/1     Running            0          11h
cluster-autoscaler-f6f5cb844-xxmsl                   1/1     Running            1          11h
cluster-health-7b8b9b944d-jb7nh                      0/1     CrashLoopBackOff   136        10h
cluster-policy-controller-5bdd77ccfc-dltbf           1/1     Running            0          11h
cluster-policy-controller-5bdd77ccfc-mkshk           1/1     Running            0          11h
cluster-policy-controller-5bdd77ccfc-wvsg2           1/1     Running            0          11h
cluster-version-operator-857bd8cc5c-xjn5r            1/1     Running            0          11h
community-operators-catalog-86c89bcd7d-r89mf         1/1     Running            0          89m
community-operators-catalog-rollout-27215316-t4psf   0/1     Completed          0          89m
control-plane-operator-f489d8c94-vs548               1/1     Running            0          11h
etcd-operator-f8c5c4f5-5cr8p                         4/4     Running            4          11h
etcd-operator-f8c5c4f5-p6qtq                         4/4     Running            4          11h
etcd-operator-f8c5c4f5-qwb5s                         4/4     Running            4          11h
etcd-prtesting49-57190034-6lc77s9fln                 2/2     Running            0          11h
etcd-prtesting49-57190034-7nx2w75jvd                 2/2     Running            0          11h
etcd-prtesting49-57190034-br8f2b7krk                 2/2     Running            0          11h
etcd-restore-operator-cert-creator-gz4zl             0/1     Completed          0          11h
hosted-cluster-config-operator-664f4b4fb9-tgfdf      1/1     Running            3          11h
ignition-server-5d6d87fb75-nb8bd                     0/1     Running            0          47s
ignition-server-5d6d87fb75-qbr5w                     0/1     Running            0          47s
ignition-server-5d6d87fb75-wh92h                     0/1     Running            0          47s
konnectivity-agent-84d4545d7-vljwj                   1/1     Running            0          11h
konnectivity-agent-84d4545d7-wdmfk                   1/1     Running            0          11h
konnectivity-agent-84d4545d7-znrl9                   1/1     Running            0          11h
konnectivity-server-778479cc58-p5dx4                 1/1     Running            0          11h
kube-apiserver-677b5667bc-4pd2j                      2/2     Running            0          11h
kube-apiserver-677b5667bc-588hh                      2/2     Running            0          11h
kube-apiserver-677b5667bc-9jdqk                      2/2     Running            0          11h
kube-controller-manager-6d4f6c5b8d-jxm29             1/1     Running            0          10h
kube-controller-manager-6d4f6c5b8d-sg9j9             1/1     Running            0          10h
kube-controller-manager-6d4f6c5b8d-ths5c             1/1     Running            0          10h
kube-scheduler-c97c77d6f-2rtnm                       1/1     Running            0          11h
kube-scheduler-c97c77d6f-f72w5                       1/1     Running            0          11h
kube-scheduler-c97c77d6f-fbhfw                       1/1     Running            0          11h
machine-config-server-1018368211                     0/1     Terminating        0          21s
manifests-bootstrapper                               0/1     Completed          3          11h
oauth-openshift-855b8c4bfc-7v4rp                     1/1     Running            0          11h
oauth-openshift-855b8c4bfc-cx7z4                     1/1     Running            0          11h
oauth-openshift-855b8c4bfc-kl57h                     1/1     Running            0          11h
olm-operator-56b5bc7747-ntmxz                        1/1     Running            4          11h
openshift-apiserver-8489b9bb65-rp7vg                 2/2     Running            0          11h
openshift-apiserver-8489b9bb65-tj62l                 2/2     Running            0          11h
openshift-apiserver-8489b9bb65-xtlhz                 2/2     Running            0          11h
openshift-controller-manager-7bcc868796-7vmtb        1/1     Running            0          11h
openshift-controller-manager-7bcc868796-lq4bw        1/1     Running            0          11h
openshift-controller-manager-7bcc868796-nr6xt        1/1     Running            0          11h
openshift-oauth-apiserver-f5c699955-dn75t            1/1     Running            3          11h
openshift-oauth-apiserver-f5c699955-jwvdg            1/1     Running            3          11h
openshift-oauth-apiserver-f5c699955-kqv68            1/1     Running            4          11h
packageserver-568ddbbdf9-25n7z                       1/1     Running            2          11h
packageserver-568ddbbdf9-cgc55                       1/1     Running            4          11h
rbac-sync-operator-84d4c6f768-mzppk                  1/1     Running            0          10h
redhat-marketplace-catalog-6f9bfbdd6f-hx7hn          1/1     Running            0          89m
redhat-marketplace-catalog-rollout-27215316-mfdpp    0/1     Completed          0          89m
redhat-operators-catalog-d57ff6f8c-2hv9m             1/1     Running            0          89m
redhat-operators-catalog-rollout-27215316-5pl66      0/1     Completed          0          89m
Tylers-MacBook-Pro:armada tylerlisowski$ kubectl get pods -n master-prtesting49-57190034
NAME                                                 READY   STATUS             RESTARTS   AGE
catalog-operator-754b6dc8f7-d82vd                    1/1     Running            4          11h
certified-operators-catalog-57d5b9775d-7zp59         1/1     Running            0          89m
certified-operators-catalog-rollout-27215316-tt7vf   0/1     Completed          0          89m
cloud-controller-manager-ddc49497f-4669r             1/1     Running            1          11h
cloud-controller-manager-ddc49497f-gpkts             1/1     Running            0          11h
cloud-controller-manager-ddc49497f-hlfsn             1/1     Running            0          11h
cluster-api-78697c5c98-4wj4f                         1/1     Running            0          11h
cluster-api-78697c5c98-ptsqf                         1/1     Running            0          11h
cluster-api-78697c5c98-v5t8h                         1/1     Running            0          11h
cluster-autoscaler-f6f5cb844-xxmsl                   1/1     Running            1          11h
cluster-health-7b8b9b944d-jb7nh                      0/1     CrashLoopBackOff   136        10h
cluster-policy-controller-5bdd77ccfc-dltbf           1/1     Running            0          11h
cluster-policy-controller-5bdd77ccfc-mkshk           1/1     Running            0          11h
cluster-policy-controller-5bdd77ccfc-wvsg2           1/1     Running            0          11h
cluster-version-operator-857bd8cc5c-xjn5r            1/1     Running            0          11h
community-operators-catalog-86c89bcd7d-r89mf         1/1     Running            0          89m
community-operators-catalog-rollout-27215316-t4psf   0/1     Completed          0          89m
control-plane-operator-f489d8c94-vs548               1/1     Running            0          11h
etcd-operator-f8c5c4f5-5cr8p                         4/4     Running            4          11h
etcd-operator-f8c5c4f5-p6qtq                         4/4     Running            4          11h
etcd-operator-f8c5c4f5-qwb5s                         4/4     Running            4          11h
etcd-prtesting49-57190034-6lc77s9fln                 2/2     Running            0          11h
etcd-prtesting49-57190034-7nx2w75jvd                 2/2     Running            0          11h
etcd-prtesting49-57190034-br8f2b7krk                 2/2     Running            0          11h
etcd-restore-operator-cert-creator-gz4zl             0/1     Completed          0          11h
hosted-cluster-config-operator-664f4b4fb9-tgfdf      1/1     Running            3          11h
ignition-server-5d6d87fb75-nb8bd                     0/1     Running            0          49s
ignition-server-5d6d87fb75-qbr5w                     0/1     Running            0          49s
ignition-server-5d6d87fb75-wh92h                     0/1     Running            0          49s
konnectivity-agent-84d4545d7-vljwj                   1/1     Running            0          11h
konnectivity-agent-84d4545d7-wdmfk                   1/1     Running            0          11h
konnectivity-agent-84d4545d7-znrl9                   1/1     Running            0          11h
konnectivity-server-778479cc58-p5dx4                 1/1     Running            0          11h
kube-apiserver-677b5667bc-4pd2j                      2/2     Running            0          11h
kube-apiserver-677b5667bc-588hh                      2/2     Running            0          11h
kube-apiserver-677b5667bc-9jdqk                      2/2     Running            0          11h
kube-controller-manager-6d4f6c5b8d-jxm29             1/1     Running            0          10h
kube-controller-manager-6d4f6c5b8d-sg9j9             1/1     Running            0          10h
kube-controller-manager-6d4f6c5b8d-ths5c             1/1     Running            0          10h
kube-scheduler-c97c77d6f-2rtnm                       1/1     Running            0          11h
kube-scheduler-c97c77d6f-f72w5                       1/1     Running            0          11h
kube-scheduler-c97c77d6f-fbhfw                       1/1     Running            0          11h
machine-config-server-1018368211                     0/1     Terminating        0          23s
manifests-bootstrapper                               0/1     Completed          3          11h
oauth-openshift-855b8c4bfc-7v4rp                     1/1     Running            0          11h
oauth-openshift-855b8c4bfc-cx7z4                     1/1     Running            0          11h
oauth-openshift-855b8c4bfc-kl57h                     1/1     Running            0          11h
olm-operator-56b5bc7747-ntmxz                        1/1     Running            4          11h
openshift-apiserver-8489b9bb65-rp7vg                 2/2     Running            0          11h
openshift-apiserver-8489b9bb65-tj62l                 2/2     Running            0          11h
openshift-apiserver-8489b9bb65-xtlhz                 2/2     Running            0          11h
openshift-controller-manager-7bcc868796-7vmtb        1/1     Running            0          11h
openshift-controller-manager-7bcc868796-lq4bw        1/1     Running            0          11h
openshift-controller-manager-7bcc868796-nr6xt        1/1     Running            0          11h
openshift-oauth-apiserver-f5c699955-dn75t            1/1     Running            3          11h
openshift-oauth-apiserver-f5c699955-jwvdg            1/1     Running            3          11h
openshift-oauth-apiserver-f5c699955-kqv68            1/1     Running            4          11h
packageserver-568ddbbdf9-25n7z                       1/1     Running            2          11h
packageserver-568ddbbdf9-cgc55                       1/1     Running            4          11h
rbac-sync-operator-84d4c6f768-mzppk                  1/1     Running            0          10h
redhat-marketplace-catalog-6f9bfbdd6f-hx7hn          1/1     Running            0          89m
redhat-marketplace-catalog-rollout-27215316-mfdpp    0/1     Completed          0          89m
redhat-operators-catalog-d57ff6f8c-2hv9m             1/1     Running            0          89m
redhat-operators-catalog-rollout-27215316-5pl66      0/1     Completed          0          89m
Tylers-MacBook-Pro:armada tylerlisowski$ kubectl get pods -n master-prtesting49-57190034
NAME                                                 READY   STATUS             RESTARTS   AGE
catalog-operator-754b6dc8f7-d82vd                    1/1     Running            4          11h
certified-operators-catalog-57d5b9775d-7zp59         1/1     Running            0          89m
certified-operators-catalog-rollout-27215316-tt7vf   0/1     Completed          0          89m
cloud-controller-manager-ddc49497f-4669r             1/1     Running            1          11h
cloud-controller-manager-ddc49497f-gpkts             1/1     Running            0          11h
cloud-controller-manager-ddc49497f-hlfsn             1/1     Running            0          11h
cluster-api-78697c5c98-4wj4f                         1/1     Running            0          11h
cluster-api-78697c5c98-ptsqf                         1/1     Running            0          11h
cluster-api-78697c5c98-v5t8h                         1/1     Running            0          11h
cluster-autoscaler-f6f5cb844-xxmsl                   1/1     Running            1          11h
cluster-health-7b8b9b944d-jb7nh                      0/1     CrashLoopBackOff   136        10h
cluster-policy-controller-5bdd77ccfc-dltbf           1/1     Running            0          11h
cluster-policy-controller-5bdd77ccfc-mkshk           1/1     Running            0          11h
cluster-policy-controller-5bdd77ccfc-wvsg2           1/1     Running            0          11h
cluster-version-operator-857bd8cc5c-xjn5r            1/1     Running            0          11h
community-operators-catalog-86c89bcd7d-r89mf         1/1     Running            0          89m
community-operators-catalog-rollout-27215316-t4psf   0/1     Completed          0          89m
control-plane-operator-f489d8c94-vs548               1/1     Running            0          11h
etcd-operator-f8c5c4f5-5cr8p                         4/4     Running            4          11h
etcd-operator-f8c5c4f5-p6qtq                         4/4     Running            4          11h
etcd-operator-f8c5c4f5-qwb5s                         4/4     Running            4          11h
etcd-prtesting49-57190034-6lc77s9fln                 2/2     Running            0          11h
etcd-prtesting49-57190034-7nx2w75jvd                 2/2     Running            0          11h
etcd-prtesting49-57190034-br8f2b7krk                 2/2     Running            0          11h
etcd-restore-operator-cert-creator-gz4zl             0/1     Completed          0          11h
hosted-cluster-config-operator-664f4b4fb9-tgfdf      1/1     Running            3          11h
ignition-server-5d6d87fb75-nb8bd                     0/1     Running            0          50s
ignition-server-5d6d87fb75-qbr5w                     0/1     Running            0          50s
ignition-server-5d6d87fb75-wh92h                     0/1     Running            0          50s
konnectivity-agent-84d4545d7-vljwj                   1/1     Running            0          11h
konnectivity-agent-84d4545d7-wdmfk                   1/1     Running            0          11h
konnectivity-agent-84d4545d7-znrl9                   1/1     Running            0          11h
konnectivity-server-778479cc58-p5dx4                 1/1     Running            0          11h
kube-apiserver-677b5667bc-4pd2j                      2/2     Running            0          11h
kube-apiserver-677b5667bc-588hh                      2/2     Running            0          11h
kube-apiserver-677b5667bc-9jdqk                      2/2     Running            0          11h
kube-controller-manager-6d4f6c5b8d-jxm29             1/1     Running            0          10h
kube-controller-manager-6d4f6c5b8d-sg9j9             1/1     Running            0          10h
kube-controller-manager-6d4f6c5b8d-ths5c             1/1     Running            0          10h
kube-scheduler-c97c77d6f-2rtnm                       1/1     Running            0          11h
kube-scheduler-c97c77d6f-f72w5                       1/1     Running            0          11h
kube-scheduler-c97c77d6f-fbhfw                       1/1     Running            0          11h
machine-config-server-1018368211                     0/1     Terminating        0          24s
manifests-bootstrapper                               0/1     Completed          3          11h
oauth-openshift-855b8c4bfc-7v4rp                     1/1     Running            0          11h
oauth-openshift-855b8c4bfc-cx7z4                     1/1     Running            0          11h
oauth-openshift-855b8c4bfc-kl57h                     1/1     Running            0          11h
olm-operator-56b5bc7747-ntmxz                        1/1     Running            4          11h
openshift-apiserver-8489b9bb65-rp7vg                 2/2     Running            0          11h
openshift-apiserver-8489b9bb65-tj62l                 2/2     Running            0          11h
openshift-apiserver-8489b9bb65-xtlhz                 2/2     Running            0          11h
openshift-controller-manager-7bcc868796-7vmtb        1/1     Running            0          11h
openshift-controller-manager-7bcc868796-lq4bw        1/1     Running            0          11h
openshift-controller-manager-7bcc868796-nr6xt        1/1     Running            0          11h
openshift-oauth-apiserver-f5c699955-dn75t            1/1     Running            3          11h
openshift-oauth-apiserver-f5c699955-jwvdg            1/1     Running            3          11h
openshift-oauth-apiserver-f5c699955-kqv68            1/1     Running            4          11h
packageserver-568ddbbdf9-25n7z                       1/1     Running            2          11h
packageserver-568ddbbdf9-cgc55                       1/1     Running            4          11h
rbac-sync-operator-84d4c6f768-mzppk                  1/1     Running            0          10h
redhat-marketplace-catalog-6f9bfbdd6f-hx7hn          1/1     Running            0          89m
redhat-marketplace-catalog-rollout-27215316-mfdpp    0/1     Completed          0          89m
redhat-operators-catalog-d57ff6f8c-2hv9m             1/1     Running            0          89m
redhat-operators-catalog-rollout-27215316-5pl66      0/1     Completed          0          89m
Tylers-MacBook-Pro:armada tylerlisowski$ kubectl get pods -n master-prtesting49-57190034
NAME                                                 READY   STATUS             RESTARTS   AGE
catalog-operator-754b6dc8f7-d82vd                    1/1     Running            4          11h
certified-operators-catalog-57d5b9775d-7zp59         1/1     Running            0          89m
certified-operators-catalog-rollout-27215316-tt7vf   0/1     Completed          0          89m
cloud-controller-manager-ddc49497f-4669r             1/1     Running            1          11h
cloud-controller-manager-ddc49497f-gpkts             1/1     Running            0          11h
cloud-controller-manager-ddc49497f-hlfsn             1/1     Running            0          11h
cluster-api-78697c5c98-4wj4f                         1/1     Running            0          11h
cluster-api-78697c5c98-ptsqf                         1/1     Running            0          11h
cluster-api-78697c5c98-v5t8h                         1/1     Running            0          11h
cluster-autoscaler-f6f5cb844-xxmsl                   1/1     Running            1          11h
cluster-health-7b8b9b944d-jb7nh                      0/1     CrashLoopBackOff   136        10h
cluster-policy-controller-5bdd77ccfc-dltbf           1/1     Running            0          11h
cluster-policy-controller-5bdd77ccfc-mkshk           1/1     Running            0          11h
cluster-policy-controller-5bdd77ccfc-wvsg2           1/1     Running            0          11h
cluster-version-operator-857bd8cc5c-xjn5r            1/1     Running            0          11h
community-operators-catalog-86c89bcd7d-r89mf         1/1     Running            0          89m
community-operators-catalog-rollout-27215316-t4psf   0/1     Completed          0          89m
control-plane-operator-f489d8c94-vs548               1/1     Running            0          11h
etcd-operator-f8c5c4f5-5cr8p                         4/4     Running            4          11h
etcd-operator-f8c5c4f5-p6qtq                         4/4     Running            4          11h
etcd-operator-f8c5c4f5-qwb5s                         4/4     Running            4          11h
etcd-prtesting49-57190034-6lc77s9fln                 2/2     Running            0          11h
etcd-prtesting49-57190034-7nx2w75jvd                 2/2     Running            0          11h
etcd-prtesting49-57190034-br8f2b7krk                 2/2     Running            0          11h
etcd-restore-operator-cert-creator-gz4zl             0/1     Completed          0          11h
hosted-cluster-config-operator-664f4b4fb9-tgfdf      1/1     Running            3          11h
ignition-server-5d6d87fb75-nb8bd                     0/1     Running            0          52s
ignition-server-5d6d87fb75-qbr5w                     0/1     Running            0          52s
ignition-server-5d6d87fb75-wh92h                     0/1     Running            0          52s
konnectivity-agent-84d4545d7-vljwj                   1/1     Running            0          11h
konnectivity-agent-84d4545d7-wdmfk                   1/1     Running            0          11h
konnectivity-agent-84d4545d7-znrl9                   1/1     Running            0          11h
konnectivity-server-778479cc58-p5dx4                 1/1     Running            0          11h
kube-apiserver-677b5667bc-4pd2j                      2/2     Running            0          11h
kube-apiserver-677b5667bc-588hh                      2/2     Running            0          11h
kube-apiserver-677b5667bc-9jdqk                      2/2     Running            0          11h
kube-controller-manager-6d4f6c5b8d-jxm29             1/1     Running            0          10h
kube-controller-manager-6d4f6c5b8d-sg9j9             1/1     Running            0          10h
kube-controller-manager-6d4f6c5b8d-ths5c             1/1     Running            0          10h
kube-scheduler-c97c77d6f-2rtnm                       1/1     Running            0          11h
kube-scheduler-c97c77d6f-f72w5                       1/1     Running            0          11h
kube-scheduler-c97c77d6f-fbhfw                       1/1     Running            0          11h
manifests-bootstrapper                               0/1     Completed          3          11h
oauth-openshift-855b8c4bfc-7v4rp                     1/1     Running            0          11h
oauth-openshift-855b8c4bfc-cx7z4                     1/1     Running            0          11h
oauth-openshift-855b8c4bfc-kl57h                     1/1     Running            0          11h
olm-operator-56b5bc7747-ntmxz                        1/1     Running            4          11h
openshift-apiserver-8489b9bb65-rp7vg                 2/2     Running            0          11h
openshift-apiserver-8489b9bb65-tj62l                 2/2     Running            0          11h
openshift-apiserver-8489b9bb65-xtlhz                 2/2     Running            0          11h
openshift-controller-manager-7bcc868796-7vmtb        1/1     Running            0          11h
openshift-controller-manager-7bcc868796-lq4bw        1/1     Running            0          11h
openshift-controller-manager-7bcc868796-nr6xt        1/1     Running            0          11h
openshift-oauth-apiserver-f5c699955-dn75t            1/1     Running            3          11h
openshift-oauth-apiserver-f5c699955-jwvdg            1/1     Running            3          11h
openshift-oauth-apiserver-f5c699955-kqv68            1/1     Running            4          11h
packageserver-568ddbbdf9-25n7z                       1/1     Running            2          11h
packageserver-568ddbbdf9-cgc55                       1/1     Running            4          11h
rbac-sync-operator-84d4c6f768-mzppk                  1/1     Running            0          10h
redhat-marketplace-catalog-6f9bfbdd6f-hx7hn          1/1     Running            0          89m
redhat-marketplace-catalog-rollout-27215316-mfdpp    0/1     Completed          0          89m
redhat-operators-catalog-d57ff6f8c-2hv9m             1/1     Running            0          89m
redhat-operators-catalog-rollout-27215316-5pl66      0/1     Completed          0          89m
relyt0925 commented 3 years ago

Key observation from post above: The key observation is the amount of machine-config-server pods spun up after one restart. I would expect only 3 pods to be spun up (one for each ignition server however if you trace the lifecyle of the pods there are actually 6 over the life span of a restart)

machine-config-server-1236201992                     0/1     Init:0/3           0          3s
machine-config-server-291020420                      0/1     Init:1/3           0          4s
machine-config-server-446334720                      1/1     Running            0          8s
machine-config-server-1463997948                     0/1     Init:0/3           0          2s
machine-config-server-1408474896                     0/1     Pending            0          0s
machine-config-server-1018368211                     0/1     Init:2/3           0          5s
relyt0925 commented 3 years ago

For the leak: I think the delete is best effort and if it were to fail the resource will be permanently orphaned. May need a separate time based cleaner that ensures anything older than 20 minutes or whatever threshold is also cleaned: https://github.com/openshift/hypershift/blob/main/ignition-server/controllers/machineconfigserver_ignitionprovider.go#L80-L98

enxebre commented 3 years ago

Key observation from post above: The key observation is the amount of machine-config-server pods spun up after one restart. I would expect only 3 pods to be spun up (one for each ignition server however if you trace the lifecyle of the pods there are actually 6 over the life span of a restart)

The expectation depends on the number of token secrets and the number of ignition server pods you have right? How may token Secret and ignition server pods are there in that clusters? Do you have the logs by chance?

relyt0925 commented 2 years ago

I don't see this being a problem after we have fixed the nodePool reconciliation in this pr for IBMCloud clusters: https://github.com/openshift/hypershift/commit/1df13b9cde220c50c80289f6f02cd693013926c6

relyt0925 commented 2 years ago

Things are looking better after the reconciliation fix I will close and reopen if I notice more problems