saurav-k
commented
3 years ago During uninstall this installation gets success, Because operator goes down and stop triggering pod deployment. +++++++++
❯ kubectl get pods -n olm NAME READY STATUS RESTARTS AGE operatorhubio-catalog-vfhx9 0/1 Running 0 15s catalog-operator-7f656b6d67-qvpvp 1/1 Terminating 0 53m packageserver-fb9fd764f-6nb2b 1/1 Terminating 0 53m packageserver-fb9fd764f-ksd5c 1/1 Terminating 0 53m ❯ kubectl get pods -n olm NAME READY STATUS RESTARTS AGE operatorhubio-catalog-vfhx9 1/1 Running 0 33s ❯ kubectl get pods -n olm NAME READY STATUS RESTARTS AGE operatorhubio-catalog-vfhx9 1/1 Running 0 37s
Bug Report
What did you do?
❯ kubectl get pods -n olm NAME READY STATUS RESTARTS AGE olm-operator-8d9bf86c9-pmp9t 1/1 Running 0 2m14s catalog-operator-7f656b6d67-qvpvp 1/1 Running 0 2m14s packageserver-fb9fd764f-ksd5c 1/1 Running 0 2m10s packageserver-fb9fd764f-6nb2b 1/1 Running 0 2m10s operatorhubio-catalog-w454x 0/1 Pending 0 0s
+++++++ Log from admission webhook +++++++++++++++++++++++
kubectl logs -f gilly-78c4785b56-ssvr9 -n gilly --tail=20
2021/02/11 15:05:54 [Mutate] Received POD create event. Name: , Namespace: olm 2021/02/11 15:05:54 [ProcessPod] Found registry => quay.io 2021/02/11 15:05:54 [ProcessPod] image registry for container registry-server is quay.io - updating 2021/02/11 15:05:54 [ReplaceImageRegistry] Replacing registry for image quay.io/operatorhubio/catalog:latest with docker.repo.eng.netapp.com 2021/02/11 15:05:54 [ReplaceImageRegistry] Image quay.io/operatorhubio/catalog:latest => docker.repo.eng.netapp.com/operatorhubio/catalog:latest 2021/02/11 15:05:54 [ProcessPod] updated registry for container registry-server to docker.repo.eng.netapp.com/operatorhubio/catalog:latest 2021/02/11 15:05:54 resp: {"kind":"AdmissionReview","apiVersion":"admission.k8s.io/v1beta1","request":{"uid":"13764b22-5161-4390-a630-ba9c3c7304da","kind":{"group":"","version":"v1","kind":"Pod"},"resource":{"group":"","version":"v1","resource":"pods"},"requestKind":{"group":"","version":"v1","kind":"Pod"},"requestResource":{"group":"","version":"v1","resource":"pods"},"namespace":"olm","operation":"CREATE","userInfo":{"username":"system:serviceaccount:olm:olm-operator-serviceaccount","uid":"60612c06-416e-407a-9f88-bbf1bed806c4","groups":["system:serviceaccounts","system:serviceaccounts:olm","system:authenticated"]},"object":{"kind":"Pod","apiVersion":"v1","metadata":{"generateName":"operatorhubio-catalog-","namespace":"olm","creationTimestamp":null,"labels":{"olm.catalogSource":"operatorhubio-catalog"},"ownerReferences":[{"apiVersion":"operators.coreos.com/v1alpha1","kind":"CatalogSource","name":"operatorhubio-catalog","uid":"7b9ebae5-9d4e-4982-bbfc-3bc933e4e119","controller":false,"blockOwnerDeletion":false}]},"spec":{"volumes":[{"name":"default-token-cf5rp","secret":{"secretName":"default-token-cf5rp"}}],"containers":[{"name":"registry-server","image":"quay.io/operatorhubio/catalog:latest","ports":[{"name":"grpc","containerPort":50051,"protocol":"TCP"}],"resources":{"requests":{"cpu":"10m","memory":"50Mi"}},"volumeMounts":[{"name":"default-token-cf5rp","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"livenessProbe":{"exec":{"command":["grpc_health_probe","-addr=:50051"]},"initialDelaySeconds":10,"timeoutSeconds":1,"periodSeconds":10,"successThreshold":1,"failureThreshold":3},"readinessProbe":{"exec":{"command":["grpc_health_probe","-addr=:50051"]},"initialDelaySeconds":5,"timeoutSeconds":5,"periodSeconds":10,"successThreshold":1,"failureThreshold":3},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","nodeSelector":{"kubernetes.io/os":"linux"},"serviceAccountName":"default","serviceAccount":"default","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true},"status":{}},"oldObject":null,"dryRun":false,"options":{"kind":"CreateOptions","apiVersion":"meta.k8s.io/v1"}},"response":{"uid":"13764b22-5161-4390-a630-ba9c3c7304da","allowed":true,"status":{"metadata":{},"status":"Success"},"patch":"W3sib3AiOiJyZXBsYWNlIiwicGF0aCI6Ii9zcGVjL2NvbnRhaW5lcnMvMC9pbWFnZSIsInZhbHVlIjoiZG9ja2VyLnJlcG8uZW5nLm5ldGFwcC5jb20vb3BlcmF0b3JodWJpby9jYXRhbG9nOmxhdGVzdCJ9XQ==","patchType":"JSONPatch","auditAnnotations":{"gilly":"review complete"}}}
What did you expect to see?
I want go code to not keeps on deploying wait for 10 seccound and then try redeployment of operatorhubio-catalog pod image.
What did you see instead? Under which circumstances?
catalog operator keeps on retrying and thus admission webhook keeps on getting request continuously to change the image tag.
Environment
❯ kubectl version --short Client Version: v1.20.2 Server Version: v1.17.0+k3s.1
Possible Solution
modify catalog operator to wait for 10 secounds before bombarding with deployment.