search

argoproj / argo-workflows

Workflow Engine for Kubernetes
https://argo-workflows.readthedocs.io/
Apache License 2.0
15.03k stars 3.2k forks source link

`expected < 2 pods, got 2 - this is a bug` #9987

Open thangamani-arun opened 1 year ago

thangamani-arun commented 1 year ago

Pre-requisites

What happened/what you expected to happen?

^[39mSTEP^[0m                             TEMPLATE             PODNAME                                                 DURATION  MESSAGE
 ^[31m✖^[0m gitlab-fullstack-wf-6prdv     gitlab-fullstack-wf                                                                    step group deemed errored due to child gitlab-fullstack-wf-6prdv[6].deploy-gitlab error: expected < 2 pods, got 2 - this is a bug
 ├───^[32m✔^[0m create-namespace          namespace            gitlab-fullstack-wf-6prdv-namespace-1622150166          5s

 ├───^[32m✔^[0m create-argocd-project     argocd-project       gitlab-fullstack-wf-6prdv-argocd-project-972177293      5s

 ├───^[32m✔^[0m create-pgo-configmap      pgo-configmap        gitlab-fullstack-wf-6prdv-pgo-configmap-1718522638      5s

 ├───^[32m✔^[0m deploy-postgres-operator  postgres-operator    gitlab-fullstack-wf-6prdv-postgres-operator-4015549576  6s

 ├───^[32m✔^[0m deploy-postgres-cluster   postgres-cluster     gitlab-fullstack-wf-6prdv-postgres-cluster-3648711258   4s

 ├───^[32m✔^[0m deploy-redis-cluster      redis-cluster        gitlab-fullstack-wf-6prdv-redis-cluster-2621838390      6s

 └───^[31m⚠^[0m deploy-gitlab             gitlab               gitlab-fullstack-wf-6prdv-gitlab-975994934              10s       expected < 2 pods, got 2 - this is a bug

Version

v3.4.1

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

Logs from the workflow controller

kubectl logs -n argo-workflows deploy/argo-workflows-workflow-controller | grep gitlab-fullstack-wf-6prdv
Found 2 pods, using pod/argo-workflows-workflow-controller-6745bb6f5c-62jjm
time="2022-11-07T09:21:53.657Z" level=info msg="Processing workflow" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:21:53.686Z" level=info msg="Updated phase  -> Running" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:21:53.693Z" level=info msg="Steps node gitlab-fullstack-wf-6prdv initialized Running" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:21:53.693Z" level=info msg="StepGroup node gitlab-fullstack-wf-6prdv-1219061841 initialized Running" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:21:53.694Z" level=info msg="Pod node gitlab-fullstack-wf-6prdv-121512663 initialized Pending" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:21:53.888Z" level=info msg="Created pod: gitlab-fullstack-wf-6prdv[0].create-namespace (gitlab-fullstack-wf-6prdv-namespace-121512663)" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:21:53.888Z" level=info msg="Workflow step group node gitlab-fullstack-wf-6prdv-1219061841 not yet completed" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:21:53.888Z" level=info msg="TaskSet Reconciliation" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:21:53.888Z" level=info msg=reconcileAgentPod namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:21:53.928Z" level=warning msg="Error updating workflow: Operation cannot be fulfilled on workflows.argoproj.io \"gitlab-fullstack-wf-6prdv\": the object has been modified; please apply your changes to the latest version and try again Conflict" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:21:53.928Z" level=info msg="Re-applying updates on latest version and retrying update" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:21:53.994Z" level=info msg="Update retry attempt 1 successful" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:21:53.995Z" level=info msg="Workflow update successful" namespace=argo-workflows phase=Running resourceVersion=513725435 workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:03.662Z" level=info msg="Processing workflow" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:03.662Z" level=info msg="Task-result reconciliation" namespace=argo-workflows numObjs=0 workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:03.663Z" level=info msg="node changed" namespace=argo-workflows new.message= new.phase=Succeeded new.progress=0/1 nodeID=gitlab-fullstack-wf-6prdv-121512663 old.message= old.phase=Pending old.progress=0/1 workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:03.663Z" level=info msg="node unchanged" namespace=argo-workflows nodeID=gitlab-fullstack-wf-6prdv-121512663 workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:03.668Z" level=info msg="Step group node gitlab-fullstack-wf-6prdv-1219061841 successful" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prd
time="2022-11-07T09:22:03.668Z" level=info msg="node gitlab-fullstack-wf-6prdv-1219061841 phase Running -> Succeeded" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:03.668Z" level=info msg="node gitlab-fullstack-wf-6prdv-1219061841 finished: 2022-11-07 09:22:03.668403453 +0000 UTC" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:03.668Z" level=info msg="StepGroup node gitlab-fullstack-wf-6prdv-2225866076 initialized Running" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:03.668Z" level=info msg="SG Outbound nodes of gitlab-fullstack-wf-6prdv-121512663 are [gitlab-fullstack-wf-6prdv-121512663]" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:03.669Z" level=info msg="Pod node gitlab-fullstack-wf-6prdv-3910150959 initialized Pending" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:03.806Z" level=info msg="Created pod: gitlab-fullstack-wf-6prdv[1].create-argocd-project (gitlab-fullstack-wf-6prdv-argocd-project-3910150959)" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:03.806Z" level=info msg="Workflow step group node gitlab-fullstack-wf-6prdv-2225866076 not yet completed" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:03.806Z" level=info msg="TaskSet Reconciliation" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:03.806Z" level=info msg=reconcileAgentPod namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:03.853Z" level=info msg="Workflow update successful" namespace=argo-workflows phase=Running resourceVersion=513725548 workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:03.861Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo-workflows/gitlab-fullstack-wf-6prdv-121512663/labelPodCompleted
time="2022-11-07T09:22:03.861Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo-workflows/gitlab-fullstack-wf-6prdv-namespace-121512663/labelPodCompleted
time="2022-11-07T09:22:13.811Z" level=info msg="Processing workflow" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:13.812Z" level=info msg="Task-result reconciliation" namespace=argo-workflows numObjs=0 workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:13.812Z" level=info msg="node changed" namespace=argo-workflows new.message= new.phase=Succeeded new.progress=0/1 nodeID=gitlab-fullstack-wf-6prdv-3910150959 old.message= old.phase=Pending old.progress=0/1 workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:13.817Z" level=info msg="SG Outbound nodes of gitlab-fullstack-wf-6prdv-121512663 are [gitlab-fullstack-wf-6prdv-121512663]" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:13.818Z" level=info msg="Step group node gitlab-fullstack-wf-6prdv-2225866076 successful" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prd
time="2022-11-07T09:22:13.818Z" level=info msg="node gitlab-fullstack-wf-6prdv-2225866076 phase Running -> Succeeded" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:13.818Z" level=info msg="node gitlab-fullstack-wf-6prdv-2225866076 finished: 2022-11-07 09:22:13.818403236 +0000 UTC" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:13.818Z" level=info msg="StepGroup node gitlab-fullstack-wf-6prdv-2225424791 initialized Running" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:13.818Z" level=info msg="SG Outbound nodes of gitlab-fullstack-wf-6prdv-3910150959 are [gitlab-fullstack-wf-6prdv-3910150959]" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:13.819Z" level=info msg="Pod node gitlab-fullstack-wf-6prdv-746054629 initialized Pending" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:13.964Z" level=info msg="Created pod: gitlab-fullstack-wf-6prdv[2].create-pgo-configmap (gitlab-fullstack-wf-6prdv-pgo-configmap-746054629)" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:13.964Z" level=info msg="Workflow step group node gitlab-fullstack-wf-6prdv-2225424791 not yet completed" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:13.964Z" level=info msg="TaskSet Reconciliation" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:13.964Z" level=info msg=reconcileAgentPod namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:14.009Z" level=info msg="Workflow update successful" namespace=argo-workflows phase=Running resourceVersion=513725658 workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:14.017Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo-workflows/gitlab-fullstack-wf-6prdv-argocd-project-3910150959/labelPodCompleted
time="2022-11-07T09:22:23.970Z" level=info msg="Processing workflow" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:23.971Z" level=info msg="Task-result reconciliation" namespace=argo-workflows numObjs=0 workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:23.971Z" level=info msg="node changed" namespace=argo-workflows new.message= new.phase=Succeeded new.progress=0/1 nodeID=gitlab-fullstack-wf-6prdv-746054629 old.message= old.phase=Pending old.progress=0/1 workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:23.971Z" level=info msg="node changed" namespace=argo-workflows new.message= new.phase=Succeeded new.progress=0/1 nodeID=gitlab-fullstack-wf-6prdv-746054629 old.message= old.phase=Succeeded old.progress=0/1 workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:23.975Z" level=info msg="SG Outbound nodes of gitlab-fullstack-wf-6prdv-121512663 are [gitlab-fullstack-wf-6prdv-121512663]" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:23.975Z" level=info msg="SG Outbound nodes of gitlab-fullstack-wf-6prdv-3910150959 are [gitlab-fullstack-wf-6prdv-3910150959]" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:23.976Z" level=info msg="Step group node gitlab-fullstack-wf-6prdv-2225424791 successful" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prd
time="2022-11-07T09:22:23.976Z" level=info msg="node gitlab-fullstack-wf-6prdv-2225424791 phase Running -> Succeeded" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:23.976Z" level=info msg="node gitlab-fullstack-wf-6prdv-2225424791 finished: 2022-11-07 09:22:23.976622352 +0000 UTC" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:23.976Z" level=info msg="StepGroup node gitlab-fullstack-wf-6prdv-1218914746 initialized Running" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:23.976Z" level=info msg="SG Outbound nodes of gitlab-fullstack-wf-6prdv-746054629 are [gitlab-fullstack-wf-6prdv-746054629]" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:23.978Z" level=info msg="Pod node gitlab-fullstack-wf-6prdv-3971340020 initialized Pending" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:24.245Z" level=info msg="Created pod: gitlab-fullstack-wf-6prdv[3].deploy-postgres-operator (gitlab-fullstack-wf-6prdv-postgres-operator-3971340020)" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:24.245Z" level=info msg="Workflow step group node gitlab-fullstack-wf-6prdv-1218914746 not yet completed" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:24.245Z" level=info msg="TaskSet Reconciliation" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:24.245Z" level=info msg=reconcileAgentPod namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:24.279Z" level=warning msg="Error updating workflow: Operation cannot be fulfilled on workflows.argoproj.io \"gitlab-fullstack-wf-6prdv\": the object has been modified; please apply your changes to the latest version and try again Conflict" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:24.279Z" level=info msg="Re-applying updates on latest version and retrying update" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:24.369Z" level=info msg="Update retry attempt 1 successful" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:24.369Z" level=info msg="Workflow update successful" namespace=argo-workflows phase=Running resourceVersion=513725795 workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:24.377Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo-workflows/gitlab-fullstack-wf-6prdv-pgo-configmap-746054629/labelPodCompleted
time="2022-11-07T09:22:24.377Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo-workflows/gitlab-fullstack-wf-6prdv-746054629/labelPodCompleted
time="2022-11-07T09:22:34.165Z" level=info msg="Processing workflow" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:34.166Z" level=info msg="Task-result reconciliation" namespace=argo-workflows numObjs=0 workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:34.166Z" level=info msg="node changed" namespace=argo-workflows new.message= new.phase=Succeeded new.progress=0/1 nodeID=gitlab-fullstack-wf-6prdv-3971340020 old.message= old.phase=Pending old.progress=0/1 workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:34.166Z" level=info msg="node changed" namespace=argo-workflows new.message= new.phase=Succeeded new.progress=0/1 nodeID=gitlab-fullstack-wf-6prdv-3971340020 old.message= old.phase=Succeeded old.progress=0/1 workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:34.170Z" level=info msg="SG Outbound nodes of gitlab-fullstack-wf-6prdv-121512663 are [gitlab-fullstack-wf-6prdv-121512663]" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:34.170Z" level=info msg="SG Outbound nodes of gitlab-fullstack-wf-6prdv-3910150959 are [gitlab-fullstack-wf-6prdv-3910150959]" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:34.170Z" level=info msg="SG Outbound nodes of gitlab-fullstack-wf-6prdv-746054629 are [gitlab-fullstack-wf-6prdv-746054629]" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:34.172Z" level=info msg="Step group node gitlab-fullstack-wf-6prdv-1218914746 successful" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prd
time="2022-11-07T09:22:34.172Z" level=info msg="node gitlab-fullstack-wf-6prdv-1218914746 phase Running -> Succeeded" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:34.172Z" level=info msg="node gitlab-fullstack-wf-6prdv-1218914746 finished: 2022-11-07 09:22:34.17233046 +0000 UTC" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:34.172Z" level=info msg="StepGroup node gitlab-fullstack-wf-6prdv-2293521005 initialized Running" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:34.173Z" level=info msg="SG Outbound nodes of gitlab-fullstack-wf-6prdv-3971340020 are [gitlab-fullstack-wf-6prdv-3971340020]" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:34.175Z" level=info msg="Pod node gitlab-fullstack-wf-6prdv-2944012881 initialized Pending" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:34.316Z" level=info msg="Created pod: gitlab-fullstack-wf-6prdv[4].deploy-postgres-cluster (gitlab-fullstack-wf-6prdv-postgres-cluster-2944012881)" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:34.316Z" level=info msg="Workflow step group node gitlab-fullstack-wf-6prdv-2293521005 not yet completed" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:34.316Z" level=info msg="TaskSet Reconciliation" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:34.316Z" level=info msg=reconcileAgentPod namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:34.366Z" level=warning msg="Error updating workflow: Operation cannot be fulfilled on workflows.argoproj.io \"gitlab-fullstack-wf-6prdv\": the object has been modified; please apply your changes to the latest version and try again Conflict" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:34.366Z" level=info msg="Re-applying updates on latest version and retrying update" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:34.454Z" level=info msg="Update retry attempt 1 successful" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:34.454Z" level=info msg="Workflow update successful" namespace=argo-workflows phase=Running resourceVersion=513725923 workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:34.462Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo-workflows/gitlab-fullstack-wf-6prdv-postgres-operator-3971340020/labelPodCompleted
time="2022-11-07T09:22:34.462Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo-workflows/gitlab-fullstack-wf-6prdv-3971340020/labelPodCompleted
time="2022-11-07T09:22:44.304Z" level=info msg="Processing workflow" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:44.306Z" level=info msg="Task-result reconciliation" namespace=argo-workflows numObjs=0 workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:44.307Z" level=info msg="node changed" namespace=argo-workflows new.message= new.phase=Succeeded new.progress=0/1 nodeID=gitlab-fullstack-wf-6prdv-2944012881 old.message= old.phase=Pending old.progress=0/1 workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:44.307Z" level=info msg="node changed" namespace=argo-workflows new.message= new.phase=Succeeded new.progress=0/1 nodeID=gitlab-fullstack-wf-6prdv-2944012881 old.message= old.phase=Succeeded old.progress=0/1 workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:44.312Z" level=info msg="SG Outbound nodes of gitlab-fullstack-wf-6prdv-121512663 are [gitlab-fullstack-wf-6prdv-121512663]" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:44.312Z" level=info msg="SG Outbound nodes of gitlab-fullstack-wf-6prdv-3910150959 are [gitlab-fullstack-wf-6prdv-3910150959]" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:44.312Z" level=info msg="SG Outbound nodes of gitlab-fullstack-wf-6prdv-746054629 are [gitlab-fullstack-wf-6prdv-746054629]" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:44.312Z" level=info msg="SG Outbound nodes of gitlab-fullstack-wf-6prdv-3971340020 are [gitlab-fullstack-wf-6prdv-3971340020]" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:44.313Z" level=info msg="Step group node gitlab-fullstack-wf-6prdv-2293521005 successful" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prd
time="2022-11-07T09:22:44.313Z" level=info msg="node gitlab-fullstack-wf-6prdv-2293521005 phase Running -> Succeeded" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:44.313Z" level=info msg="node gitlab-fullstack-wf-6prdv-2293521005 finished: 2022-11-07 09:22:44.313374064 +0000 UTC" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:44.313Z" level=info msg="StepGroup node gitlab-fullstack-wf-6prdv-2226557624 initialized Running" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:44.313Z" level=info msg="SG Outbound nodes of gitlab-fullstack-wf-6prdv-2944012881 are [gitlab-fullstack-wf-6prdv-2944012881]" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:44.314Z" level=info msg="Pod node gitlab-fullstack-wf-6prdv-777451944 initialized Pending" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:44.468Z" level=info msg="Created pod: gitlab-fullstack-wf-6prdv[5].deploy-redis-cluster (gitlab-fullstack-wf-6prdv-redis-cluster-777451944)" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:44.468Z" level=info msg="Workflow step group node gitlab-fullstack-wf-6prdv-2226557624 not yet completed" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:44.468Z" level=info msg="TaskSet Reconciliation" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:44.469Z" level=info msg=reconcileAgentPod namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:44.518Z" level=warning msg="Error updating workflow: Operation cannot be fulfilled on workflows.argoproj.io \"gitlab-fullstack-wf-6prdv\": the object has been modified; please apply your changes to the latest version and try again Conflict" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:44.518Z" level=info msg="Re-applying updates on latest version and retrying update" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:44.617Z" level=info msg="Update retry attempt 1 successful" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:44.617Z" level=info msg="Workflow update successful" namespace=argo-workflows phase=Running resourceVersion=513726064 workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:44.626Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo-workflows/gitlab-fullstack-wf-6prdv-2944012881/labelPodCompleted
time="2022-11-07T09:22:44.626Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo-workflows/gitlab-fullstack-wf-6prdv-postgres-cluster-2944012881/labelPodCompleted
time="2022-11-07T09:22:54.459Z" level=info msg="Processing workflow" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:54.460Z" level=info msg="Task-result reconciliation" namespace=argo-workflows numObjs=0 workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:54.460Z" level=info msg="node changed" namespace=argo-workflows new.message= new.phase=Succeeded new.progress=0/1 nodeID=gitlab-fullstack-wf-6prdv-777451944 old.message= old.phase=Pending old.progress=0/1 workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:54.460Z" level=info msg="node unchanged" namespace=argo-workflows nodeID=gitlab-fullstack-wf-6prdv-777451944 workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:54.464Z" level=info msg="SG Outbound nodes of gitlab-fullstack-wf-6prdv-121512663 are [gitlab-fullstack-wf-6prdv-121512663]" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:54.464Z" level=info msg="SG Outbound nodes of gitlab-fullstack-wf-6prdv-3910150959 are [gitlab-fullstack-wf-6prdv-3910150959]" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:54.464Z" level=info msg="SG Outbound nodes of gitlab-fullstack-wf-6prdv-746054629 are [gitlab-fullstack-wf-6prdv-746054629]" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:54.464Z" level=info msg="SG Outbound nodes of gitlab-fullstack-wf-6prdv-3971340020 are [gitlab-fullstack-wf-6prdv-3971340020]" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:54.464Z" level=info msg="SG Outbound nodes of gitlab-fullstack-wf-6prdv-2944012881 are [gitlab-fullstack-wf-6prdv-2944012881]" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:54.466Z" level=info msg="Step group node gitlab-fullstack-wf-6prdv-2226557624 successful" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prd
time="2022-11-07T09:22:54.466Z" level=info msg="node gitlab-fullstack-wf-6prdv-2226557624 phase Running -> Succeeded" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:54.466Z" level=info msg="node gitlab-fullstack-wf-6prdv-2226557624 finished: 2022-11-07 09:22:54.466143638 +0000 UTC" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:54.466Z" level=info msg="StepGroup node gitlab-fullstack-wf-6prdv-2226013171 initialized Running" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:54.466Z" level=info msg="SG Outbound nodes of gitlab-fullstack-wf-6prdv-777451944 are [gitlab-fullstack-wf-6prdv-777451944]" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:54.468Z" level=info msg="Pod node gitlab-fullstack-wf-6prdv-658713792 initialized Pending" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:54.753Z" level=info msg="Created pod: gitlab-fullstack-wf-6prdv[6].deploy-gitlab (gitlab-fullstack-wf-6prdv-gitlab-658713792)" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:54.753Z" level=info msg="Workflow step group node gitlab-fullstack-wf-6prdv-2226013171 not yet completed" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:54.753Z" level=info msg="TaskSet Reconciliation" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:54.753Z" level=info msg=reconcileAgentPod namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:54.812Z" level=warning msg="Error updating workflow: Operation cannot be fulfilled on workflows.argoproj.io \"gitlab-fullstack-wf-6prdv\": the object has been modified; please apply your changes to the latest version and try again Conflict" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:54.812Z" level=info msg="Re-applying updates on latest version and retrying update" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:54.919Z" level=info msg="Update retry attempt 1 successful" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:54.919Z" level=info msg="Workflow update successful" namespace=argo-workflows phase=Running resourceVersion=513726233 workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:22:54.927Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo-workflows/gitlab-fullstack-wf-6prdv-777451944/labelPodCompleted
time="2022-11-07T09:22:54.927Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo-workflows/gitlab-fullstack-wf-6prdv-redis-cluster-777451944/labelPodCompleted
time="2022-11-07T09:23:04.729Z" level=info msg="Processing workflow" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:23:04.730Z" level=info msg="Task-result reconciliation" namespace=argo-workflows numObjs=0 workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:23:04.730Z" level=info msg="node changed" namespace=argo-workflows new.message= new.phase=Running new.progress=0/1 nodeID=gitlab-fullstack-wf-6prdv-658713792 old.message= old.phase=Pending old.progress=0/1 workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:23:04.730Z" level=info msg="node changed" namespace=argo-workflows new.message= new.phase=Running new.progress=0/1 nodeID=gitlab-fullstack-wf-6prdv-658713792 old.message= old.phase=Running old.progress=0/1 workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:23:04.733Z" level=info msg="SG Outbound nodes of gitlab-fullstack-wf-6prdv-121512663 are [gitlab-fullstack-wf-6prdv-121512663]" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:23:04.733Z" level=info msg="SG Outbound nodes of gitlab-fullstack-wf-6prdv-3910150959 are [gitlab-fullstack-wf-6prdv-3910150959]" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:23:04.734Z" level=info msg="SG Outbound nodes of gitlab-fullstack-wf-6prdv-746054629 are [gitlab-fullstack-wf-6prdv-746054629]" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:23:04.734Z" level=info msg="SG Outbound nodes of gitlab-fullstack-wf-6prdv-3971340020 are [gitlab-fullstack-wf-6prdv-3971340020]" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:23:04.734Z" level=info msg="SG Outbound nodes of gitlab-fullstack-wf-6prdv-2944012881 are [gitlab-fullstack-wf-6prdv-2944012881]" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:23:04.734Z" level=info msg="SG Outbound nodes of gitlab-fullstack-wf-6prdv-777451944 are [gitlab-fullstack-wf-6prdv-777451944]" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:23:04.736Z" level=error msg="Mark error node" error="expected < 2 pods, got 2 - this is a bug" namespace=argo-workflows nodeName="gitlab-fullstack-wf-6prdv[6].deploy-gitlab" workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:23:04.736Z" level=info msg="node gitlab-fullstack-wf-6prdv-658713792 phase Running -> Error" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prd
time="2022-11-07T09:23:04.736Z" level=info msg="node gitlab-fullstack-wf-6prdv-658713792 message: expected < 2 pods, got 2 - this is a bug" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:23:04.736Z" level=info msg="node gitlab-fullstack-wf-6prdv-658713792 finished: 2022-11-07 09:23:04.736214121 +0000 UTC" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:23:04.736Z" level=error msg="Mark error node" error="step group deemed errored due to child gitlab-fullstack-wf-6prdv[6].deploy-gitlab error: expected < 2 pods, got 2 - this is a bug" namespace=argo-workflows nodeName="gitlab-fullstack-wf-6prdv[6]" workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:23:04.736Z" level=info msg="node gitlab-fullstack-wf-6prdv-2226013171 phase Running -> Error" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:23:04.736Z" level=info msg="node gitlab-fullstack-wf-6prdv-2226013171 message: step group deemed errored due to child gitlab-fullstack-wf-6prdv[6].deploy-gitlab error: expected < 2 pods, got 2 - this is a bug" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:23:04.736Z" level=info msg="node gitlab-fullstack-wf-6prdv-2226013171 finished: 2022-11-07 09:23:04.736280434 +0000 UTC" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:23:04.736Z" level=info msg="step group gitlab-fullstack-wf-6prdv-2226013171 was unsuccessful: step group deemed errored due to child gitlab-fullstack-wf-6prdv[6].deploy-gitlab error: expected < 2 pods, got 2 - this is a bug" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:23:04.736Z" level=info msg="Outbound nodes of gitlab-fullstack-wf-6prdv-658713792 is [gitlab-fullstack-wf-6prdv-658713792]" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:23:04.736Z" level=info msg="Outbound nodes of gitlab-fullstack-wf-6prdv is [gitlab-fullstack-wf-6prdv-658713792]" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:23:04.736Z" level=info msg="node gitlab-fullstack-wf-6prdv phase Running -> Failed" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:23:04.736Z" level=info msg="node gitlab-fullstack-wf-6prdv message: step group deemed errored due to child gitlab-fullstack-wf-6prdv[6].deploy-gitlab error: expected < 2 pods, got 2 - this is a bug" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:23:04.736Z" level=info msg="node gitlab-fullstack-wf-6prdv finished: 2022-11-07 09:23:04.736378597 +0000 UTC" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:23:04.736Z" level=info msg="Checking daemoned children of gitlab-fullstack-wf-6prdv" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:23:04.736Z" level=info msg="TaskSet Reconciliation" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:23:04.736Z" level=info msg=reconcileAgentPod namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:23:04.736Z" level=info msg="Updated phase Running -> Failed" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:23:04.736Z" level=info msg="Updated message  -> step group deemed errored due to child gitlab-fullstack-wf-6prdv[6].deploy-gitlab error: expected < 2 pods, got 2 - this is a bug" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:23:04.736Z" level=info msg="Marking workflow completed" namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:23:04.736Z" level=info msg="Checking daemoned children of " namespace=argo-workflows workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:23:04.741Z" level=info msg="cleaning up pod" action=deletePod key=argo-workflows/gitlab-fullstack-wf-6prdv-1340600742-agent/deletePod
time="2022-11-07T09:23:04.792Z" level=info msg="Workflow update successful" namespace=argo-workflows phase=Failed resourceVersion=513726394 workflow=gitlab-fullstack-wf-6prdv
time="2022-11-07T09:23:04.810Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo-workflows/gitlab-fullstack-wf-6prdv-658713792/labelPodCompleted
time="2022-11-07T09:23:04.810Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo-workflows/gitlab-fullstack-wf-6prdv-gitlab-658713792/labelPodCompleted

Logs from in your workflow's wait container

kubectl logs -n argo-workflows -c wait -l workflows.argoproj.io/workflow=gitlab-fullstack-wf-6prdv,workflow.

part1

argoproj.io/phase!=Succeeded
time="2022-11-07T09:22:56.748Z" level=info msg="Starting Workflow Executor" version=v3.4.1
time="2022-11-07T09:22:56.752Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2022-11-07T09:22:56.752Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=argo-workflows podName=gitlab-fullstack-wf-6prdv-gitlab-658713792 template="{\"name\":\"gitlab\",\"inputs\":{\"parameters\":[{\"name\":\"app_name\",\"value\":\"gitlab\"},{\"name\":\"namespace\",\"value\":\"gitlab-dev\"},{\"name\":\"argocd_url\",\"value\":\"argo-cd-argocd-server.argocd.svc\"},{\"name\":\"argocd_user\",\"value\":\"user\"},{\"name\":\"argocd_password\",\"value\":\"password\"},{\"name\":\"argo_cd_k8s_target\",\"value\":\"https://kubernetes.default.svc\"},{\"name\":\"helm_repo_url\",\"value\":\"https://private-registry/repository/helm/\"},{\"name\":\"helm_repo_user\",\"value\":\"argocd_user\"},{\"name\":\"helm_repo_password\",\"value\":\"password\"},{\"name\":\"gitlab_chart_name\",\"value\":\"gitlab\"},{\"name\":\"gitlab_chart_version\",\"value\":\"5.8.2\"},{\"name\":\"registry\",\"value\":\"private-registry:8444\"},{\"name\":\"storage_class\",\"value\":\"ceph-xfs\"},{\"name\":\"gitlab_edition\",\"value\":\"ce\"},{\"name\":\"gitlab_version\",\"value\":\"v14.8.2\"},{\"name\":\"gitlab_kubectl\",\"value\":\"1.13.12\"},{\"name\":\"gitlab_certificates\",\"value\":\"20191127-r2\"},{\"name\":\"gitlab_minio\",\"value\":\"RELEASE.2017-12-28T01-21-00Z\"},{\"name\":\"gitlab_minioMc\",\"value\":\"RELEASE.2018-07-13T00-53-22Z\"},{\"name\":\"gitlab_runner\",\"value\":\"v14.8.0\"},{\"name\":\"gitlab_runner_securityContext\",\"value\":\"securityContext\"},{\"name\":\"gitlab_runner_helper\",\"value\":\"x86_64-v14.8.0\"},{\"name\":\"gitlab_shell\",\"value\":\"v13.23.2\"},{\"name\":\"gitlab_exporter\",\"value\":\"11.11.0\"},{\"name\":\"gitlab_kas\",\"value\":\"v14.8.1\"},{\"name\":\"gitlab_gitaly_size\",\"value\":\"60Gi\"},{\"name\":\"gitlab_minio_size\",\"value\":\"10Gi\"},{\"name\":\"ingress_class_name\",\"value\":\"nginx-default\"},{\"name\":\"gitlab_domain\",\"value\":\"dev.local.io\"},{\"name\":\"cert_issuer\",\"value\":\"vault-issuer\"}],\"artifacts\":[{\"name\":\"gitlab-apps\",\"path\":\"gitlab-helm-values.yml\",\"raw\":{\"data\":\"global:\\n  edition: ce\\n  gitlabVersion: v14.8.2\\n  image:\\n    pullPolicy: IfNotPresent\\n  hosts:\\n    domain: dev.local.io\\n    #hostSuffix: gitlab-dev\\n    https: true\\n  gitlab:\\n    name: dev.local.io\\n    https: true\\n  shell:\\n    port: 32022\\n  registry:\\n    name: registry.dev.local.io\\n    https: false\\n  smartcard:\\n    name: smartcard.dev.local.io\\n    https: false\\n  kas:\\n    name: kas.dev.local.io\\n    https: false\\n  minio:\\n    enabled: true\\n    name: minio.dev.local.io\\n    https: false\\n    tls:\\n      secretName: gitlab-minio-svc-ingress-tls\\n  pages:\\n    name: pages.dev.local.io\\n    https: false\\n  ingress:\\n    install: false\\n    configureCertmanager: false\\n    annotations:\\n      kubernetes.io/ingress.provider: \\\"nginx-default\\\"\\n      nginx.ingress.kubernetes.io/ssl-redirect: \\\"true\\\"\\n      nginx.ingress.kubernetes.io/redirect: \\\"true\\\"\\n      cert-manager.io/cluster-issuer: vault-issuer\\n      cert-manager.io/common-name: dev.local.io\\n    enabled: true\\n    class: nginx-default\\n    tls:\\n      secretName: gitlab-webservice-default-ingress-tls\\n  psql:\\n    host: automation-gitlab-postgres\\n    password:\\n      secret: gitlab.automation-gitlab-postgres.credentials.postgresql.acid.zalan.do\\n      key: password\\n    port: 5432\\n    database: gitlabhq_production\\n    username: gitlab\\n  redis:\\n    host: gitlab-redis-master\\n    password:\\n      secret: gitlab-redis\\n      key: redis-password\\n    port: 6379\\n  grafana:\\n    enabled: false\\n  busybox:\\n    image:\\n      repository: private-registry:8444/busybox\\n      tag: latest\\n  kubectl:\\n    image:\\n      repository: private-registry:8444/registry.gitlab.com/gitlab-org/build/cng/kubectl\\n      tag: 1.13.12\\n    securityContext:\\n      # in most base images, this is `nobody:nogroup`\\n      runAsUser: 65534\\n      fsGroup: 65534\\n  email:\\n    display_name: \\\"GitLab\\\"\\n    from: \\\"gitlab@project1.poc\\\"\\n    reply_to: \\\"noreply@project1.poc\\\"\\n  smtp:\\n    enabled: true\\n    address: \\\"mailhog-service.mailhog.svc\\\"\\n    tls: false\\n    authentication: \\\"\\\"\\n    port: 1025\\n  certificates:\\n    image:\\n      repository: private-registry:8444/registry.gitlab.com/gitlab-org/build/cng/alpine-certificates\\n      tag: 20191127-r2\\n  communityImages:\\n    # Default repositories used to pull Gitlab Community Edition images.\\n    # See the image.repository and workhorse.repository template helpers.\\n    migrations:\\n      repository: private-registry:8444/registry.gitlab.com/gitlab-org/build/cng/gitlab-toolbox-ce\\n    sidekiq:\\n      repository: private-registry:8444/registry.gitlab.com/gitlab-org/build/cng/gitlab-sidekiq-ce\\n    toolbox:\\n      repository: private-registry:8444/registry.gitlab.com/gitlab-org/build/cng/gitlab-toolbox-ce\\n    webservice:\\n      repository: private-registry:8444/registry.gitlab.com/gitlab-org/build/cng/gitlab-webservice-ce\\n    workhorse:\\n      repository: private-registry:8444/registry.gitlab.com/gitlab-org/build/cng/gitlab-workhorse-ce\\n\\ncertmanager:\\n  install: false\\n\\ncertmanager-issuer:\\n  email: cert@example.com\\n\\nnginx-ingress:\\n  enabled: false\\n\\nprometheus:\\n  install: false\\n  alertmanager:\\n    enabled: false\\n    persistentVolume:\\n      enabled: true\\n      size: 2Gi\\n  pushgateway:\\n    enabled: false\\n    persistentVolume:\\n      enabled: true\\n      size: 2Gi\\n  server:\\n    persistentVolume:\\n      enabled: false\\n      size: 8Gi\\n\\nredis:\\n  install: false\\n\\npostgresql:\\n  enabled: false\\n  install: false\\n\\nminio:\\n  persistence:\\n    enabled: true\\n    storageClass: \\\"ceph-xfs\\\"\\n    size: 10Gi\\n  image: private-registry:8444/minio/minio\\n  pullPolicy: IfNotPresent\\n  imageTag: RELEASE.2017-12-28T01-21-00Z\\n  replicas: 3\\n\\n  minioMc:\\n    image: private-registry:8444/minio/mc\\n    tag: RELEASE.2018-07-13T00-53-22Z\\n\\nkas:\\n  enabled: true\\n\\nregistry:\\n  enabled: false\\n\\npages:\\n  enabled: false\\n\\nmailroon:\\n  enabled: false\\n\\ngitlab-runner:\\n  install: true\\n  gitlabUrl: http://gitlab-webservice-default.gitlab-dev.svc:8181\\n  image: private-registry:8444/gitlab/gitlab-runner:v14.8.0\\n  imagePullPolicy: IfNotPresent\\n  concurrent: 30\\n  checkInterval: 30\\n  unregisterRunners: true\\n  replicas: 2\\n  metrics:\\n    enabled: true\\n\\n  # Update the security context values to the user ID in the ubuntu image\\n  securityContext:\\n    runAsUser: 999\\n    fsGroup: 999\\n\\n  init:\\n    image:\\n      repository: private-registry:8444/busybox\\n      tag: latest\\n\\n  runners:\\n    helpers:\\n      image: private-registry:8444/gitlab/gitlab-runner-helper:x86_64-v14.8.0\\n    unregisterRunners: true\\n    cache:\\n      cacheType: s3\\n      s3ServerAddress: http://gitlab-minio-svc.gitlab-dev.svc:9000\\n      s3CacheInsecure: true\\n    config: |\\n      [[runners]]\\n        clone_url = \\\"http://gitlab-webservice-default.gitlab-dev.svc:8181\\\"\\n        [runners.kubernetes]\\n          image = \\\"private-registry:8444/private/gitlab-runner:latest\\\"\\n          namespace = \\\"gitlab-dev\\\"\\n          privileged = false\\n          allow_privilege_escalation = false\\n\\ngitlab:\\n  toolbox:\\n    enabled: true\\n    image:\\n      pullPolicy: IfNotPresent\\n      repository: private-registry:8444/registry.gitlab.com/gitlab-org/build/cng/gitlab-toolbox-ce\\n      tag: v14.8.2\\n\\n  migrations:\\n    enabled: true\\n    psql:\\n      password:\\n        secret: gitlab.automation-gitlab-postgres.credentials.postgresql.acid.zalan.do\\n        key: password\\n    image:\\n      pullPolicy: IfNotPresent\\n      repository: private-registry:8444/registry.gitlab.com/gitlab-org/build/cng/gitlab-toolbox-ce\\n      tag: v14.8.2\\n    init:\\n      image:\\n        repository: private-registry:8444/busybox\\n        tag: latest\\n\\n  webservice:\\n    tls:\\n      secretName: gitlab-webservice-default-ingress-tls\\n    psql:\\n      password:\\n        secret: gitlab.automation-gitlab-postgres.credentials.postgresql.acid.zalan.do\\n        key: password\\n    resources:\\n      requests:\\n        memory: 1.5G\\n    image:\\n      pullPolicy: IfNotPresent\\n      repository: private-registry:8444/registry.gitlab.com/gitlab-org/build/cng/gitlab-webservice-ce\\n      tag: v14.8.2\\n    workhorse:\\n      image: private-registry:8444/registry.gitlab.com/gitlab-org/build/cng/gitlab-workhorse-ce\\n      tag: v14.8.2\\n\\n  sidekiq:\\n    image:\\n      pullPolicy: IfNotPresent\\n      repository: private-registry:8444/registry.gitlab.com/gitlab-org/build/cng/gitlab-sidekiq-ce\\n      tag: v14.8.2\\n    psql:\\n      password:\\n        secret: gitlab.automation-gitlab-postgres.credentials.postgresql.acid.zalan.do\\n        key: password\\n    resources:\\n      requests:\\n        memory: 650M\\n        cpu: 50m\\n\\n  gitaly:\\n    persistence:\\n      enabled: true\\n      storageClass: ceph-xfs\\n      accessMode: ReadWriteOnce\\n      size: 60Gi\\n    image:\\n      pullPolicy: IfNotPresent\\n      repository: private-registry:8444/registry.gitlab.com/gitlab-org/build/cng/gitaly\\n      tag: v14.8.2\\n    metrics:\\n      enabled: true\\n      metricsPort: 9236\\n\\n  gitlab-shell:\\n    image:\\n      pullPolicy: IfNotPresent\\n      repository: private-registry:8444/registry.gitlab.com/gitlab-org/build/cng/gitlab-shell\\n      tag: v13.23.2\\n\\n  gitlab-exporter:\\n    image:\\n      pullPolicy: IfNotPresent\\n      repository: private-registry:8444/registry.gitlab.com/gitlab-org/build/cng/gitlab-exporter\\n      tag: 11.11.0\\n      \\n  kas:\\n    image:\\n      pullPolicy: IfNotPresent\\n      repository: private-registry:8444/registry.gitlab.com/gitlab-org/build/cng/gitlab-kas\\n      tag: v14.8.1\\n    ingress:\\n      enabled: true\\n\"}}]},\"outputs\":{},\"metadata\":{},\"container\":{\"name\":\"\",\"image\":\"private-registry:8444/private/argo-executor:latest\",\"command\":[\"/bin/sh\",\"-c\"],\"args\":[\" helm repo add repo1 --username argocd_user --password password https://private-registry/repository/helm/ --insecure-skip-tls-verify ; helm repo update repo1; helm upgrade --install gitlab repo1/gitlab -f gitlab-helm-values.yml -n gitlab-dev --version 5.8.2 --insecure-skip-tls-verify; helm ls -a -n gitlab-dev; \"],\"resources\":{},\"volumeMounts\":[{\"name\":\"kubeconfig-volume\",\"mountPath\":\"/.kube/\"}]}}" version="&Version{Version:v3.4.1,BuildDate:2022-10-01T15:03:42Z,GitCommit:0546fef0b096d84c9e3362d2b241614e743ebe97,GitTag:v3.4.1,GitTreeState:clean,GoVersion:go1.18.6,Compiler:gc,Platform:linux/amd64,}"
time="2022-11-07T09:22:56.752Z" level=info msg="Starting deadline monitor"
time="2022-11-07T09:23:05.755Z" level=info msg="Main container completed" error="<nil>"
time="2022-11-07T09:23:05.755Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2022-11-07T09:23:05.755Z" level=info msg="No output parameters"
time="2022-11-07T09:23:05.755Z" level=info msg="No output artifacts"
time="2022-11-07T09:23:05.755Z" level=info msg="Alloc=7001 TotalAlloc=12739 Sys=25554 NumGC=4 Goroutines=6"
time="2022-11-07T09:23:05.756Z" level=info msg="Deadline monitor stopped"
time="2022-11-07T09:21:55.781Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2022-11-07T09:21:55.781Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=argo-workflows podName=gitlab-fullstack-wf-6prdv-namespace-121512663 template="{\"name\":\"namespace\",\"inputs\":{\"parameters\":[{\"name\":\"app_name\",\"value\":\"gitlab\"},{\"name\":\"namespace\",\"value\":\"gitlab-dev\"},{\"name\":\"registry\",\"value\":\"private-registry:8444\"}],\"artifacts\":[{\"name\":\"gitlab-namespace\",\"path\":\"gitlab-namespace.yml\",\"raw\":{\"data\":\"kind: Namespace\\napiVersion: v1\\nmetadata:\\n  name: \\\"gitlab-dev\\\"\\n  labels:\\n    name: \\\"gitlab-dev\\\"\\n\"}}]},\"outputs\":{},\"metadata\":{},\"container\":{\"name\":\"\",\"image\":\"private-registry:8444/private/argo-executor:latest\",\"command\":[\"/bin/sh\",\"-c\"],\"args\":[\"kubectl --kubeconfig /.kube/config.yaml apply -f gitlab-namespace.yml\"],\"resources\":{},\"volumeMounts\":[{\"name\":\"kubeconfig-volume\",\"mountPath\":\"/.kube/\"}]}}" version="&Version{Version:v3.4.1,BuildDate:2022-10-01T15:03:42Z,GitCommit:0546fef0b096d84c9e3362d2b241614e743ebe97,GitTag:v3.4.1,GitTreeState:clean,GoVersion:go1.18.6,Compiler:gc,Platform:linux/amd64,}"
time="2022-11-07T09:21:55.781Z" level=info msg="Starting deadline monitor"
time="2022-11-07T09:21:58.782Z" level=info msg="Main container completed" error="<nil>"
time="2022-11-07T09:21:58.782Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2022-11-07T09:21:58.782Z" level=info msg="No output parameters"
time="2022-11-07T09:21:58.782Z" level=info msg="No output artifacts"
time="2022-11-07T09:21:58.782Z" level=info msg="Deadline monitor stopped"
time="2022-11-07T09:21:58.782Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2022-11-07T09:21:58.783Z" level=info msg="Alloc=7075 TotalAlloc=12592 Sys=25554 NumGC=4 Goroutines=5"
time="2022-11-07T09:22:26.478Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2022-11-07T09:22:26.478Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=argo-workflows podName=gitlab-fullstack-wf-6prdv-postgres-operator-3971340020 template="{\"name\":\"postgres-operator\",\"inputs\":{\"parameters\":[{\"name\":\"app_name\",\"value\":\"gitlab\"},{\"name\":\"namespace\",\"value\":\"gitlab-dev\"},{\"name\":\"registry\",\"value\":\"private-registry:8444\"},{\"name\":\"argocd_url\",\"value\":\"argo-cd-argocd-server.argocd.svc\"},{\"name\":\"argocd_user\",\"value\":\"argocd_user\"},{\"name\":\"argocd_password\",\"value\":\"password\"},{\"name\":\"argo_cd_k8s_target\",\"value\":\"https://kubernetes.default.svc\"},{\"name\":\"helm_repo_url\",\"value\":\"https://private-registry/repository/helm/\"},{\"name\":\"helm_repo_user\",\"value\":\"argocd_user\"},{\"name\":\"helm_repo_password\",\"value\":\"password\"},{\"name\":\"pgo_chart_version\",\"value\":\"1.8.1\"},{\"name\":\"pgo_chart_name\",\"value\":\"postgres-operator\"},{\"name\":\"pgo_repo\",\"value\":\"registry.opensource.zalan.do/acid\"},{\"name\":\"pgo_tag\",\"value\":\"v1.8.1\"},{\"name\":\"pgo_spilo_tag\",\"value\":\"spilo-14:2.1-p5\"},{\"name\":\"pgo_pgbouncer_tag\",\"value\":\"master-22\"},{\"name\":\"pgo_configmap_name\",\"value\":\"postgres-pod-config\"},{\"name\":\"pgo_secret_name\",\"value\":\"postgres-secret\"},{\"name\":\"ceph_s3_endpoint\",\"value\":\"https://xx.xx.xx.xx\"},{\"name\":\"ceph_s3_access_key_id\",\"value\":\"AAAAAAA\"},{\"name\":\"ceph_s3_access_key\",\"value\":\"AAAAAAA\"},{\"name\":\"backup_num_to_retain\",\"value\":\"14\"},{\"name\":\"backup_schedule\",\"value\":\"*/5 * * * *\"},{\"name\":\"pgo_backup_s3_bucket\",\"value\":\"postgres-backups\"},{\"name\":\"pgo_logs_s3_bucket\",\"value\":\"postgres-logs\"},{\"name\":\"pgo_wal_s3_bucket\",\"value\":\"postgres-geo-standby\"},{\"name\":\"pgo_etcd_host\",\"value\":\"etcd-host:2379\"}],\"artifacts\":[{\"name\":\"postgres-operator\",\"path\":\"pgo-helm-values.yml\",\"raw\":{\"data\":\"image:\\n  registry: \\\"private-registry:8444\\\"\\n  repository: \\\"registry.opensource.zalan.do/acid/postgres-operator\\\"\\n  tag: \\\"v1.8.1\\\"\\n  pullPolicy: \\\"IfNotPresent\\\"\\n\\nconfigGeneral:\\n  docker_image: \\\"private-registry:8444/private/spilo-14:2.1-p5\\\"\\n  # etcd_host: \\\"etcd-host:2379\\\"\\n  enable_spilo_wal_path_compat: true\\n  repair_period: 5m\\n  resync_period: 3m\\n  workers: 8\\n\\nconfigDebug:\\n  debug_logging: false\\n  enable_database_access: true\\n\\nconfigUsers:\\n  replication_username: \\\"standby\\\"\\n  super_username: \\\"postgres\\\"\\n\\nconfigMajorVersionUpgrade:\\n  major_version_upgrade_mode: \\\"off\\\"\\n  target_major_version: \\\"14\\\"\\n\\nconfigKubernetes:\\n  cluster_domain: cluster.local\\n  cluster_labels:\\n  cluster_name_label: cluster-name\\n  enable_cross_namespace_secret: false\\n  pod_antiaffinity_topology_key: \\\"kubernetes.io/hostname\\\"\\n  enable_init_containers: true\\n  enable_pod_antiaffinity: false\\n  enable_pod_disruption_budget: true\\n  enable_sidecars: true\\n  pod_terminate_grace_period: 30s\\n  pod_environment_configmap: \\\"gitlab-dev/postgres-pod-config\\\"\\n  #pod_environment_secret: \\\"my-secret\\\"\\n  #spilo_runasuser: \\\"1000\\\"\\n  #spilo_runasgroup: \\\"1000\\\"\\n  #spilo_fsgroup: \\\"103\\\"\\n  #spilo_privileged: \\\"false\\\"\\n  storage_resize_mode: pvc\\n  watched_namespace: \\\"gitlab-dev\\\"\\n\\nconfigPostgresPodResources:\\n  # CPU limits for the postgres containers\\n  default_cpu_limit: \\\"1\\\"\\n  # CPU request value for the postgres containers\\n  default_cpu_request: 100m\\n  # memory limits for the postgres containers\\n  default_memory_limit: 500Mi\\n  # memory request value for the postgres containers\\n  default_memory_request: 100Mi\\n  # hard CPU minimum required to properly run a Postgres cluster\\n  min_cpu_limit: 250m\\n  # hard memory minimum required to properly run a Postgres cluster\\n  min_memory_limit: 250Mi\\n\\nconfigTimeouts:\\n  patroni_api_check_interval: 1s\\n  # timeout when waiting for successful response from Patroni API\\n  patroni_api_check_timeout: 5s\\n  # timeout when waiting for the Postgres pods to be deleted\\n  pod_deletion_wait_timeout: 10m\\n  # timeout when waiting for pod role and cluster labels\\n  pod_label_wait_timeout: 10m\\n  # interval between consecutive attempts waiting for postgresql CRD to be created\\n  ready_wait_interval: 3s\\n  # timeout for the complete postgres CRD creation\\n  ready_wait_timeout: 30s\\n  # interval to wait between consecutive attempts to check for some K8s resources\\n  resource_check_interval: 3s\\n  # timeout when waiting for the presence of a certain K8s resource (e.g. Sts, PDB)\\n  resource_check_timeout: 3m\\n\\n# WAL-E configuration for streaming replication\\nconfigAwsOrGcp:\\n  wal_s3_bucket: \\\"postgres-geo-standby\\\"\\n  log_s3_bucket: \\\"postgres-logs\\\"\\n\\nconfigLogicalBackup:\\n  logical_backup_docker_image: \\\"private-registry:8444/private/logical-backup:v1.8.1\\\"\\n  logical_backup_job_prefix: \\\"logical-backup-\\\"\\n  logical_backup_provider: \\\"s3\\\"\\n  # S3 Access Key ID\\n  logical_backup_s3_access_key_id: \\\"AAAAAAA\\\"\\n  # S3 bucket to store backup results\\n  logical_backup_s3_bucket: \\\"postgres-backups\\\"\\n  # S3 region of bucket\\n  logical_backup_s3_region: \\\"default\\\"\\n  # S3 endpoint url when not using AWS\\n  logical_backup_s3_endpoint: \\\"https://xx.xx.xx.xx\\\"\\n  # S3 Secret Access Key\\n  logical_backup_s3_secret_access_key: \\\"AAAAAAA\\\"\\n  # S3 server side encryption\\n  logical_backup_s3_sse: \\\"\\\"\\n  # backup schedule in the cron format\\n  logical_backup_schedule: \\\"*/5 * * * *\\\"\\n\\nconfigConnectionPooler:\\n  connection_pooler_image: \\\"private-registry:8444/registry.opensource.zalan.do/acid/pgbouncer:master-22\\\"\\n\\nrbac:\\n  create: true\\n\\ncrd:\\n  create: true\\n\\nserviceAccount:\\n  create: true\\n  name: \\\"operator-pod-gitlab-dev\\\"\\n\\npodServiceAccount:\\n  name: \\\"postgres-pod-gitlab-dev\\\"\\n\\nresources:\\n  limits:\\n    cpu: 900m\\n    memory: 2Gi\\n  requests:\\n    cpu: 500m\\n    memory: 1Gi\\n\\n# securityContext:\\n#   runAsUser: 1000\\n#   runAsNonRoot: true\\n#   readOnlyRootFilesystem: true\\n#   allowPrivilegeEscalation: false\\n\"}}]},\"outputs\":{},\"metadata\":{},\"container\":{\"name\":\"\",\"image\":\"private-registry:8444/private/argo-executor:latest\",\"command\":[\"/bin/sh\",\"-c\"],\"args\":[\" helm repo add repo1 --username user --password password https://private-registry/repository/helm/ --insecure-skip-tls-verify ; helm repo update repo1; helm upgrade --install postgres-operator repo1/postgres-operator -f pgo-helm-values.yml -n gitlab-dev --version 1.8.1 --insecure-skip-tls-verify; helm ls -a -n gitlab-dev; \"],\"resources\":{},\"volumeMounts\":[{\"name\":\"kubeconfig-volume\",\"mountPath\":\"/.kube/\"}]}}" version="&Version{Version:v3.4.1,BuildDate:2022-10-01T15:03:42Z,GitCommit:0546fef0b096d84c9e3362d2b241614e743ebe97,GitTag:v3.4.1,GitTreeState:clean,GoVersion:go1.18.6,Compiler:gc,Platform:linux/amd64,}"
time="2022-11-07T09:22:26.479Z" level=info msg="Starting deadline monitor"
time="2022-11-07T09:22:29.480Z" level=info msg="Main container completed" error="<nil>"
time="2022-11-07T09:22:29.481Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2022-11-07T09:22:29.481Z" level=info msg="No output parameters"
time="2022-11-07T09:22:29.481Z" level=info msg="No output artifacts"
time="2022-11-07T09:22:29.481Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2022-11-07T09:22:29.481Z" level=info msg="Deadline monitor stopped"
time="2022-11-07T09:22:29.481Z" level=info msg="Alloc=7880 TotalAlloc=12508 Sys=23506 NumGC=3 Goroutines=5"
time="2022-11-07T09:22:46.352Z" level=info msg="Starting Workflow Executor" version=v3.4.1
time="2022-11-07T09:22:46.357Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2022-11-07T09:22:46.357Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=argo-workflows podName=gitlab-fullstack-wf-6prdv-redis-cluster-777451944 template="{\"name\":\"redis-cluster\",\"inputs\":{\"parameters\":[{\"name\":\"app_name\",\"value\":\"gitlab\"},{\"name\":\"namespace\",\"value\":\"gitlab-dev\"},{\"name\":\"argocd_url\",\"value\":\"argo-cd-argocd-server.argocd.svc\"},{\"name\":\"argocd_user\",\"value\":\"user\"},{\"name\":\"argocd_password\",\"value\":\"password\"},{\"name\":\"argo_cd_k8s_target\",\"value\":\"https://kubernetes.default.svc\"},{\"name\":\"registry\",\"value\":\"private-registry:8444\"},{\"name\":\"helm_repo_url\",\"value\":\"https://private-registry/repository/helm/\"},{\"name\":\"helm_repo_user\",\"value\":\"user\"},{\"name\":\"helm_repo_password\",\"value\":\"password\"},{\"name\":\"redis_chart_version\",\"value\":\"16.12.2\"},{\"name\":\"redis_chart_name\",\"value\":\"redis\"},{\"name\":\"redis_image_tag\",\"value\":\"6.2.7-debian-11-r3\"},{\"name\":\"redis_exporter_tag\",\"value\":\"1.40.0-debian-11-r0\"},{\"name\":\"storage_class\",\"value\":\"ceph-xfs\"},{\"name\":\"redis_size\",\"value\":\"8Gi\"},{\"name\":\"redis_replica\",\"value\":\"2\"}],\"artifacts\":[{\"name\":\"redis-cluster\",\"path\":\"redis-helm-values.yml\",\"raw\":{\"data\":\"global:\\n  imageRegistry: private-registry:8444\\n\\nimage:\\n  registry: private-registry:8444\\n  repository: bitnami/redis\\n  tag: 6.2.7-debian-11-r3\\n  pullPolicy: IfNotPresent\\n\\npersistence:\\n  enabled: true\\n\\nmaster:\\n  persistence:\\n    enabled: true\\n    storageClass: ceph-xfs\\n    size: 8Gi\\n  livenessProbe:\\n    enabled: true\\n  readinessProbe:\\n    enabled: true\\n\\nreplica:\\n  replicaCount: 2\\n  persistence:\\n    enabled: true\\n    storageClass: ceph-xfs\\n    size: 8Gi\\n\\nmetrics:\\n  enabled: true\\n  image:\\n    registry: private-registry:8444\\n    repository: bitnami/redis-exporter\\n    tag: 1.40.0-debian-11-r0\\n    pullPolicy: IfNotPresent\\n\\nsentinel:\\n  enabled: false\\n  image:\\n    registry: private-registry:8444\\n    repository: bitnami/redis-sentinel\\n    tag: 6.2.7-debian-11-r3\\n    pullPolicy: IfNotPresent\\n  masterSet: redismaster\\n  quorum: 2\\n  containerPort: 26379\\n  downAfterMilliseconds: 60000\\n  failoverTimeout: 18000\\n  parallelSyncs: 1\\n\\nsysctl:\\n  enabled: false\\n  image:\\n    registry: private-registry:8444\\n    repository: bitnami/bitnami-shell\\n    tag: 11-debian-11-r3\\n    pullPolicy: Always\\n  command: []\\n  mountHostSys: false\\n\"}}]},\"outputs\":{},\"metadata\":{},\"container\":{\"name\":\"\",\"image\":\"private-registry:8444/private/argo-executor:latest\",\"command\":[\"/bin/sh\",\"-c\"],\"args\":[\" helm repo add repo1 --username user --password password https://private-registry/repository/helm/ --insecure-skip-tls-verify ; helm repo update repo1; helm upgrade --install redis repo1/redis -f redis-helm-values.yml -n gitlab-dev --version 16.12.2 --insecure-skip-tls-verify; helm ls -a -n gitlab-dev; \"],\"resources\":{},\"volumeMounts\":[{\"name\":\"kubeconfig-volume\",\"mountPath\":\"/.kube/\"}]}}" version="&Version{Version:v3.4.1,BuildDate:2022-10-01T15:03:42Z,GitCommit:0546fef0b096d84c9e3362d2b241614e743ebe97,GitTag:v3.4.1,GitTreeState:clean,GoVersion:go1.18.6,Compiler:gc,Platform:linux/amd64,}"
time="2022-11-07T09:22:46.358Z" level=info msg="Starting deadline monitor"
time="2022-11-07T09:22:50.359Z" level=info msg="Main container completed" error="<nil>"
time="2022-11-07T09:22:50.359Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2022-11-07T09:22:50.359Z" level=info msg="No output parameters"
time="2022-11-07T09:22:50.359Z" level=info msg="No output artifacts"
time="2022-11-07T09:22:50.359Z" level=info msg="Deadline monitor stopped"
time="2022-11-07T09:22:50.359Z" level=info msg="Alloc=6761 TotalAlloc=12720 Sys=24530 NumGC=4 Goroutines=7"
time="2022-11-07T09:22:36.539Z" level=info msg="Starting deadline monitor"
time="2022-11-07T09:22:36.541Z" level=info msg="secured root for pid 42 root: kubectl (\"/proc/42/root\")"
time="2022-11-07T09:22:36.541Z" level=info msg="mapped container name \"main\" to pid 42"
time="2022-11-07T09:22:37.557Z" level=info msg="Watch pods 200"
time="2022-11-07T09:22:38.065Z" level=info msg="Main container completed"
time="2022-11-07T09:22:38.065Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2022-11-07T09:22:38.065Z" level=info msg="No output parameters"
time="2022-11-07T09:22:38.065Z" level=info msg="No output artifacts"
time="2022-11-07T09:22:38.065Z" level=info msg="Killing sidecars []"
time="2022-11-07T09:22:38.065Z" level=info msg="Alloc=6505 TotalAlloc=11738 Sys=75089 NumGC=4 Goroutines=7"
time="2022-11-07T09:22:56.815Z" level=warning msg="failed to secure root file handle for 61" error="open /proc/61/root: permission denied"
time="2022-11-07T09:22:56.866Z" level=info msg="secured root for pid 61 root: sh (\"/proc/61/root\")"
time="2022-11-07T09:22:56.867Z" level=info msg="mapped container name \"main\" to pid 61"
time="2022-11-07T09:22:57.526Z" level=info msg="Watch pods 200"
time="2022-11-07T09:23:51.143Z" level=info msg="Main container completed"
time="2022-11-07T09:23:51.143Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2022-11-07T09:23:51.143Z" level=info msg="No output parameters"
time="2022-11-07T09:23:51.143Z" level=info msg="No output artifacts"
time="2022-11-07T09:23:51.143Z" level=info msg="Killing sidecars []"
time="2022-11-07T09:23:51.144Z" level=info msg="Alloc=6187 TotalAlloc=13708 Sys=75857 NumGC=5 Goroutines=7"
thangamani-arun commented 1 year ago

part2

time="2022-11-07T09:22:16.500Z" level=warning msg="failed to secure root file handle for 55" error="open /proc/55/root: permission denied"
time="2022-11-07T09:22:16.551Z" level=info msg="secured root for pid 55 root: kubectl (\"/proc/55/root\")"
time="2022-11-07T09:22:16.551Z" level=info msg="mapped container name \"main\" to pid 55"
time="2022-11-07T09:22:17.365Z" level=info msg="Watch pods 200"
time="2022-11-07T09:22:18.182Z" level=info msg="Main container completed"
time="2022-11-07T09:22:18.182Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2022-11-07T09:22:18.182Z" level=info msg="No output parameters"
time="2022-11-07T09:22:18.182Z" level=info msg="No output artifacts"
time="2022-11-07T09:22:18.182Z" level=info msg="Killing sidecars []"
time="2022-11-07T09:22:18.183Z" level=info msg="Alloc=6897 TotalAlloc=12064 Sys=75345 NumGC=4 Goroutines=7"
time="2022-11-07T09:22:46.837Z" level=info msg="Starting deadline monitor"
time="2022-11-07T09:22:46.837Z" level=info msg="secured root for pid 53 root: sh (\"/proc/53/root\")"
time="2022-11-07T09:22:46.837Z" level=info msg="mapped container name \"main\" to pid 53"
time="2022-11-07T09:22:47.934Z" level=info msg="Watch pods 200"
time="2022-11-07T09:22:50.221Z" level=info msg="Main container completed"
time="2022-11-07T09:22:50.221Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2022-11-07T09:22:50.221Z" level=info msg="No output parameters"
time="2022-11-07T09:22:50.221Z" level=info msg="No output artifacts"
time="2022-11-07T09:22:50.222Z" level=info msg="Killing sidecars []"
time="2022-11-07T09:22:50.222Z" level=info msg="Alloc=6525 TotalAlloc=11818 Sys=74577 NumGC=4 Goroutines=7"
time="2022-11-07T09:22:36.610Z" level=info msg="Starting Workflow Executor" version=v3.4.1
time="2022-11-07T09:22:36.614Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2022-11-07T09:22:36.614Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=argo-workflows podName=gitlab-fullstack-wf-6prdv-postgres-cluster-2944012881 template="{\"name\":\"postgres-cluster\",\"inputs\":{\"parameters\":[{\"name\":\"app_name\",\"value\":\"gitlab\"},{\"name\":\"namespace\",\"value\":\"gitlab-dev\"},{\"name\":\"storage_class\",\"value\":\"ceph-xfs\"},{\"name\":\"postgres_db_size\",\"value\":\"10Gi\"},{\"name\":\"postgres_version\",\"value\":\"14\"},{\"name\":\"registry\",\"value\":\"registry:8444\"},{\"name\":\"pgo_wal_archive_timeout\",\"value\":\"60\"},{\"name\":\"pgo_spilo_tag\",\"value\":\"spilo-14:2.1-p5\"},{\"name\":\"backup_schedule\",\"value\":\"*/5 * * * *\"}],\"artifacts\":[{\"name\":\"postgres-cluster\",\"path\":\"gitlab-postgres-cluster.yml\",\"raw\":{\"data\":\"kind: \\\"postgresql\\\"\\napiVersion: \\\"acid.zalan.do/v1\\\"\\nmetadata:\\n  name: \\\"automation-gitlab-postgres\\\"\\n  namespace: \\\"gitlab-dev\\\"\\n  labels:\\n    team: automation\\nspec:\\n  dockerImage: \\\"registry:8444/private/spilo-14:2.1-p5\\\"\\n  teamId: \\\"automation\\\"\\n  postgresql:\\n    version: \\\"14\\\"\\n    parameters:\\n      archive_timeout: \\\"60\\\"\\n      max_connections: \\\"250\\\"\\n  enableLogicalBackup: true\\n  logicalBackupSchedule: \\\"*/5 * * * *\\\"\\n  numberOfInstances: 3\\n  enableConnectionPooler: true\\n  volume:\\n    size: \\\"10Gi\\\"\\n    storageClass: \\\"ceph-xfs\\\"\\n  users:\\n    gitlab:\\n      - superuser\\n      - createdb\\n  databases:\\n    gitlabhq_production: gitlab\\n  allowedSourceRanges:\\n    # IP ranges to access your cluster go here\\n  patroni:\\n    initdb:\\n      encoding: \\\"UTF8\\\"\\n      locale: \\\"en_US.UTF-8\\\"\\n      data-checksums: \\\"true\\\"\\n    pg_hba:\\n      - local   all   all   trust\\n      - local   replication   standby   trust\\n      - host   replication   standby   all   md5\\n      - host   all   all   all   md5\\n  resources:\\n    requests:\\n      cpu: 400m\\n      memory: 1Gi\\n    limits:\\n      cpu: 800m\\n      memory: 1Gi\\n\"}}]},\"outputs\":{},\"metadata\":{},\"container\":{\"name\":\"\",\"image\":\"registry:8444/private/argo-executor:latest\",\"command\":[\"/bin/sh\",\"-c\"],\"args\":[\"kubectl --kubeconfig /.kube/config.yaml -n gitlab-dev apply -f gitlab-postgres-cluster.yml\"],\"resources\":{},\"volumeMounts\":[{\"name\":\"kubeconfig-volume\",\"mountPath\":\"/.kube/\"}]}}" version="&Version{Version:v3.4.1,BuildDate:2022-10-01T15:03:42Z,GitCommit:0546fef0b096d84c9e3362d2b241614e743ebe97,GitTag:v3.4.1,GitTreeState:clean,GoVersion:go1.18.6,Compiler:gc,Platform:linux/amd64,}"
time="2022-11-07T09:22:36.615Z" level=info msg="Starting deadline monitor"
time="2022-11-07T09:22:38.616Z" level=info msg="Main container completed" error="<nil>"
time="2022-11-07T09:22:38.616Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2022-11-07T09:22:38.616Z" level=info msg="No output parameters"
time="2022-11-07T09:22:38.616Z" level=info msg="No output artifacts"
time="2022-11-07T09:22:38.616Z" level=info msg="Deadline monitor stopped"
time="2022-11-07T09:22:38.617Z" level=info msg="Alloc=7167 TotalAlloc=12564 Sys=28882 NumGC=4 Goroutines=5"
time="2022-11-07T09:21:55.910Z" level=warning msg="failed to secure root file handle for 60" error="open /proc/60/root: permission denied"
time="2022-11-07T09:21:55.961Z" level=info msg="secured root for pid 60 root: kubectl (\"/proc/60/root\")"
time="2022-11-07T09:21:55.961Z" level=info msg="mapped container name \"main\" to pid 60"
time="2022-11-07T09:21:56.797Z" level=info msg="Watch pods 200"
time="2022-11-07T09:21:58.610Z" level=info msg="Main container completed"
time="2022-11-07T09:21:58.610Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2022-11-07T09:21:58.610Z" level=info msg="No output parameters"
time="2022-11-07T09:21:58.610Z" level=info msg="No output artifacts"
time="2022-11-07T09:21:58.610Z" level=info msg="Killing sidecars []"
time="2022-11-07T09:21:58.611Z" level=info msg="Alloc=6466 TotalAlloc=11670 Sys=75857 NumGC=4 Goroutines=7"
time="2022-11-07T09:22:26.273Z" level=warning msg="failed to secure root file handle for 60" error="open /proc/60/root: permission denied"
time="2022-11-07T09:22:26.325Z" level=info msg="secured root for pid 60 root: sh (\"/proc/60/root\")"
time="2022-11-07T09:22:26.325Z" level=info msg="mapped container name \"main\" to pid 60"
time="2022-11-07T09:22:27.137Z" level=info msg="Watch pods 200"
time="2022-11-07T09:22:28.958Z" level=info msg="Main container completed"
time="2022-11-07T09:22:28.959Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2022-11-07T09:22:28.959Z" level=info msg="No output parameters"
time="2022-11-07T09:22:28.959Z" level=info msg="No output artifacts"
time="2022-11-07T09:22:28.959Z" level=info msg="Killing sidecars []"
time="2022-11-07T09:22:28.959Z" level=info msg="Alloc=6956 TotalAlloc=12499 Sys=75857 NumGC=4 Goroutines=7"
time="2022-11-07T09:22:05.891Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2022-11-07T09:22:05.892Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=argo-workflows podName=gitlab-fullstack-wf-6prdv-argocd-project-3910150959 template="{\"name\":\"argocd-project\",\"inputs\":{\"parameters\":[{\"name\":\"app_name\",\"value\":\"gitlab\"},{\"name\":\"namespace\",\"value\":\"gitlab-dev\"},{\"name\":\"registry\",\"value\":\"registry:8444\"},{\"name\":\"argocd_url\",\"value\":\"argo-cd-argocd-server.argocd.svc\"},{\"name\":\"argocd_user\",\"value\":\"user\"},{\"name\":\"argocd_password\",\"value\":\"password\"},{\"name\":\"argo_cd_k8s_target\",\"value\":\"https://kubernetes.default.svc\"}],\"artifacts\":[{\"name\":\"argocd-project\",\"path\":\"argocd-project.yml\",\"raw\":{\"data\":\"apiVersion: argoproj.io/v1alpha1\\nkind: AppProject\\nmetadata:\\n  name: gitlab-dev\\n  namespace: argocd\\nspec:\\n  sourceRepos:\\n  - '*'\\n  destinations:\\n  - namespace: gitlab-dev\\n    server: https://kubernetes.default.svc\\n  clusterResourceWhitelist:\\n  - group: '*'\\n    kind: '*'\\n\"}}]},\"outputs\":{},\"metadata\":{},\"container\":{\"name\":\"\",\"image\":\"registry:8444/private/argo-executor:latest\",\"command\":[\"/bin/sh\",\"-c\"],\"args\":[\"argocd login argo-cd-argocd-server.argocd.svc --grpc-web --username user --password password --insecure --plaintext; argocd proj create --upsert -f argocd-project.yml\"],\"resources\":{},\"volumeMounts\":[{\"name\":\"kubeconfig-volume\",\"mountPath\":\"/.kube/\"}]}}" version="&Version{Version:v3.4.1,BuildDate:2022-10-01T15:03:42Z,GitCommit:0546fef0b096d84c9e3362d2b241614e743ebe97,GitTag:v3.4.1,GitTreeState:clean,GoVersion:go1.18.6,Compiler:gc,Platform:linux/amd64,}"
time="2022-11-07T09:22:05.892Z" level=info msg="Starting deadline monitor"
time="2022-11-07T09:22:08.894Z" level=info msg="Main container completed" error="<nil>"
time="2022-11-07T09:22:08.894Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2022-11-07T09:22:08.894Z" level=info msg="No output parameters"
time="2022-11-07T09:22:08.894Z" level=info msg="No output artifacts"
time="2022-11-07T09:22:08.895Z" level=info msg="Deadline monitor stopped"
time="2022-11-07T09:22:08.895Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2022-11-07T09:22:08.895Z" level=info msg="Alloc=6400 TotalAlloc=12571 Sys=25298 NumGC=4 Goroutines=5"
time="2022-11-07T09:22:15.976Z" level=info msg="Starting Workflow Executor" version=v3.4.1
time="2022-11-07T09:22:15.979Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2022-11-07T09:22:15.980Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=argo-workflows podName=gitlab-fullstack-wf-6prdv-pgo-configmap-746054629 template="{\"name\":\"pgo-configmap\",\"inputs\":{\"parameters\":[{\"name\":\"namespace\",\"value\":\"gitlab-dev\"},{\"name\":\"pgo_configmap_name\",\"value\":\"postgres-pod-config\"},{\"name\":\"registry\",\"value\":\"rehgistry:8444\"},{\"name\":\"ceph_s3_endpoint\",\"value\":\"https://xx.xx.xx.xx\"},{\"name\":\"ceph_s3_access_key_id\",\"value\":\"AAAAA\"},{\"name\":\"ceph_s3_access_key\",\"value\":\"AAAA\"},{\"name\":\"backup_num_to_retain\",\"value\":\"14\"},{\"name\":\"backup_schedule\",\"value\":\"*/5 * * * *\"}],\"artifacts\":[{\"name\":\"pgo-configmap\",\"path\":\"pgo-configmap.yml\",\"raw\":{\"data\":\"apiVersion: v1\\nkind: ConfigMap\\nmetadata:\\n  name: postgres-pod-config\\n  namespace: gitlab-dev\\ndata:\\n  AWS_ENDPOINT: https://xx.xx.xx.xx\\n  AWS_REGION: default\\n  AWS_S3_FORCE_PATH_STYLE: \\\"true\\\"\\n  AWS_ACCESS_KEY_ID: AAAAAAA\\n  AWS_SECRET_ACCESS_KEY: AAAA\\n  BACKUP_SCHEDULE: \\\"*/5 * * * *\\\"\\n  BACKUP_NUM_TO_RETAIN: \\\"14\\\"\\n  WALG_DISABLE_S3_SSE: \\\"true\\\"\\n  USE_WALG_BACKUP: \\\"true\\\"\\n  USE_WALG_RESTORE: \\\"false\\\"\\n\\n  STANDBY_AWS_ENDPOINT: https://xx.xx.xx.xx\\n  STANDBY_AWS_REGION: default\\n  STANDBY_AWS_ACCESS_KEY_ID: AAAA\\n  STANDBY_AWS_SECRET_ACCESS_KEY: AAA\\n  STANDBY_AWS_S3_FORCE_PATH_STYLE: \\\"true\\\"\\n  STANDBY_WALE_ENV_DIR: /run/etc/wal-e.d/env-standby\\n  STANDBY_USE_WALG_RESTORE: \\\"true\\\"\\n  STANDBY_WALG_DISABLE_S3_SSE: \\\"true\\\"\\n\"}}]},\"outputs\":{},\"metadata\":{},\"container\":{\"name\":\"\",\"image\":\"private-registry:8444/private/argo-executor:latest\",\"command\":[\"/bin/sh\",\"-c\"],\"args\":[\"kubectl --kubeconfig /.kube/config.yaml apply -f pgo-configmap.yml\"],\"resources\":{},\"volumeMounts\":[{\"name\":\"kubeconfig-volume\",\"mountPath\":\"/.kube/\"}]}}" version="&Version{Version:v3.4.1,BuildDate:2022-10-01T15:03:42Z,GitCommit:0546fef0b096d84c9e3362d2b241614e743ebe97,GitTag:v3.4.1,GitTreeState:clean,GoVersion:go1.18.6,Compiler:gc,Platform:linux/amd64,}"
time="2022-11-07T09:22:15.980Z" level=info msg="Starting deadline monitor"
time="2022-11-07T09:22:18.980Z" level=info msg="Main container completed" error="<nil>"
time="2022-11-07T09:22:18.980Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2022-11-07T09:22:18.980Z" level=info msg="No output parameters"
time="2022-11-07T09:22:18.980Z" level=info msg="No output artifacts"
time="2022-11-07T09:22:18.981Z" level=info msg="Alloc=7427 TotalAlloc=12500 Sys=24018 NumGC=3 Goroutines=7"
thangamani-arun commented 1 year ago

the same workflow was failed before above run,

STEP                             TEMPLATE             PODNAME                                                 DURATION  MESSAGE
 ✖ gitlab-fullstack-wf-2k6lb     gitlab-fullstack-wf                                                                    child 'gitlab-fullstack-wf-2k6lb-917790860' failed
 ├───✔ create-namespace          namespace            gitlab-fullstack-wf-2k6lb-namespace-1622150166          4s
 ├───✔ create-argocd-project     argocd-project       gitlab-fullstack-wf-2k6lb-argocd-project-972177293      7s
 ├───✔ create-pgo-configmap      pgo-configmap        gitlab-fullstack-wf-2k6lb-pgo-configmap-1718522638      4s
 ├───✔ deploy-postgres-operator  postgres-operator    gitlab-fullstack-wf-2k6lb-postgres-operator-4015549576  6s
 └───✖ deploy-postgres-cluster   postgres-cluster     gitlab-fullstack-wf-2k6lb-postgres-cluster-1514961193   5s        Error (exit code 1)

after 2nd run it errored with expected < 2 pods, got 2 - this is a bug. I have encountered this kind of once in a while.

thangamani-arun commented 1 year ago

I have heard from https://github.com/argoproj/argo-workflows/issues/9839#issuecomment-1281999154 comment that upgrading to v3.4.1 will solve, but I encountered this is in v3.4.1.

so wondering what is causing the failure ?

thangamani-arun commented 1 year ago

@sarabala1979 @alexec: your help would be greatly appreciated.

terrytangyuan commented 1 year ago

Can you try v3.4.3?

alexec commented 1 year ago

9238 tells why this happens.

It occurs here:

https://github.com/argoproj/argo-workflows/blob/67bcdb5e6da76a5f3dfb0fe71a16cf086e7ea26a/workflow/controller/workflowpod.go#L464

Essentially, there are two pods saying they’re running the work for the same node.

This func could be simplified to return objectCount > 0.

Would someone like to submit a PR?

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If this is a mentoring request, please provide an update here. Thank you for your contributions.

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If this is a mentoring request, please provide an update here. Thank you for your contributions.

kulinskyvs commented 1 year ago

I'm observing this issue with 3.4.7

ntdt commented 1 year ago

I have the same problem and the culprit is a rogue workflow-controller, e.g.

$ kubectl get pod -A|grep workflow
argo-workflows             argo-workflows-server-77645f86cc-fhkp4                            1/1     Running     0             9d
argo-workflows             argo-workflows-server-77645f86cc-hb266                            1/1     Running     0             9d
argo-workflows             argo-workflows-server-77645f86cc-x5h4r                            1/1     Running     0             9d
argo-workflows             argo-workflows-workflow-controller-7674d6687-4hsfj                1/1     Running     0             9d
argo-workflows             argo-workflows-workflow-controller-7674d6687-jxbbl                1/1     Running     0             9d
argo-workflows             argo-workflows-workflow-controller-7674d6687-kf2wc                1/1     Running     0             9d
argo                       workflow-controller-845d5d67f4-tqktq                              1/1     Running     0             14d

The rogue one is the controller inside the namespace argo, remove the deployment/workflow-controller fixes the problem.

YMMV but it's worth to verify if there is any workflow controller outside of your install

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If this is a mentoring request, please provide an update here. Thank you for your contributions.

gregsheremeta commented 2 weeks ago

it's worth to verify if there is any workflow controller outside of your install

Had a colleague hit this today and this was indeed the cause -- two controllers running.

I opened #13760 to call that out as a likely culprit.