Open menzbua opened 1 month ago
they pointed out to me that the permission is missed in the "manifests/cluster-install/argo-server-rbac/argo-server-clusterole.yaml" file, too.
I think you meant this file? The Controller ClusterRole? https://github.com/argoproj/argo-workflows/blob/5aac5a8f61f4e8273d04509dffe7d80123ff67f5/manifests/namespace-install/workflow-controller-rbac/workflow-controller-role.yaml#L47
I've opened an issue at the Helm-Chart repo (argoproj/argo-helm#2824)
It looks like Tim responded in your PR rather: https://github.com/argoproj/argo-helm/pull/2825#issuecomment-2225142489
I researched the behavior and found an error message in the workflow-controller pod:
Are you sure that was in your Controller, and not in the workflow's own Pod?
In your Helm issue, you pointed to the workflow role; that is the SA attached to Pods started by a workflow.
It seems to not be separated out per type of workflow, but the workflowtasksets/status
permission inherits from the agent RBAC, which you can find here: https://github.com/argoproj/argo-workflows/blob/5aac5a8f61f4e8273d04509dffe7d80123ff67f5/manifests/quick-start/base/agent-role.yaml#L22
And indeed the agent communicates its status to the Controller that way. I don't believe the Controller otherwise needs that permission.
workflowartifactgctasks/status
is similar for the ArtifactGC Pod: https://github.com/argoproj/argo-workflows/blob/5aac5a8f61f4e8273d04509dffe7d80123ff67f5/manifests/quick-start/base/artifactgc-role.yaml#L20
I reproduced the issue again. When i remove this permission from the ClusterRole argo-workflows-workflow-controller:
- apiGroups:
- argoproj.io
resources:
- workflowtasksets/status
- workflowartifactgctasks/status
verbs:
- patch
I see the errors in the workflow-controller pod:
time="2024-07-15T06:30:01.923Z" level=warning msg="error updating taskset" error="failed patching taskset: workflowtasksets.argoproj.io \"http-template-fjt8f\" is forbidden: User \"system:serviceaccount:argocd:argo-workflows-workflow-controller\" cannot patch resource \"workflowtasksets/status\" in API group \"argoproj.io\" in the namespace \"argocd\"" namespace=argocd workflow=http-template-fjt8f
time="2024-07-15T06:30:01.942Z" level=info msg="Workflow update successful" namespace=argocd phase=Running resourceVersion=260438096 workflow=http-template-fjt8f
time="2024-07-15T06:30:07.550Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=260437855 namespace=argocd workflow=http-template-75bnp
time="2024-07-15T06:30:07.552Z" level=info msg="Task-result reconciliation" namespace=argocd numObjs=0 workflow=http-template-75bnp
time="2024-07-15T06:30:07.552Z" level=info msg=updateAgentPodStatus namespace=argocd workflow=http-template-75bnp
time="2024-07-15T06:30:07.552Z" level=info msg=assessAgentPodStatus namespace=argocd podName=http-template-75bnp-1340600742-agent
time="2024-07-15T06:30:07.552Z" level=error msg="was unable to obtain node for http-template-75bnp-2166136261" namespace=argocd workflow=http-template-75bnp
time="2024-07-15T06:30:07.552Z" level=info msg="Workflow step group node http-template-75bnp-2750993691 not yet completed" namespace=argocd workflow=http-template-75bnp
time="2024-07-15T06:30:07.552Z" level=info msg="TaskSet Reconciliation" namespace=argocd workflow=http-template-75bnp
time="2024-07-15T06:30:07.552Z" level=info msg="Creating TaskSet" namespace=argocd workflow=http-template-75bnp
time="2024-07-15T06:30:07.579Z" level=info msg=reconcileAgentPod namespace=argocd workflow=http-template-75bnp
time="2024-07-15T06:30:07.579Z" level=info msg=updateAgentPodStatus namespace=argocd workflow=http-template-75bnp
time="2024-07-15T06:30:07.579Z" level=info msg=assessAgentPodStatus namespace=argocd podName=http-template-75bnp-1340600742-agent
time="2024-07-15T06:30:07.579Z" level=info msg="Workflow to be dehydrated" Workflow Size=22261
time="2024-07-15T06:30:07.582Z" level=warning msg="error updating taskset" error="failed patching taskset: workflowtasksets.argoproj.io \"http-template-75bnp\" is forbidden: User \"system:serviceaccount:argocd:argo-workflows-workflow-controller\" cannot patch resource \"workflowtasksets/status\" in API group \"argoproj.io\" in the namespace \"argocd\"" namespace=argocd workflow=http-template-75bnp
time="2024-07-15T06:30:07.600Z" level=info msg="Workflow update successful" namespace=argocd phase=Running resourceVersion=260438180 workflow=http-template-75bnp
time="2024-07-15T06:30:10.059Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=260437898 namespace=argocd workflow=http-template-nfrn9
time="2024-07-15T06:30:10.060Z" level=info msg="Task-result reconciliation" namespace=argocd numObjs=0 workflow=http-template-nfrn9
time="2024-07-15T06:30:10.060Z" level=info msg=updateAgentPodStatus namespace=argocd workflow=http-template-nfrn9
time="2024-07-15T06:30:10.060Z" level=info msg=assessAgentPodStatus namespace=argocd podName=http-template-nfrn9-1340600742-agent
time="2024-07-15T06:30:10.060Z" level=error msg="was unable to obtain node for http-template-nfrn9-2166136261" namespace=argocd workflow=http-template-nfrn9
time="2024-07-15T06:30:10.061Z" level=info msg="Workflow step group node http-template-nfrn9-2088566058 not yet completed" namespace=argocd workflow=http-template-nfrn9
time="2024-07-15T06:30:10.061Z" level=info msg="TaskSet Reconciliation" namespace=argocd workflow=http-template-nfrn9
time="2024-07-15T06:30:10.061Z" level=info msg="Creating TaskSet" namespace=argocd workflow=http-template-nfrn9
time="2024-07-15T06:30:10.076Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=260437900 namespace=argocd workflow=http-template-h5mcv
time="2024-07-15T06:30:10.077Z" level=info msg="Task-result reconciliation" namespace=argocd numObjs=0 workflow=http-template-h5mcv
time="2024-07-15T06:30:10.077Z" level=info msg=updateAgentPodStatus namespace=argocd workflow=http-template-h5mcv
time="2024-07-15T06:30:10.077Z" level=info msg=assessAgentPodStatus namespace=argocd podName=http-template-h5mcv-1340600742-agent
time="2024-07-15T06:30:10.077Z" level=error msg="was unable to obtain node for http-template-h5mcv-2166136261" namespace=argocd workflow=http-template-h5mcv
time="2024-07-15T06:30:10.078Z" level=info msg="Workflow step group node http-template-h5mcv-2436720626 not yet completed" namespace=argocd workflow=http-template-h5mcv
time="2024-07-15T06:30:10.078Z" level=info msg="TaskSet Reconciliation" namespace=argocd workflow=http-template-h5mcv
time="2024-07-15T06:30:10.078Z" level=info msg="Creating TaskSet" namespace=argocd workflow=http-template-h5mcv
time="2024-07-15T06:30:10.095Z" level=info msg=reconcileAgentPod namespace=argocd workflow=http-template-nfrn9
time="2024-07-15T06:30:10.095Z" level=info msg=updateAgentPodStatus namespace=argocd workflow=http-template-nfrn9
time="2024-07-15T06:30:10.095Z" level=info msg=assessAgentPodStatus namespace=argocd podName=http-template-nfrn9-1340600742-agent
time="2024-07-15T06:30:10.095Z" level=info msg="Workflow to be dehydrated" Workflow Size=22168
time="2024-07-15T06:30:10.101Z" level=warning msg="error updating taskset" error="failed patching taskset: workflowtasksets.argoproj.io \"http-template-nfrn9\" is forbidden: User \"system:serviceaccount:argocd:argo-workflows-workflow-controller\" cannot patch resource \"workflowtasksets/status\" in API group \"argoproj.io\" in the namespace \"argocd\"" namespace=argocd workflow=http-template-nfrn9
time="2024-07-15T06:30:10.109Z" level=info msg=reconcileAgentPod namespace=argocd workflow=http-template-h5mcv
time="2024-07-15T06:30:10.109Z" level=info msg=updateAgentPodStatus namespace=argocd workflow=http-template-h5mcv
time="2024-07-15T06:30:10.109Z" level=info msg=assessAgentPodStatus namespace=argocd podName=http-template-h5mcv-1340600742-agent
time="2024-07-15T06:30:10.109Z" level=info msg="Workflow to be dehydrated" Workflow Size=22204
time="2024-07-15T06:30:10.113Z" level=warning msg="error updating taskset" error="failed patching taskset: workflowtasksets.argoproj.io \"http-template-h5mcv\" is forbidden: User \"system:serviceaccount:argocd:argo-workflows-workflow-controller\" cannot patch resource \"workflowtasksets/status\" in API group \"argoproj.io\" in the namespace \"argocd\"" namespace=argocd workflow=http-template-h5mcv
time="2024-07-15T06:30:10.120Z" level=info msg="Workflow update successful" namespace=argocd phase=Running resourceVersion=260438215 workflow=http-template-nfrn9
time="2024-07-15T06:30:10.131Z" level=info msg="Workflow update successful" namespace=argocd phase=Running resourceVersion=260438216 workflow=http-template-h5mcv
time="2024-07-15T06:31:42.349Z" level=info msg="Alloc=9776 TotalAlloc=4844300 Sys=32357 NumGC=2193 Goroutines=171"
And the tasks with HTTP-Template lasts very long:
When I reattach the policy to the ClusterRole no errors in the workflow-controller pod:
time="2024-07-15T06:38:11.910Z" level=info msg="Processing workflow" Phase= ResourceVersion=260445416 namespace=argocd workflow=http-template-tw7cw
time="2024-07-15T06:38:11.921Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2024-07-15T06:38:11.921Z" level=info msg="resolved artifact repository" artifactRepositoryRef=default-artifact-repository
time="2024-07-15T06:38:11.921Z" level=info msg="Task-result reconciliation" namespace=argocd numObjs=0 workflow=http-template-tw7cw
time="2024-07-15T06:38:11.921Z" level=info msg="Updated phase -> Running" namespace=argocd workflow=http-template-tw7cw
time="2024-07-15T06:38:11.921Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argocd workflow=http-template-tw7cw
time="2024-07-15T06:38:11.921Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=argocd workflow=http-template-tw7cw
time="2024-07-15T06:38:11.921Z" level=info msg="Steps node http-template-tw7cw initialized Running" namespace=argocd workflow=http-template-tw7cw
time="2024-07-15T06:38:11.921Z" level=info msg="StepGroup node http-template-tw7cw-3315032375 initialized Running" namespace=argocd workflow=http-template-tw7cw
time="2024-07-15T06:38:11.922Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argocd workflow=http-template-tw7cw
time="2024-07-15T06:38:11.922Z" level=info msg="HTTP node http-template-tw7cw-337888994 initialized Pending" namespace=argocd workflow=http-template-tw7cw
time="2024-07-15T06:38:11.922Z" level=info msg="Workflow step group node http-template-tw7cw-3315032375 not yet completed" namespace=argocd workflow=http-template-tw7cw
time="2024-07-15T06:38:11.922Z" level=info msg="TaskSet Reconciliation" namespace=argocd workflow=http-template-tw7cw
time="2024-07-15T06:38:11.922Z" level=info msg="Creating TaskSet" namespace=argocd workflow=http-template-tw7cw
time="2024-07-15T06:38:11.941Z" level=info msg=reconcileAgentPod namespace=argocd workflow=http-template-tw7cw
W0715 06:38:12.035764 1 warnings.go:70] autopilot-default-resources-mutator:Autopilot updated Pod argocd/http-template-tw7cw-1340600742-agent: adjusted 'cpu' resource to meet requirements for containers [main] (see http://g.co/gke/autopilot-defaults).
time="2024-07-15T06:38:12.036Z" level=info msg="Created Agent pod" namespace=argocd podName=http-template-tw7cw-1340600742-agent workflow=http-template-tw7cw
time="2024-07-15T06:38:12.036Z" level=info msg=updateAgentPodStatus namespace=argocd workflow=http-template-tw7cw
time="2024-07-15T06:38:12.036Z" level=info msg=assessAgentPodStatus namespace=argocd podName=http-template-tw7cw-1340600742-agent
time="2024-07-15T06:38:12.036Z" level=info msg="Workflow to be dehydrated" Workflow Size=1828
time="2024-07-15T06:38:12.079Z" level=info msg="Workflow update successful" namespace=argocd phase=Running resourceVersion=260445423 workflow=http-template-tw7cw
time="2024-07-15T06:38:22.037Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=260445423 namespace=argocd workflow=http-template-tw7cw
time="2024-07-15T06:38:22.038Z" level=info msg="Task-result reconciliation" namespace=argocd numObjs=0 workflow=http-template-tw7cw
time="2024-07-15T06:38:22.038Z" level=info msg=updateAgentPodStatus namespace=argocd workflow=http-template-tw7cw
time="2024-07-15T06:38:22.038Z" level=info msg=assessAgentPodStatus namespace=argocd podName=http-template-tw7cw-1340600742-agent
time="2024-07-15T06:38:22.039Z" level=info msg="Workflow step group node http-template-tw7cw-3315032375 not yet completed" namespace=argocd workflow=http-template-tw7cw
time="2024-07-15T06:38:22.039Z" level=info msg="TaskSet Reconciliation" namespace=argocd workflow=http-template-tw7cw
time="2024-07-15T06:38:22.039Z" level=info msg="Creating TaskSet" namespace=argocd workflow=http-template-tw7cw
time="2024-07-15T06:38:22.066Z" level=info msg=reconcileAgentPod namespace=argocd workflow=http-template-tw7cw
time="2024-07-15T06:38:22.066Z" level=info msg=updateAgentPodStatus namespace=argocd workflow=http-template-tw7cw
time="2024-07-15T06:38:22.066Z" level=info msg=assessAgentPodStatus namespace=argocd podName=http-template-tw7cw-1340600742-agent
time="2024-07-15T06:38:32.068Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=260445423 namespace=argocd workflow=http-template-tw7cw
time="2024-07-15T06:38:32.069Z" level=info msg="Task-result reconciliation" namespace=argocd numObjs=0 workflow=http-template-tw7cw
time="2024-07-15T06:38:32.069Z" level=info msg=updateAgentPodStatus namespace=argocd workflow=http-template-tw7cw
time="2024-07-15T06:38:32.069Z" level=info msg=assessAgentPodStatus namespace=argocd podName=http-template-tw7cw-1340600742-agent
time="2024-07-15T06:38:32.070Z" level=info msg="Workflow step group node http-template-tw7cw-3315032375 not yet completed" namespace=argocd workflow=http-template-tw7cw
time="2024-07-15T06:38:32.070Z" level=info msg="TaskSet Reconciliation" namespace=argocd workflow=http-template-tw7cw
time="2024-07-15T06:38:32.070Z" level=info msg="Creating TaskSet" namespace=argocd workflow=http-template-tw7cw
time="2024-07-15T06:38:32.104Z" level=info msg=reconcileAgentPod namespace=argocd workflow=http-template-tw7cw
time="2024-07-15T06:38:32.104Z" level=info msg=updateAgentPodStatus namespace=argocd workflow=http-template-tw7cw
time="2024-07-15T06:38:32.104Z" level=info msg=assessAgentPodStatus namespace=argocd podName=http-template-tw7cw-1340600742-agent
time="2024-07-15T06:38:32.104Z" level=info msg="Workflow to be dehydrated" Workflow Size=22306
time="2024-07-15T06:38:32.148Z" level=info msg="Workflow update successful" namespace=argocd phase=Running resourceVersion=260445722 workflow=http-template-tw7cw
time="2024-07-15T06:38:42.120Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=260445722 namespace=argocd workflow=http-template-tw7cw
time="2024-07-15T06:38:42.122Z" level=info msg="Task-result reconciliation" namespace=argocd numObjs=0 workflow=http-template-tw7cw
time="2024-07-15T06:38:42.123Z" level=info msg=updateAgentPodStatus namespace=argocd workflow=http-template-tw7cw
time="2024-07-15T06:38:42.123Z" level=info msg=assessAgentPodStatus namespace=argocd podName=http-template-tw7cw-1340600742-agent
time="2024-07-15T06:38:42.124Z" level=info msg="Step group node http-template-tw7cw-3315032375 successful" namespace=argocd workflow=http-template-tw7cw
time="2024-07-15T06:38:42.124Z" level=info msg="node http-template-tw7cw-3315032375 phase Running -> Succeeded" namespace=argocd workflow=http-template-tw7cw
time="2024-07-15T06:38:42.124Z" level=info msg="node http-template-tw7cw-3315032375 finished: 2024-07-15 06:38:42.124665946 +0000 UTC" namespace=argocd workflow=http-template-tw7cw
time="2024-07-15T06:38:42.124Z" level=info msg="Outbound nodes of http-template-tw7cw-337888994 is [http-template-tw7cw-337888994]" namespace=argocd workflow=http-template-tw7cw
time="2024-07-15T06:38:42.124Z" level=info msg="Outbound nodes of http-template-tw7cw is [http-template-tw7cw-337888994]" namespace=argocd workflow=http-template-tw7cw
time="2024-07-15T06:38:42.124Z" level=info msg="node http-template-tw7cw phase Running -> Succeeded" namespace=argocd workflow=http-template-tw7cw
time="2024-07-15T06:38:42.124Z" level=info msg="node http-template-tw7cw finished: 2024-07-15 06:38:42.124839581 +0000 UTC" namespace=argocd workflow=http-template-tw7cw
time="2024-07-15T06:38:42.124Z" level=info msg="TaskSet Reconciliation" namespace=argocd workflow=http-template-tw7cw
time="2024-07-15T06:38:42.124Z" level=info msg=reconcileAgentPod namespace=argocd workflow=http-template-tw7cw
time="2024-07-15T06:38:42.124Z" level=info msg="Updated phase Running -> Succeeded" namespace=argocd workflow=http-template-tw7cw
time="2024-07-15T06:38:42.125Z" level=info msg="Marking workflow completed" namespace=argocd workflow=http-template-tw7cw
time="2024-07-15T06:38:42.125Z" level=info msg="Workflow to be dehydrated" Workflow Size=22390
time="2024-07-15T06:38:42.131Z" level=info msg="cleaning up pod" action=deletePod key=argocd/http-template-tw7cw-1340600742-agent/deletePod
time="2024-07-15T06:38:42.173Z" level=info msg="Workflow update successful" namespace=argocd phase=Succeeded resourceVersion=260445867 workflow=http-template-tw7cw
time="2024-07-15T06:38:42.176Z" level=info msg="Queueing Succeeded workflow argocd/http-template-tw7cw for delete in 5m0s due to TTL"
And the jobs last for round about 30 seconds:
The workflow-controller uses the ServiceAccount whre the ClusterRole is bound to:
The workflows pod uses another ServiceAccount that have also the permissions to patch the workflowtasksets/status
In the worker pod I did not see any errors.
I've installed Argo-Workflows via Helm-Chart with this values:
server:
extraArgs:
- --auth-mode=sso
- --auth-mode=client
sso:
enabled: true
issuer: <private-value>
clientId:
name: dex-credentials
key: gitlabClientId
clientSecret:
name: dex-credentials
key: gitlabClientSecret
redirectUrl: <private-value>
rbac:
enabled: false
controller:
workflowNamespaces:
- argocd
workflowDefaults:
spec:
serviceAccountName: argo-workflow
After installing the Chart I create the Role, RoleBinding and ServiceAccount for the workflow-worker via a manifest:
# These configuration is needed to give workflows workers to cluster objects.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: argo-workflow
namespace: argocd
rules:
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- watch
- patch
- apiGroups:
- ""
resources:
- pods/log
verbs:
- get
- watch
- apiGroups:
- ""
resources:
- pods/exec
verbs:
- create
- apiGroups:
- ""
resources:
- secrets
verbs:
- get
- apiGroups:
- argoproj.io
resources:
- workflowtaskresults
verbs:
- create
- patch
- apiGroups:
- argoproj.io
resources:
- workflowtasksets
- workflowartifactgctasks
verbs:
- list
- watch
- apiGroups:
- argoproj.io
resources:
- workflowtasksets/status
- workflowartifactgctasks/status
verbs:
- patch
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: argo-workflow
namespace: argocd
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: argo-workflows-workflow
namespace: argocd
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: argo-workflow
subjects:
- kind: ServiceAccount
name: argo-workflow
namespace: argocd
---
# As of Kubernetes v1.24, secrets are no longer automatically created for service accounts.
# See: https://argo-workflows.readthedocs.io/en/latest/service-account-secrets/
apiVersion: v1
kind: Secret
metadata:
name: argo-workflow.service-account-token
namespace: argocd
annotations:
kubernetes.io/service-account.name: argo-workflow
type: kubernetes.io/service-account-token
I hope this helps to clarify the issue.
Regards, Manuel
Pre-requisites
:latest
image tag (i.e.quay.io/argoproj/workflow-controller:latest
) and can confirm the issue still exists on:latest
. If not, I have explained why, in detail, in my description below.What happened? What did you expect to happen?
Describe the bug
Hi,
I have a problem with Argo workflows HTTP Template. When I run a workflow with using this template, the workflow lasts for 30 seconds or many minutes until it is finished. I researched the behavior and found an error message in the workflow-controller pod:
I've installed Argo workflows using the latest Helm-Chart. After I investigated the issue, i found missing permission in the ClusterRole for the workflow-controller.
After adding this rule to the ClusterRole on my cluster, all HTTP Template jobs are running perfectly fine. I've opened an issue at the Helm-Chart repo (https://github.com/argoproj/argo-helm/issues/2824) and they pointed out to me that the permission is missed in the "manifests/cluster-install/argo-server-rbac/argo-server-clusterole.yaml" file, too. Is it possible that these permissions are missing? Or did I some misconfiguration in my cluster? I'm a little confused that this issue should only happen on my site.
Thank you so much for helping.
Version(s)
v3.5.8
Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.
Logs from the workflow controller
Logs from in your workflow's wait container