When creating an agent pod, transient error is not retried

[fyp711]

Pre-requisites

• [X] I have double-checked my configuration
• [X] I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
• [X] I have searched existing issues and could not find a match for this bug
• [X] I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

What happened ？

When I use the HTTP template, I find it pending all the time. Then I saw that the workflow status was error, and encountered a transient error. error is : failed to create Agent pod. Reason: Operation cannot be fulfilled on resourcequotas "xxxx": the object has been modified; please apply your changes to the latest version and try again

What did you expect to happen?

I hope to encounter transient errors that can be retried.

Version(s)

v3.4.17

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

None

Logs from the workflow controller

None

Logs from in your workflow's wait container

None

[fyp711]

Is anyone paying attention to this issue?

[shuangkun]

Is it possible to have a stable and reproducible workflow?

[fyp711]

Is it possible to have a stable and reproducible workflow?

ResourceQuota conflicts are not easy to reproduce. I simulated another resourcequota shortage situation by modifying the requests value of the agent pod for reproduction and testing.

1. Change this value to an impossible value. https://github.com/argoproj/argo-workflows/blob/0dfecd6e3a18c7bb884000e1e98d8305440d8d49/workflow/controller/agent.go#L180
2. Create and run an HTTP template，then the agent pod will create failed.
3. The workflow failed.

if you have others can reproduce the transient error, it's always ok

[fyp711]

@shuangkun cc

[shuangkun]

okay