MenD32 commented 1 week ago

Summary

daemon containers should be able to retry in order to ensure the stability of the workflow

Use Cases

CICD unit testing pipelines - when testing features that rely on external services, I want the other services (that may be daemon containers) to be consistent
"Rare resource" dependent workflows - for workflows that depend on "rare resources" (such as GPUs), they might have preemptive scheduling (like spot instances), so I'd like my workflow to wait for the resource to be available again

Message from the maintainers:

Love this feature request? Give it a 👍. We prioritise the proposals with the most 👍.

MenD32 commented 1 week ago

I'd also like to contribute to solve this issue

MenD32 commented 1 week ago

after looking at the code, i think a better fix would be to amend retry policy because of the createWorkflowPod function, which is kinda huge.

the something like retryStrategy: always

which i might add, doesn't work currently on daemon containers, from my testing