MenD32 commented 1 month ago

Summary

daemon containers should be able to retry in order to ensure the stability of the workflow

Use Cases

CICD unit testing pipelines - when testing features that rely on external services, I want the other services (that may be daemon containers) to be consistent
"Rare resource" dependent workflows - for workflows that depend on "rare resources" (such as GPUs), they might have preemptive scheduling (like spot instances), so I'd like my workflow to wait for the resource to be available again

Message from the maintainers:

Love this feature request? Give it a 👍. We prioritise the proposals with the most 👍.

MenD32 commented 1 month ago

I'd also like to contribute to solve this issue

MenD32 commented 1 month ago

after looking at the code, i think a better fix would be to amend retry policy because of the createWorkflowPod function, which is kinda huge.

the something like retryStrategy: always

which i might add, doesn't work currently on daemon containers, from my testing

agilgur5 commented 1 month ago

This also would have to handle the difference between the Controller intentionally killing it at the end vs it dying on its own