Open jesseanttila-cai opened 2 months ago
It seems unreliable to determine whether the task result has been successfully reported through the AnnotationKeyReportOutputsCompleted
.
After reproducing it, there are indeed some problems and need to think about how to fix it.
Yeah we are just concluding an RCA after we had a cohort of workflows "stuck" in Running state as the upgrade of the workflow-controller
was rolled out from v3.4.16
to v3.5.6
. We had the same conclusion that likely https://github.com/argoproj/argo-workflows/pull/11947 was the root cause, but https://github.com/argoproj/argo-workflows/pull/12402 seems related too! In either case looks like it's isolated to a transient issue on our end due to incompatibility between argoexec:v3.4.16
sidecars inside the active pods of workflows running at the time of the upgrade and the new workflow-controller:v3.5.6
control plane! Thanks for the original report and the maintainers for all the effort here!
Is the primary case for this one in which the Controller itself issues a SIGKILL to the container because the container isn't responding fast enough to a SIGTERM?
If so, it seems like we need to indicate on the Controller side that if we are doing a SIGKILL not to wait for the WorkflowTaskResult for that task, right?
Is the primary case for this one in which the Controller itself issues a SIGKILL to the container because the container isn't responding fast enough to a SIGTERM?
The root cause for the pod interruption in my case was primarily related to node deprovisioning, as the environment where this issue appeared consistently uses EC2 Spot Instances along with an aggressive deprovisioning strategy for underutilized nodes. I believe that node-pressure eviction could also cause this issue, along with any other external condition that would result in a non-graceful pod termination.
Got it. That's interesting that node-deprovisioning and node-pressure eviction would result in SIGKILL rather than SIGTERM.
Got it. That's interesting that node-deprovisioning and node-pressure eviction would result in SIGKILL rather than SIGTERM.
There are projects like NTH which should get you a SIGTERM with some time to do some work. Perhaps they're not being used here. For non-graceful termination though, we still need a solution.
I think we need to consider "Pod gone away" after a reasonable period (to allow for propogation of the WorkflowTaskResult) to be pod failure and mark the outputs as completed (with error) to allow the workflow to fail/retry.
Pre-requisites
:latest
image tag (i.e.quay.io/argoproj/workflow-controller:latest
) and can confirm the issue still exists on:latest
. If not, I have explained why, in detail, in my description below.What happened/what you expected to happen?
As part of #12402 (included from v3.5.3 onwards), workflow pod
wait
-container behavior was changed to create a placeholder (incomplete) WorkflowTaskResult before waiting for themain
-container to complete.https://github.com/argoproj/argo-workflows/blob/0fdf74511d4671cf0c8c334aa2d90ecd61c5acce/cmd/argoexec/commands/wait.go#L38-L42
The WorkflowTaskResult is finalized after output artifacts, logs etc. have been handled:
https://github.com/argoproj/argo-workflows/blob/0fdf74511d4671cf0c8c334aa2d90ecd61c5acce/cmd/argoexec/commands/wait.go#L34
If the
wait
-container is interrupted in a way that preventsFinalizeOutput
from being called (e.g. pod deletion without sufficient grace period), an incomplete WorkflowTaskResult remains with theworkflows.argoproj.io/report-outputs-completed
label set tofalse
. Retries of the same task will produce additional WorkflowTaskResults and will not mark the previous one complete. This leaves the workflow stuck inProcessing
state until the WorkflowTaskResult is manually edited to mark it complete.The reproduction example workflow simulates forced pod deletion using a pod that deletes itself, leaving behind an incomplete WorkflowTaskResult. The included workflow controller log snippet shows the resulting processing loop.
This issue may be one of the causes of #12103.
Version
v3.5.3
Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.
Logs from the workflow controller
Logs from in your workflow's wait container