search

argoproj / argo-workflows

Workflow Engine for Kubernetes
https://argo-workflows.readthedocs.io/
Apache License 2.0
14.94k stars 3.19k forks source link

Supplied valueFrom ignores defaults after duration elapsed #12230

Open frauniki opened 10 months ago

frauniki commented 10 months ago

Pre-requisites

What happened/what you expected to happen?

If the following WF is executed, the default value should be output to outputs.message after the duration of the approval step, but it is not. The actual output is a template message "{{steps.approve.outputs.parameters.message}}".

This is the same issue as this discussion. https://github.com/argoproj/argo-workflows/discussions/7905

Expected output

 _________________ 
< default message >
 ----------------- 
    \
     \
      \     
                    ##        .            
              ## ## ##       ==            
           ## ## ## ##      ===            
       /""""""""""""""""___/ ===        
  ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~   
       \______ o          __/            
        \    \        __/             
          \____\______/

Actual output

 _________________________________________ 
/ {{steps.approve.outputs.parameters.mess \
\ age}}                                   /
 ----------------------------------------- 
    \
     \
      \     
                    ##        .            
              ## ## ##       ==            
           ## ## ## ##      ===            
       /""""""""""""""""___/ ===        
  ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~   
       \______ o          __/            
        \    \        __/             
          \____\______/

Version

v3.5.0

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: suspend-outputs
spec:
  entrypoint: suspend
  templates:
  - name: suspend
    steps:
    - - name: approve
        template: approve
    - - name: release
        template: whalesay
        arguments:
          parameters:
            - name: message
              value: "{{steps.approve.outputs.parameters.message}}"

  - name: approve
    suspend:
      duration: 20s
    inputs:
      parameters:
        - name: message
          default: default message
    outputs:
      parameters:
        - name: message
          valueFrom:
            #default: default message
            supplied: {}

  - name: whalesay
    inputs:
      parameters:
        - name: message
    container:
      image: docker/whalesay
      command: [cowsay]
      args: ["{{inputs.parameters.message}}"]

Logs from the workflow controller

time="2023-11-20T05:13:04.666Z" level=info msg="Processing workflow" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:04.722Z" level=info msg="Updated phase  -> Running" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:04.722Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:04.722Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:04.723Z" level=info msg="Steps node suspend-outputs-7hhgl initialized Running" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:04.723Z" level=info msg="StepGroup node suspend-outputs-7hhgl-849034079 initialized Running" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:04.723Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:04.723Z" level=info msg="Suspend node suspend-outputs-7hhgl-3077708136 initialized Pending" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:04.723Z" level=info msg="node suspend-outputs-7hhgl[0].approve suspended" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:04.723Z" level=info msg="node suspend-outputs-7hhgl-3077708136 phase Pending -> Running" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:04.723Z" level=info msg="Workflow step group node suspend-outputs-7hhgl-849034079 not yet completed" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:04.723Z" level=info msg="TaskSet Reconciliation" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:04.723Z" level=info msg=reconcileAgentPod namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:04.735Z" level=info msg="Workflow update successful" namespace=argo-workflows phase=Running resourceVersion=70269789 workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:14.724Z" level=info msg="Processing workflow" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:14.725Z" level=info msg="Task-result reconciliation" namespace=argo-workflows numObjs=0 workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:14.726Z" level=info msg="node suspend-outputs-7hhgl[0].approve suspended" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:14.726Z" level=info msg="auto resuming node suspend-outputs-7hhgl[0].approve" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:14.726Z" level=info msg="node suspend-outputs-7hhgl-3077708136 phase Running -> Succeeded" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:14.726Z" level=info msg="node suspend-outputs-7hhgl-3077708136 finished: 2023-11-20 05:13:14.726177679 +0000 UTC" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:14.726Z" level=info msg="Step group node suspend-outputs-7hhgl-849034079 successful" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:14.726Z" level=info msg="node suspend-outputs-7hhgl-849034079 phase Running -> Succeeded" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:14.726Z" level=info msg="node suspend-outputs-7hhgl-849034079 finished: 2023-11-20 05:13:14.726321191 +0000 UTC" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:14.726Z" level=info msg="StepGroup node suspend-outputs-7hhgl-1990059266 initialized Running" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:14.726Z" level=info msg="SG Outbound nodes of suspend-outputs-7hhgl-3077708136 are [suspend-outputs-7hhgl-3077708136]" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:14.726Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:14.726Z" level=info msg="Pod node suspend-outputs-7hhgl-2887505401 initialized Pending" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:14.763Z" level=info msg="Created pod: suspend-outputs-7hhgl[1].release (suspend-outputs-7hhgl-whalesay-2887505401)" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:14.763Z" level=info msg="Workflow step group node suspend-outputs-7hhgl-1990059266 not yet completed" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:14.771Z" level=info msg="TaskSet Reconciliation" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:14.771Z" level=info msg=reconcileAgentPod namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:14.786Z" level=info msg="Workflow update successful" namespace=argo-workflows phase=Running resourceVersion=70269882 workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:24.770Z" level=info msg="Processing workflow" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:24.772Z" level=info msg="Task-result reconciliation" namespace=argo-workflows numObjs=0 workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:24.772Z" level=info msg="node changed" namespace=argo-workflows new.message=PodInitializing new.phase=Pending new.progress=0/1 nodeID=suspend-outputs-7hhgl-2887505401 old.message= old.phase=Pending old.progress=0/1 workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:24.772Z" level=info msg="SG Outbound nodes of suspend-outputs-7hhgl-3077708136 are [suspend-outputs-7hhgl-3077708136]" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:24.773Z" level=info msg="Workflow step group node suspend-outputs-7hhgl-1990059266 not yet completed" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:24.773Z" level=info msg="TaskSet Reconciliation" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:24.773Z" level=info msg=reconcileAgentPod namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:24.784Z" level=info msg="Workflow update successful" namespace=argo-workflows phase=Running resourceVersion=70269953 workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:34.789Z" level=info msg="Processing workflow" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:34.814Z" level=info msg="Task-result reconciliation" namespace=argo-workflows numObjs=0 workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:34.814Z" level=info msg="node unchanged" namespace=argo-workflows nodeID=suspend-outputs-7hhgl-2887505401 workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:34.815Z" level=info msg="SG Outbound nodes of suspend-outputs-7hhgl-3077708136 are [suspend-outputs-7hhgl-3077708136]" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:34.825Z" level=info msg="Workflow step group node suspend-outputs-7hhgl-1990059266 not yet completed" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:34.825Z" level=info msg="TaskSet Reconciliation" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:34.826Z" level=info msg=reconcileAgentPod namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:50.312Z" level=info msg="Processing workflow" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:50.313Z" level=info msg="Task-result reconciliation" namespace=argo-workflows numObjs=0 workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:50.313Z" level=info msg="node unchanged" namespace=argo-workflows nodeID=suspend-outputs-7hhgl-2887505401 workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:50.313Z" level=info msg="SG Outbound nodes of suspend-outputs-7hhgl-3077708136 are [suspend-outputs-7hhgl-3077708136]" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:50.314Z" level=info msg="Workflow step group node suspend-outputs-7hhgl-1990059266 not yet completed" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:50.314Z" level=info msg="TaskSet Reconciliation" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:13:50.314Z" level=info msg=reconcileAgentPod namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:14:17.514Z" level=info msg="Processing workflow" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:14:17.516Z" level=info msg="Task-result reconciliation" namespace=argo-workflows numObjs=1 workflow=suspend-outputs-7hhgl
time="2023-11-20T05:14:17.516Z" level=info msg="task-result changed" namespace=argo-workflows nodeID=suspend-outputs-7hhgl-2887505401 workflow=suspend-outputs-7hhgl
time="2023-11-20T05:14:17.516Z" level=info msg="node changed" namespace=argo-workflows new.message= new.phase=Succeeded new.progress=0/1 nodeID=suspend-outputs-7hhgl-2887505401 old.message=PodInitializing old.phase=Pending old.progress=0/1 workflow=suspend-outputs-7hhgl
time="2023-11-20T05:14:17.516Z" level=info msg="SG Outbound nodes of suspend-outputs-7hhgl-3077708136 are [suspend-outputs-7hhgl-3077708136]" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:14:17.516Z" level=info msg="Step group node suspend-outputs-7hhgl-1990059266 successful" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:14:17.516Z" level=info msg="node suspend-outputs-7hhgl-1990059266 phase Running -> Succeeded" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:14:17.516Z" level=info msg="node suspend-outputs-7hhgl-1990059266 finished: 2023-11-20 05:14:17.516880854 +0000 UTC" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:14:17.516Z" level=info msg="Outbound nodes of suspend-outputs-7hhgl-2887505401 is [suspend-outputs-7hhgl-2887505401]" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:14:17.516Z" level=info msg="Outbound nodes of suspend-outputs-7hhgl is [suspend-outputs-7hhgl-2887505401]" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:14:17.516Z" level=info msg="node suspend-outputs-7hhgl phase Running -> Succeeded" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:14:17.516Z" level=info msg="node suspend-outputs-7hhgl finished: 2023-11-20 05:14:17.516961617 +0000 UTC" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:14:17.516Z" level=info msg="TaskSet Reconciliation" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:14:17.517Z" level=info msg=reconcileAgentPod namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:14:17.517Z" level=info msg="Updated phase Running -> Succeeded" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:14:17.517Z" level=info msg="Marking workflow completed" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:14:17.517Z" level=info msg="Marking workflow as pending archiving" namespace=argo-workflows workflow=suspend-outputs-7hhgl
time="2023-11-20T05:14:17.523Z" level=info msg="cleaning up pod" action=deletePod key=argo-workflows/suspend-outputs-7hhgl-1340600742-agent/deletePod
time="2023-11-20T05:14:17.529Z" level=info msg="Workflow update successful" namespace=argo-workflows phase=Succeeded resourceVersion=70270646 workflow=suspend-outputs-7hhgl
time="2023-11-20T05:14:17.559Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo-workflows/suspend-outputs-7hhgl-whalesay-2887505401/labelPodCompleted
time="2023-11-20T05:14:17.561Z" level=info msg="archiving workflow" namespace=argo-workflows uid=f19b2638-fefd-42d5-bd45-33d952801b5e workflow=suspend-outputs-7hhgl

Logs from in your workflow's wait container

time="2023-11-20T05:14:08.640Z" level=info msg="No output artifacts"
time="2023-11-20T05:14:08.642Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: (secret)/suspend-outputs-7hhgl/suspend-outputs-7hhgl-whalesay-2887505401/main.log"
time="2023-11-20T05:14:08.666Z" level=info msg="Creating minio client using AWS SDK credentials"
time="2023-11-20T05:14:09.019Z" level=info msg="Saving file to s3" bucket=(secret) endpoint=s3.amazonaws.com key=(secret)/suspend-outputs-7hhgl/suspend-outputs-7hhgl-whalesay-2887505401/main.log path=/tmp/argo/outputs/logs/main.log
time="2023-11-20T05:14:09.384Z" level=info msg="Save artifact" artifactName=main-logs duration=742.728571ms error="<nil>" key=(secret)/suspend-outputs-7hhgl/suspend-outputs-7hhgl-whalesay-2887505401/main.log
time="2023-11-20T05:14:09.385Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2023-11-20T05:14:09.385Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2023-11-20T05:14:09.620Z" level=info msg="Alloc=10149 TotalAlloc=16985 Sys=23397 NumGC=4 Goroutines=12"
time="2023-11-20T05:14:09.620Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2023-11-20T05:14:09.621Z" level=info msg="Deadline monitor stopped"
jswxstw commented 6 months ago

https://github.com/argoproj/argo-workflows/blob/ec5b5d5f49d33172cd5dd018b4c5434551d93970/workflow/controller/operator.go#L3355-L3372

Node is marked to Succeeded when suspension is expired, but the value of node outputs parameters was not filled up with ValueFrom.Default if ValueFrom.Supplied != nil.

jswxstw commented 6 months ago

"{{steps.approve.outputs.parameters.message}}" can not be resolved when ValueFrom.Default in the outputs is not set. The behavior now is that the raw text will be passed to the template which I think it is not very reasonable.