inputs.parameters.message not supplied error depending on submission method

snsavage commented 1 year ago

Pre-requisites

[X] I have double-checked my configuration
[X] I can confirm the issues exists when I tested with :latest
[ ] I'd like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

After upgrading Argo Workflows to v3.4.6, we noticed that certain methods of submitting a WorkflowTemplate will result in an error while others will not.

The error that we're seeing is:

invalid spec: templates.on-exit inputs.parameters.message was not supplied

Specifically, we're seeing this when using onExit and referencing {{inputs.parameters.message}} where no default value is provided for the parameter in the on-exit step. When submitting the WorkflowTemplate through the Argo UI, it runs as expected and the WorkflowTemplate argument parameter is passed along to the onExit step.

Here are the logs when the WorkflowTemplate is successfully submitted through the Argo UI on v3.4.6.

test-on-exit-input-parameters-t5bzg: time="2023-04-17T19:40:39 UTC" level=info msg="capturing logs" argo=true
test-on-exit-input-parameters-t5bzg: entrypoint: hello world
test-on-exit-input-parameters-t5bzg: time="2023-04-17T19:40:40 UTC" level=info msg="sub-process exited" argo=true error="<nil>"
test-on-exit-input-parameters-t5bzg-on-exit-3464203860: time="2023-04-17T19:40:50 UTC" level=info msg="capturing logs" argo=true
test-on-exit-input-parameters-t5bzg-on-exit-3464203860: on-exit: hello world
test-on-exit-input-parameters-t5bzg-on-exit-3464203860: time="2023-04-17T19:40:51 UTC" level=info msg="sub-process exited" argo=true error="<nil>"

However, in Argo Workflows v3.4.6+ the following submission types trigger the error:

argo submit --from cronworkflow/test-on-exit-input-parameters-with-cron-workflow
argo submit --from workflowtemplate/test-on-exit-input-parameters
The Argo UI "Submit" button for the CronWorkflow.

The example CronWorkflow and WorkflowTemplate work as expected in v3.4.5, but not in any version greater than v3.4.6, including latest.

Version

v3.4.6

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

---
apiVersion: argoproj.io/v1alpha1
kind: CronWorkflow
metadata:
  name: test-on-exit-input-parameters-with-cron-workflow
spec:
  schedule: "0 5 * * *"
  timezone: "America/New_York"
  workflowSpec:
    workflowTemplateRef:
      name: test-on-exit-input-parameters

---
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: test-on-exit-input-parameters
spec:
  arguments:
    parameters:
      - name: message
        value: hello world

  entrypoint: entrypoint
  onExit: on-exit

  templates:
    - name: entrypoint
      inputs:
        parameters:
          - name: message
            value: '{{workflow.parameters.message}}'
      container:
        name: main
        image: argoproj/argosay:v2
        command:
          - /argosay
        args:
          - echo
          - 'entrypoint: {{inputs.parameters.message}}'

    - name: on-exit
      inputs:
        parameters:
          - name: message
      container:
        name: main
        image: argoproj/argosay:v2
        command:
          - /argosay
        args:
          - echo
          - 'on-exit: {{inputs.parameters.message}}'

Logs from the workflow controller

***** Logs from CronWorkflow *****
time="2023-04-17T19:18:24.709Z" level=info msg="Processing workflow" namespace=argo workflow=test-on-exit-input-parameters-with-cron-workflow-476rr
time="2023-04-17T19:18:24.710Z" level=info msg="Enforcing history limit for 'test-on-exit-input-parameters-with-cron-workflow'" namespace=argo workflow=test-on-exit-input-parameters-with-cron-workflow
time="2023-04-17T19:18:24.718Z" level=info msg="Updated phase  -> Failed" namespace=argo workflow=test-on-exit-input-parameters-with-cron-workflow-476rr
time="2023-04-17T19:18:24.718Z" level=info msg="Updated message  -> invalid spec: templates.on-exit inputs.parameters.message was not supplied" namespace=argo workflow=test-on-exit-input-parameters-with-cron-workflow-476rr
time="2023-04-17T19:18:24.718Z" level=info msg="Marking workflow completed" namespace=argo workflow=test-on-exit-input-parameters-with-cron-workflow-476rr
time="2023-04-17T19:18:24.718Z" level=error msg="Unable to set ExecWorkflow" error="templates.on-exit inputs.parameters.message was not supplied" namespace=argo workflow=test-on-exit-input-parameters-with-cron-workflow-476rr
time="2023-04-17T19:18:24.718Z" level=info msg="Checking daemoned children of " namespace=argo workflow=test-on-exit-input-parameters-with-cron-workflow-476rr
time="2023-04-17T19:18:24.723Z" level=info msg="cleaning up pod" action=deletePod key=argo/test-on-exit-input-parameters-with-cron-workflow-476rr-1340600742-agent/deletePod
time="2023-04-17T19:18:24.727Z" level=info msg="Workflow update successful" namespace=argo phase=Failed resourceVersion=596838162 workflow=test-on-exit-input-parameters-with-cron-workflow-476rr
time="2023-04-17T19:18:24.728Z" level=info msg="Queueing Failed workflow argo/test-on-exit-input-parameters-with-cron-workflow-476rr for delete in 72h0m0s due to TTL"

***** Logs from WorkflowTemplate Submitted through the Argo CLI ******
time="2023-04-17T19:55:15.625Z" level=info msg="Processing workflow" namespace=argo workflow=test-on-exit-input-parameters-ggbv6
time="2023-04-17T19:55:15.634Z" level=info msg="Updated phase  -> Failed" namespace=argo workflow=test-on-exit-input-parameters-ggbv6
time="2023-04-17T19:55:15.634Z" level=info msg="Updated message  -> invalid spec: templates.on-exit inputs.parameters.message was not supplied" namespace=argo workflow=test-on-exit-input-parameters-ggbv6
time="2023-04-17T19:55:15.634Z" level=info msg="Marking workflow completed" namespace=argo workflow=test-on-exit-input-parameters-ggbv6
time="2023-04-17T19:55:15.634Z" level=error msg="Unable to set ExecWorkflow" error="templates.on-exit inputs.parameters.message was not supplied" namespace=argo workflow=test-on-exit-input-parameters-ggbv6
time="2023-04-17T19:55:15.634Z" level=info msg="Checking daemoned children of " namespace=argo workflow=test-on-exit-input-parameters-ggbv6
time="2023-04-17T19:55:15.639Z" level=info msg="cleaning up pod" action=deletePod key=argo/test-on-exit-input-parameters-ggbv6-1340600742-agent/deletePod
time="2023-04-17T19:55:15.643Z" level=info msg="Workflow update successful" namespace=argo phase=Failed resourceVersion=596859819 workflow=test-on-exit-input-parameters-ggbv6
time="2023-04-17T19:55:15.644Z" level=info msg="Queueing Failed workflow argo/test-on-exit-input-parameters-ggbv6 for delete in 72h0m0s due to TTL"

Logs from in your workflow's wait container

N/A (Workflow never created.)

sarabala1979 commented 1 year ago

@snsavage I could be a bug in substituting parameters in exit handler. Do you like to triage and submit the PR?

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If this is a mentoring request, please provide an update here. Thank you for your contributions.

WhiteDiamondz commented 1 year ago

Hello,

I am encountering the same issue after an upgrade from v3.4.4 to v3.4.10. Has any workaround or fix been found ? @snsavage

snsavage commented 1 year ago

@WhiteDiamondz Our workaround for this has been to explicitly pull the onExit parameter values from the workflow parameters. From the example above, something like...

---
apiVersion: argoproj.io/v1alpha1
kind: CronWorkflow
metadata:
  name: test-on-exit-input-parameters-with-cron-workflow
spec:
  schedule: "0 5 * * *"
  timezone: "America/New_York"
  workflowSpec:
    workflowTemplateRef:
      name: test-on-exit-input-parameters

---
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: test-on-exit-input-parameters
spec:
  arguments:
    parameters:
      - name: message
        value: hello world

  entrypoint: entrypoint
  onExit: on-exit

  templates:
  ...

    - name: on-exit
      inputs:
        parameters:
          - name: message
            value: "{{workflow.parameters.message}}"
      container:
        name: main
        image: argoproj/argosay:v2
        command:
          - /argosay
        args:
          - echo
          - 'on-exit: {{inputs.parameters.message}}'

argoproj / argo-workflows