search

argoproj / argo-workflows

Workflow Engine for Kubernetes
https://argo-workflows.readthedocs.io/
Apache License 2.0
15.11k stars 3.2k forks source link

Inconsistent variable substitution in Inputs.Parameters block #13890

Open encigem opened 1 week ago

encigem commented 1 week ago

Pre-requisites

What happened? What did you expect to happen?

I'm having some issue with variable substitutions for 'inputs.paramters' blocks in Workflow files. For example, i've provided a YAML file which reproduces my issue. In 'step1' the name: "{{workflow.parameters.cm-name-full}}" works within the configMapKeyRef , but with the name: "{{workflow.parameters.cm-name}}-cm" in step2 it does not.

screenshot_2024-11-08_at_16-30-12_substitute-test-fpg66___docklin-il___workflows_-_argo

What Happened?: The step with partial variable string input failed, but the step with input consisting only of a string with a variable succeeded.

What I expected?: I expected that both of the steps should've been able to substitute the given variable in the 'inputs.parameters' block

Version(s)

v3.5.10, latest(b26ed4aa4dee395844531efa4a76a022183bec22)

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: v1
kind: ConfigMap
data:
  testVar: testVal
metadata:
  labels:
    workflows.argoproj.io/configmap-type: Parameter
  name: test-cm
  namespace: argo

---

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: substitute-test-
  namespace: argo
spec:
  entrypoint: steps
  arguments:
    parameters:
      - name: image
        value: "alpine:3.17"
      - name: cm-key
        value: "testVar"
      - name: cm-name
        value: "test"
      - name: cm-name-full
        value: "test-cm"
  templates:
  - name: steps
    steps:
    - - name: step1
        template: step1
    - - name: step2
        template: step2

  - name: step1
    inputs:
      parameters:
        - name: cm-var
          valueFrom:
            configMapKeyRef:
              name: "{{workflow.parameters.cm-name-full}}" # only variable defined; succeeds
              key: "{{workflow.parameters.cm-key}}"
    container:
      image: "{{workflow.parameters.image}}"
      command: [sh, -c]
      args: ["echo 'testVar value is: {{inputs.parameters.cm-var}}'"]

  - name: step2
    inputs:
      parameters:
        - name: cm-var
          valueFrom:
            configMapKeyRef:
              name: "{{workflow.parameters.cm-name}}-cm"  # combine variable with string; fails
              key: "{{workflow.parameters.cm-key}}"
    container:
      image: "{{workflow.parameters.image}}"
      command: [sh, -c]
      args: ["echo 'testVar value is: {{inputs.parameters.cm-var}}'"]

Logs from the workflow controller

kubectl logs -n argo deploy/workflow-controller | grep ${workflow}

time="2024-11-08T16:25:42.231Z" level=info msg="Processing workflow" Phase= ResourceVersion=725462700 namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:42.237Z" level=info msg="Task-result reconciliation" namespace=test-il numObjs=0 workflow=substitute-test-fpg66
time="2024-11-08T16:25:42.237Z" level=info msg="Updated phase  -> Running" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:42.237Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:42.237Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:42.237Z" level=info msg="Steps node substitute-test-fpg66 initialized Running" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:42.238Z" level=info msg="StepGroup node substitute-test-fpg66-2823940808 initialized Running" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:42.238Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:42.238Z" level=info msg="Pod node substitute-test-fpg66-296026515 initialized Pending" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:42.264Z" level=info msg="Created pod: substitute-test-fpg66[0].step1 (substitute-test-fpg66-step1-296026515)" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:42.265Z" level=info msg="Workflow step group node substitute-test-fpg66-2823940808 not yet completed" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:42.265Z" level=info msg="TaskSet Reconciliation" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:42.265Z" level=info msg=reconcileAgentPod namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:42.286Z" level=info msg="Workflow update successful" namespace=test-il phase=Running resourceVersion=725462704 workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.266Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=725462704 namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.267Z" level=info msg="Task-result reconciliation" namespace=test-il numObjs=1 workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.267Z" level=info msg="task-result changed" namespace=test-il nodeID=substitute-test-fpg66-296026515 workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="node changed" namespace=test-il new.message= new.phase=Succeeded new.progress=0/1 nodeID=substitute-test-fpg66-296026515 old.message= old.phase=Pending old.progress=0/1 workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="Step group node substitute-test-fpg66-2823940808 successful" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="node substitute-test-fpg66-2823940808 phase Running -> Succeeded" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="node substitute-test-fpg66-2823940808 finished: 2024-11-08 16:25:52.268414392 +0000 UTC" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="StepGroup node substitute-test-fpg66-3964671805 initialized Running" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="SG Outbound nodes of substitute-test-fpg66-296026515 are [substitute-test-fpg66-296026515]" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="Skipped node substitute-test-fpg66-3148240723 initialized Error (message: unable to retrieve inputs.parameters.cm-var from ConfigMap: ConfigMap '{{workflow.parameters.cm-name}}-cm' does not exist. Please make sure it has the label workflows.argoproj.io/configmap-type: Parameter to be detectable by the controller)" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=error msg="Mark error node" error="step group deemed errored due to child substitute-test-fpg66[1].step2 error: unable to retrieve inputs.parameters.cm-var from ConfigMap: ConfigMap '{{workflow.parameters.cm-name}}-cm' does not exist. Please make sure it has the label workflows.argoproj.io/configmap-type: Parameter to be detectable by the controller" namespace=test-il nodeName="substitute-test-fpg66[1]" workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="node substitute-test-fpg66-3964671805 phase Running -> Error" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="node substitute-test-fpg66-3964671805 message: step group deemed errored due to child substitute-test-fpg66[1].step2 error: unable to retrieve inputs.parameters.cm-var from ConfigMap: ConfigMap '{{workflow.parameters.cm-name}}-cm' does not exist. Please make sure it has the label workflows.argoproj.io/configmap-type: Parameter to be detectable by the controller" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="node substitute-test-fpg66-3964671805 finished: 2024-11-08 16:25:52.268667688 +0000 UTC" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="step group substitute-test-fpg66-3964671805 was unsuccessful: step group deemed errored due to child substitute-test-fpg66[1].step2 error: unable to retrieve inputs.parameters.cm-var from ConfigMap: ConfigMap '{{workflow.parameters.cm-name}}-cm' does not exist. Please make sure it has the label workflows.argoproj.io/configmap-type: Parameter to be detectable by the controller" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="Outbound nodes of substitute-test-fpg66-3148240723 is [substitute-test-fpg66-3148240723]" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="Outbound nodes of substitute-test-fpg66 is [substitute-test-fpg66-3148240723]" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="node substitute-test-fpg66 phase Running -> Failed" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="node substitute-test-fpg66 message: step group deemed errored due to child substitute-test-fpg66[1].step2 error: unable to retrieve inputs.parameters.cm-var from ConfigMap: ConfigMap '{{workflow.parameters.cm-name}}-cm' does not exist. Please make sure it has the label workflows.argoproj.io/configmap-type: Parameter to be detectable by the controller" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="node substitute-test-fpg66 finished: 2024-11-08 16:25:52.268736598 +0000 UTC" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="TaskSet Reconciliation" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg=reconcileAgentPod namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="Updated phase Running -> Failed" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="Updated message  -> step group deemed errored due to child substitute-test-fpg66[1].step2 error: unable to retrieve inputs.parameters.cm-var from ConfigMap: ConfigMap '{{workflow.parameters.cm-name}}-cm' does not exist. Please make sure it has the label workflows.argoproj.io/configmap-type: Parameter to be detectable by the controller" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="Marking workflow completed" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.300Z" level=info msg="Workflow update successful" namespace=test-il phase=Failed resourceVersion=725462793 workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.302Z" level=info msg="Queueing Failed workflow test-il/substitute-test-fpg66 for delete in 168h0m0s due to TTL"
time="2024-11-08T16:26:02.382Z" level=info msg="Queueing Failed workflow test-il/substitute-test-fpg66 for delete in 167h59m50s due to TTL"
time="2024-11-08T16:46:02.378Z" level=info msg="Queueing Failed workflow test-il/substitute-test-fpg66 for delete in 167h39m50s due to TTL"
time="2024-11-08T17:06:02.389Z" level=info msg="Queueing Failed workflow test-il/substitute-test-fpg66 for delete in 167h19m50s due to TTL"
time="2024-11-08T17:25:52.321Z" level=info msg="cleaning up pod" action=deletePod key=test-il/substitute-test-fpg66-step1-296026515/deletePod
time="2024-11-08T17:26:02.383Z" level=info msg="Queueing Failed workflow test-il/substitute-test-fpg66 for delete in 166h59m50s due to TTL"

Logs from in your workflow's wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

time="2024-11-11T13:15:40.526Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2024-11-11T13:15:40.526Z" level=info msg="No output parameters"
time="2024-11-11T13:15:40.526Z" level=info msg="No output artifacts"
time="2024-11-11T13:15:40.527Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: artifacts/2024/11/11/substitute-test-7bbdj/substitute-test-7bbdj-step1-1047580867/main.log"
time="2024-11-11T13:15:40.527Z" level=info msg="Creating minio client using static credentials" endpoint="seliis3obj.seli.gic.company.se:10444"
time="2024-11-11T13:15:40.527Z" level=info msg="Saving file to s3" bucket=testdev-moore030 endpoint="seliis3obj.seli.gic.company.se:10444" key=artifacts/2024/11/11/substitute-test-7bbdj/substitute-test-7bbdj-step1-1047580867/main.log path=/tmp/argo/outputs/logs/main.log
time="2024-11-11T13:15:40.601Z" level=info msg="Save artifact" artifactName=main-logs duration=74.237363ms error="<nil>" key=artifacts/2024/11/11/substitute-test-7bbdj/substitute-test-7bbdj-step1-1047580867/main.log
time="2024-11-11T13:15:40.601Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2024-11-11T13:15:40.601Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2024-11-11T13:15:40.632Z" level=info msg="Alloc=7710 TotalAlloc=17424 Sys=22357 NumGC=5 Goroutines=10"
tooptoop4 commented 1 week ago

this should be marked as feature not bug

chengjoey commented 5 days ago 
      - name: cm-key
        value: "testVar"

Directly replace it with cm-var here, configMapKeyRef can solve your problem?

MasonM commented 2 days ago

I agree that this constitutes a bug. I entered https://github.com/argoproj/argo-workflows/pull/13921 to fix it.

Thanks for the detailed bug report @encigem!