Unknown paths are appended to file paths in parameters

Pre-requisites

[X] I have double-checked my configuration
[X] I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
[X] I have searched existing issues and could not find a match for this bug
[ ] I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

When i set the path value like this, the file will not be found because it is prefixed with the path /mainctrfs

      outputs:
        parameters:
        - name: deployment-yaml
          valueFrom:
            path: /work2/cdk8s/backend/dist/Deployment.spring-deployment.k8s.yaml
        - name: service-yaml
          valueFrom:
            path: /work2/cdk8s/backend/dist/Service.spring-service.k8s.yaml
        - name: ingress-yaml
          valueFrom:
            path: /work2/cdk8s/backend/dist/Ingress.spring-ingress.k8s.yaml

The strangest thing is that this issue doesn't happen every time you run the workflow, but rather periodically for the same template. I think the next issue is a similar phenomenon. https://github.com/argoproj/argo-workflows/issues/5825

Version(s)

v.3.5.8

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

The entire of that stage.

 - name: make-deploy-yaml
      inputs:
        parameters:
          - name: image
          - name: port
          - name: app_name
          - name: replicas
          - name: cpu
          - name: memory
      container:
        env:
          - name: DOCKER_IMAGE
            value: "{{inputs.parameters.image}}"
          - name: PORT
            value: "{{inputs.parameters.port}}"
          - name: APP_NAME
            value: "{{inputs.parameters.app_name}}"
          - name: REPLICAS
            value: "{{inputs.parameters.replicas}}"
          - name: CPU
            value: "{{inputs.parameters.cpu}}"
          - name: MEMORY
            value: "{{inputs.parameters.memory}}"
        volumeMounts:
          - mountPath: /work2
            name: work2
        workingDir: /work2/cdk8s/backend
        image: brennerm/cdk8s:latest
        args:
          - synth
      outputs:
        parameters:
        - name: deployment-yaml
          valueFrom:
            path: /work2/cdk8s/backend/dist/Deployment.spring-deployment.k8s.yaml
        - name: service-yaml
          valueFrom:
            path: /work2/cdk8s/backend/dist/Service.spring-service.k8s.yaml
        - name: ingress-yaml
          valueFrom:
            path: /work2/cdk8s/backend/dist/Ingress.spring-ingress.k8s.yaml

Logs from the workflow controller

time="2024-11-06T16:30:44.004Z" level=info msg="Task-result reconciliation" namespace=argo-events numObjs=0 workflow=yacht-workflow-sbwhr
time="2024-11-06T16:30:44.004Z" level=warning msg="workflow uses legacy/insecure pod patch, see https://argoproj.github.io/argo-workflows/workflow-rbac/" namespace=argo-events workflow=yacht-workflow-sbwhr
time="2024-11-06T16:30:44.004Z" level=info msg="node changed" namespace=argo-events new.message="Error (exit code 1): open /mainctrfs/work2/cdk8s/backend/dist/Deployment.spring-deployment.k8s.yaml: no such file or directory" new.phase=Error new.progress=0/1 nodeID=yacht-workflow-sbwhr-3406326865 old.message= old.phase=Running old.progress=0/1 workflow=yacht-workflow-sbwhr
time="2024-11-06T16:30:44.005Z" level=info msg="node unchanged" namespace=argo-events nodeID=yacht-workflow-sbwhr-1623128504 workflow=yacht-workflow-sbwhr
time="2024-11-06T16:30:44.006Z" level=info msg="TaskSet Reconciliation" namespace=argo-events workflow=yacht-workflow-sbwhr
time="2024-11-06T16:30:44.006Z" level=info msg=reconcileAgentPod namespace=argo-events workflow=yacht-workflow-sbwhr
time="2024-11-06T16:30:44.031Z" level=info msg="Workflow update successful" namespace=argo-events phase=Running resourceVersion=9443424 workflow=yacht-workflow-sbwhr
time="2024-11-06T16:30:44.038Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo-events/yacht-workflow-sbwhr-make-deploy-yaml-3406326865/labelPodCompleted
time="2024-11-06T16:30:54.072Z" level=info msg="Processing workflow" namespace=argo-events workflow=yacht-workflow-sbwhr
time="2024-11-06T16:30:54.073Z" level=info msg="Task-result reconciliation" namespace=argo-events numObjs=0 workflow=yacht-workflow-sbwhr
time="2024-11-06T16:30:54.073Z" level=info msg="node unchanged" namespace=argo-events nodeID=yacht-workflow-sbwhr-1623128504 workflow=yacht-workflow-sbwhr
time="2024-11-06T16:30:54.074Z" level=info msg="TaskSet Reconciliation" namespace=argo-events workflow=yacht-workflow-sbwhr
time="2024-11-06T16:30:54.074Z" level=info msg=reconcileAgentPod namespace=argo-events workflow=yacht-workflow-sbwhr
time="2024-11-06T16:31:06.242Z" level=info msg="Processing workflow" namespace=argo-events workflow=yacht-workflow-sbwhr
time="2024-11-06T16:31:06.243Z" level=info msg="Task-result reconciliation" namespace=argo-events numObjs=0 workflow=yacht-workflow-sbwhr

Logs from in your workflow's wait container

time="2024-11-06T16:29:03.319Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2024-11-06T16:29:03.320Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=argo-events podName=yacht-workflow-sbwhr-build-and-push-1623128504 template="{\"name\":\"build-and-push\",\"inputs\":{\"parameters\":[{\"name\":\"path\",\"value\":\".\"},{\"name\":\"image\",\"value\":\"yacht24/new-test:v0.0.1\"}]},\"outputs\":{},\"metadata\":{},\"container\":{\"name\":\"\",\"image\":\"moby/buildkit:v0.9.3-rootless\",\"command\":[\"buildctl-daemonless.sh\"],\"args\":[\"build\",\"--frontend\",\"dockerfile.v0\",\"--local\",\"context=.\",\"--local\",\"dockerfile=.\",\"--output\",\"type=image,name=docker.io/yacht24/new-test:v0.0.1,push=true\"],\"workingDir\":\"/work/.\",\"env\":[{\"name\":\"BUILDKITD_FLAGS\",\"value\":\"--oci-worker-no-process-sandbox\"},{\"name\":\"DOCKER_CONFIG\",\"value\":\"/.docker\"}],\"resources\":{},\"volumeMounts\":[{\"name\":\"work\",\"mountPath\":\"/work\"},{\"name\":\"docker-config\",\"mountPath\":\"/.docker\"}],\"readinessProbe\":{\"exec\":{\"command\":[\"sh\",\"-c\",\"buildctl debug workers\"]}},\"securityContext\":{\"privileged\":true}},\"volumes\":[{\"name\":\"docker-config\",\"secret\":{\"secretName\":\"docker-config\"}}]}" version="&Version{Version:v3.4.4,BuildDate:2022-11-29T16:49:53Z,GitCommit:3b2626ff900aff2424c086a51af5929fb0b2d7e5,GitTag:v3.4.4,GitTreeState:clean,GoVersion:go1.18.8,Compiler:gc,Platform:linux/amd64,}"
time="2024-11-06T16:29:03.320Z" level=info msg="Starting deadline monitor"
time="2024-11-06T16:30:56.138Z" level=info msg="Main container completed" error="<nil>"
time="2024-11-06T16:30:56.138Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2024-11-06T16:30:56.138Z" level=info msg="No output parameters"
time="2024-11-06T16:30:56.138Z" level=info msg="No output artifacts"
time="2024-11-06T16:30:56.182Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2024-11-06T16:30:56.182Z" level=info msg="Alloc=6340 TotalAlloc=12135 Sys=18386 NumGC=4 Goroutines=6"
time="2024-11-06T16:30:56.182Z" level=info msg="Deadline monitor stopped"
unable to retrieve container logs for containerd://994c06f0221ad054f95fb2dc6181ffbed511a35a59f26bc2b780f0b74ca92240unable to retrieve container logs for containerd://7e3674891112f0b3b4c68877be96b894f1f8ce13b470fc49d0180b7d89f887f9time="2024-11-06T16:30:33.681Z" level=info msg="Copying /work2/cdk8s/backend/dist/Deployment.spring-deployment.k8s.yaml from volume mount"
time="2024-11-06T16:30:33.681Z" level=error msg="executor error: open /mainctrfs/work2/cdk8s/backend/dist/Deployment.spring-deployment.k8s.yaml: no such file or directory"
time="2024-11-06T16:30:33.681Z" level=info msg="No output artifacts"
time="2024-11-06T16:30:33.929Z" level=info msg="Create workflowtaskresults 403"
time="2024-11-06T16:30:33.932Z" level=warning msg="failed to patch task set, falling back to legacy/insecure pod patch, see https://argoproj.github.io/argo-workflows/workflow-rbac/" error="workflowtaskresults.argoproj.io is forbidden: User \"system:serviceaccount:argo-events:default\" cannot create resource \"workflowtaskresults\" in API group \"argoproj.io\" in the namespace \"argo-events\""
time="2024-11-06T16:30:34.003Z" level=info msg="Patch pods 200"
time="2024-11-06T16:30:34.089Z" level=info msg="Deadline monitor stopped"
time="2024-11-06T16:30:34.092Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2024-11-06T16:30:34.092Z" level=info msg="Alloc=7011 TotalAlloc=12920 Sys=18386 NumGC=4 Goroutines=7"
time="2024-11-06T16:30:34.125Z" level=fatal msg="open /mainctrfs/work2/cdk8s/backend/dist/Deployment.spring-deployment.k8s.yaml: no such file or directory"

argoproj / argo-workflows

Unknown paths are appended to file paths in parameters #13872