WorkflowTemplate podMetadata not working

Pre-requisites

[X] I have double-checked my configuration
[X] I can confirm the issue exists when I tested with :latest
[X] I have searched existing issues and could not find a match for this bug
[ ] I'd like to contribute the fix myself (see contributing guide)

What happened/what did you expect to happen?

When podMetadata is specified in a ClusterWorkflowTemplate and the template is invoked from a Workflow instance, the pods are not getting the annotations and labels.

I'm not familiar with the code but the #6512 change seems to have replaced one behavior with another. Since both Workflow and (Cluster)WorkflowTemplate resources have the spec.podMetadata field, users rightfully expect that they should be able to specify ClusterWorkflowTemplate-level annotations and labels for pods instead of repeating the same information in eg. every container template within that ClusterWorkflowTemplate.

In addition, when the ClusterWorkflowTemplate is invoked from a Workflow, these annotations and labels should be possible to override, so IMO the Workflow spec.podMetadata should take precedence.

I'm not sure if this is correct, but perhaps in workflow/controller/workflowpod.go something like this would work:

// addMetadata applies metadata specified in the template
func (woc *wfOperationCtx) addMetadata(pod *apiv1.Pod, tmpl *wfv1.Template) {
    // Support WorkflowTemplate-level podMetadata 
    workflowMetadata := woc.wf.Spec.PodMetadata
    if executingWorkflowMetadata != nil {
        for k, v := range workflowMetadata.Annotations {
            pod.ObjectMeta.Annotations[k] = v
        }
        for k, v := range workflowMetadata.Labels {
            pod.ObjectMeta.Labels[k] = v
        }
    }

    // Workflow-level podMetadata takes precedence over WorkflowTemplate-level podMetadata
    executingWorkflowMetadata := woc.execWf.Spec.PodMetadata
    if executingWorkflowMetadata != nil {
        // add workflow-level pod annotations and labels
        for k, v := range executingWorkflowMetadata.Annotations {
            pod.ObjectMeta.Annotations[k] = v
        }
        for k, v := range executingWorkflowMetadata.Labels {
            pod.ObjectMeta.Labels[k] = v
        }
    }

    for k, v := range tmpl.Metadata.Annotations {
        pod.ObjectMeta.Annotations[k] = v
    }
    for k, v := range tmpl.Metadata.Labels {
        pod.ObjectMeta.Labels[k] = v
    }
}

Version

v3.5.5

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: ClusterWorkflowTemplate
metadata:
  name: workflow-template
spec:
  entrypoint: whalesay-template

  podMetadata:
    labels:
      workflow-template-label: hello
    annotations:
      all-pods-should-have-this: value

  arguments:
    parameters:
      - name: message
        value: hello world

  templates:
    - name: whalesay-template
      inputs:
        parameters:
          - name: message
      container:
        image: docker/whalesay
        command: [cowsay]
        args: ["{{inputs.parameters.message}}"]
---
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: test-workflow
  namespace: argo-workflows-system
spec:
  podMetadata:
    labels:
      caller-label: hello

  entrypoint: start
  templates:
    - name: start
      steps:
        - - name: hello
            templateRef:
              name: workflow-template
              template: whalesay-template
              clusterScope: true
            arguments:
              parameters:
                - name: message
                  value: Hello Bug

Logs from the workflow controller

{"Phase":"","ResourceVersion":"74873485","level":"info","msg":"Processing workflow","namespace":"argo","time":"2024-04-16T13:40:36.947Z","workflow":"test-workflow"}
{"level":"info","msg":"Task-result reconciliation","namespace":"argo","numObjs":0,"time":"2024-04-16T13:40:36.970Z","workflow":"test-workflow"}
{"level":"info","msg":"Updated phase  -\u003e Running","namespace":"argo","time":"2024-04-16T13:40:36.970Z","workflow":"test-workflow"}
{"level":"warning","msg":"Node was nil, will be initialized as type Skipped","namespace":"argo","time":"2024-04-16T13:40:36.970Z","workflow":"test-workflow"}
{"level":"info","msg":"was unable to obtain node for , letting display name to be nodeName","namespace":"argo","time":"2024-04-16T13:40:36.971Z","workflow":"test-workflow"}
{"level":"info","msg":"Steps node test-workflow initialized Running","namespace":"argo","time":"2024-04-16T13:40:36.971Z","workflow":"test-workflow"}
{"level":"info","msg":"StepGroup node test-workflow-3484831391 initialized Running","namespace":"argo","time":"2024-04-16T13:40:36.971Z","workflow":"test-workflow"}
{"level":"warning","msg":"Node was nil, will be initialized as type Skipped","namespace":"argo","time":"2024-04-16T13:40:36.971Z","workflow":"test-workflow"}
{"level":"info","msg":"Pod node test-workflow-2831459201 initialized Pending","namespace":"argo","time":"2024-04-16T13:40:36.971Z","workflow":"test-workflow"}
{"level":"info","msg":"Created pod: test-workflow[0].hello (test-workflow-whalesay-template-2831459201)","namespace":"argo","time":"2024-04-16T13:40:37.075Z","workflow":"test-workflow"}
{"level":"info","msg":"Workflow step group node test-workflow-3484831391 not yet completed","namespace":"argo","time":"2024-04-16T13:40:37.075Z","workflow":"test-workflow"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"argo","time":"2024-04-16T13:40:37.075Z","workflow":"test-workflow"}
{"level":"info","msg":"reconcileAgentPod","namespace":"argo","time":"2024-04-16T13:40:37.075Z","workflow":"test-workflow"}
{"level":"info","msg":"Workflow update successful","namespace":"argo","phase":"Running","resourceVersion":"74873494","time":"2024-04-16T13:40:37.092Z","workflow":"test-workflow"}
{"Phase":"","ResourceVersion":"74873605","level":"info","msg":"Processing workflow","namespace":"argo","time":"2024-04-16T13:40:41.910Z","workflow":"test-workflow"}
{"level":"info","msg":"Task-result reconciliation","namespace":"argo","numObjs":0,"time":"2024-04-16T13:40:41.940Z","workflow":"test-workflow"}
{"level":"info","msg":"Updated phase  -\u003e Running","namespace":"argo","time":"2024-04-16T13:40:41.940Z","workflow":"test-workflow"}
{"level":"warning","msg":"Node was nil, will be initialized as type Skipped","namespace":"argo","time":"2024-04-16T13:40:41.940Z","workflow":"test-workflow"}
{"level":"info","msg":"was unable to obtain node for , letting display name to be nodeName","namespace":"argo","time":"2024-04-16T13:40:41.940Z","workflow":"test-workflow"}
{"level":"info","msg":"Steps node test-workflow initialized Running","namespace":"argo","time":"2024-04-16T13:40:41.940Z","workflow":"test-workflow"}
{"level":"info","msg":"StepGroup node test-workflow-3484831391 initialized Running","namespace":"argo","time":"2024-04-16T13:40:41.940Z","workflow":"test-workflow"}
{"level":"warning","msg":"Node was nil, will be initialized as type Skipped","namespace":"argo","time":"2024-04-16T13:40:41.940Z","workflow":"test-workflow"}
{"level":"info","msg":"Pod node test-workflow-2831459201 initialized Pending","namespace":"argo","time":"2024-04-16T13:40:41.940Z","workflow":"test-workflow"}
{"level":"info","msg":"Workflow step group node test-workflow-3484831391 not yet completed","namespace":"argo","time":"2024-04-16T13:40:41.940Z","workflow":"test-workflow"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"argo","time":"2024-04-16T13:40:41.940Z","workflow":"test-workflow"}
{"level":"info","msg":"reconcileAgentPod","namespace":"argo","time":"2024-04-16T13:40:41.940Z","workflow":"test-workflow"}
{"level":"info","msg":"Workflow update successful","namespace":"argo","phase":"Running","resourceVersion":"74873609","time":"2024-04-16T13:40:41.957Z","workflow":"test-workflow"}
{"Phase":"Running","ResourceVersion":"74873609","level":"info","msg":"Processing workflow","namespace":"argo","time":"2024-04-16T13:40:47.072Z","workflow":"test-workflow"}
{"level":"info","msg":"Task-result reconciliation","namespace":"argo","numObjs":0,"time":"2024-04-16T13:40:47.073Z","workflow":"test-workflow"}
{"level":"info","msg":"Workflow pod is missing","namespace":"argo","nodeName":"test-workflow[0].hello","nodePhase":"Pending","recentlyStarted":true,"time":"2024-04-16T13:40:47.073Z","workflow":"test-workflow"}
{"level":"info","msg":"Created pod: test-workflow[0].hello (test-workflow-whalesay-template-2831459201)","namespace":"argo","time":"2024-04-16T13:40:47.171Z","workflow":"test-workflow"}
{"level":"info","msg":"Workflow step group node test-workflow-3484831391 not yet completed","namespace":"argo","time":"2024-04-16T13:40:47.171Z","workflow":"test-workflow"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"argo","time":"2024-04-16T13:40:47.171Z","workflow":"test-workflow"}
{"level":"info","msg":"reconcileAgentPod","namespace":"argo","time":"2024-04-16T13:40:47.171Z","workflow":"test-workflow"}
{"Phase":"Running","ResourceVersion":"74873609","level":"info","msg":"Processing workflow","namespace":"argo","time":"2024-04-16T13:40:57.172Z","workflow":"test-workflow"}
{"level":"info","msg":"Task-result reconciliation","namespace":"argo","numObjs":1,"time":"2024-04-16T13:40:57.173Z","workflow":"test-workflow"}
{"level":"info","msg":"task-result changed","namespace":"argo","nodeID":"test-workflow-2831459201","time":"2024-04-16T13:40:57.173Z","workflow":"test-workflow"}
{"level":"info","msg":"node changed","namespace":"argo","new.message":"","new.phase":"Succeeded","new.progress":"0/1","nodeID":"test-workflow-2831459201","old.message":"","old.phase":"Pending","old.progress":"0/1","time":"2024-04-16T13:40:57.173Z","workflow":"test-workflow"}
{"level":"info","msg":"Step group node test-workflow-3484831391 successful","namespace":"argo","time":"2024-04-16T13:40:57.174Z","workflow":"test-workflow"}
{"level":"info","msg":"node test-workflow-3484831391 phase Running -\u003e Succeeded","namespace":"argo","time":"2024-04-16T13:40:57.174Z","workflow":"test-workflow"}
{"level":"info","msg":"node test-workflow-3484831391 finished: 2024-04-16 13:40:57.174058552 +0000 UTC","namespace":"argo","time":"2024-04-16T13:40:57.174Z","workflow":"test-workflow"}
{"level":"info","msg":"Outbound nodes of test-workflow-2831459201 is [test-workflow-2831459201]","namespace":"argo","time":"2024-04-16T13:40:57.174Z","workflow":"test-workflow"}
{"level":"info","msg":"Outbound nodes of test-workflow is [test-workflow-2831459201]","namespace":"argo","time":"2024-04-16T13:40:57.174Z","workflow":"test-workflow"}
{"level":"info","msg":"node test-workflow phase Running -\u003e Succeeded","namespace":"argo","time":"2024-04-16T13:40:57.174Z","workflow":"test-workflow"}
{"level":"info","msg":"node test-workflow finished: 2024-04-16 13:40:57.174184294 +0000 UTC","namespace":"argo","time":"2024-04-16T13:40:57.174Z","workflow":"test-workflow"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"argo","time":"2024-04-16T13:40:57.174Z","workflow":"test-workflow"}
{"level":"info","msg":"reconcileAgentPod","namespace":"argo","time":"2024-04-16T13:40:57.174Z","workflow":"test-workflow"}
{"level":"info","msg":"Updated phase Running -\u003e Succeeded","namespace":"argo","time":"2024-04-16T13:40:57.174Z","workflow":"test-workflow"}
{"level":"info","msg":"Marking workflow completed","namespace":"argo","time":"2024-04-16T13:40:57.174Z","workflow":"test-workflow"}
{"action":"deletePod","key":"argo/test-workflow-1340600742-agent/deletePod","level":"info","msg":"cleaning up pod","time":"2024-04-16T13:40:57.179Z"}
{"level":"info","msg":"Workflow update successful","namespace":"argo","phase":"Succeeded","resourceVersion":"74873926","time":"2024-04-16T13:40:57.267Z","workflow":"test-workflow"}
{"action":"labelPodCompleted","key":"argo/test-workflow-whalesay-template-2831459201/labelPodCompleted","level":"info","msg":"cleaning up pod","time":"2024-04-16T13:40:57.291Z"}

Logs from in your workflow's wait container

{"level":"info","msg":"Starting Workflow Executor","time":"2024-04-16T13:40:48.883Z","version":"v3.5.5"}
{"Duration":1000000000,"Factor":1.6,"Jitter":0.5,"Steps":5,"level":"info","msg":"Using executor retry strategy","time":"2024-04-16T13:40:48.886Z"}
{"deadline":"0001-01-01T00:00:00Z","includeScriptOutput":false,"level":"info","msg":"Executor initialized","namespace":"argo","podName":"test-workflow-whalesay-template-2831459201","templateName":"whalesay-template","time":"2024-04-16T13:40:48.886Z","version":"\u0026Version{Version:v3.5.5,BuildDate:2024-02-29T20:59:20Z,GitCommit:c80b2e91ebd7e7f604e88442f45ec630380effa0,GitTag:v3.5.5,GitTreeState:clean,GoVersion:go1.21.7,Compiler:gc,Platform:linux/amd64,}"}
{"level":"info","msg":"Starting deadline monitor","time":"2024-04-16T13:40:48.905Z"}
{"error":null,"level":"info","msg":"Main container completed","time":"2024-04-16T13:40:50.906Z"}
{"level":"info","msg":"No Script output reference in workflow. Capturing script output ignored","time":"2024-04-16T13:40:50.906Z"}
{"level":"info","msg":"No output parameters","time":"2024-04-16T13:40:50.906Z"}
{"level":"info","msg":"No output artifacts","time":"2024-04-16T13:40:50.906Z"}
{"level":"info","msg":"Alloc=7172 TotalAlloc=13211 Sys=23653 NumGC=4 Goroutines=8","time":"2024-04-16T13:40:50.932Z"}
{"level":"info","msg":"Deadline monitor stopped","time":"2024-04-16T13:40:50.944Z"}

argoproj / argo-workflows