vermaxik commented 1 month ago

Pre-requisites

[X] I have double-checked my configuration
[X] I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
[X] I have searched existing issues and could not find a match for this bug
[ ] I'd like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

After upgrading to 3.5.6 we can see in logs a lot of warning of

{"level":"warning","msg":"Error updating workflow: Operation cannot be fulfilled on workflows.argoproj.io \"workflow-6gndn\": the object has been modified; please apply your changes to the latest version and try again Conflict","namespace":"customer-workflows","time":"2024-05-29T11:47:22.010Z","workflow":"workflow-6gndn"}

The issue was separated from https://github.com/argoproj/argo-workflows/issues/13140

Version

3.5.6

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

parallelism: 50
resourceRateLimit:
  limit: 100
  burst: 3
mainContainer:
  imagePullPolicy: IfNotPresent
  resources:
    limits:
      cpu: 0.3
      memory: 256M
    requests:
      cpu: 0.1
      memory: 128M
executor:
  imagePullPolicy: IfNotPresent
  resources:
    limits:
      cpu: 0.3
      memory: 256M
    requests:
      cpu: 0.1
      memory: 128M
artifactRepository:
  archiveLogs: true
  s3:
    bucket: argo-wf
    endpoint: s3.eu-west-1.amazonaws.com
    keyFormat: "{{workflow.labels.customer-id}}/{{workflow.name}}/{{pod.name}}"
    region: eu-west-1
persistence:
  archive: true
  nodeStatusOffLoad: true
  connectionPool:
    connMaxLifetime: 300s
  postgresql:
    host: argo-rds
    port: 5432
    database: argo_production
    tableName: argo_workflows
    userNameSecret:
      key: username
      name: db-secret
    passwordSecret:
      key: password
      name: db-secret

Logs from the workflow controller

kubectl logs -n argo deploy/argo-workflows-workflow-controller -c controller | grep workflow-c6d2s

{"key":"customer-workflows/workflow-c6d2s","level":"info","msg":"Workflow processing has been postponed due to max parallelism limit","time":"2024-06-05T14:04:46.775Z"}
{"level":"info","msg":"Updated phase  -\u003e Pending","namespace":"customer-workflows","time":"2024-06-05T14:04:46.775Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Updated message  -\u003e Workflow processing has been postponed because too many workflows are already running","namespace":"customer-workflows","time":"2024-06-05T14:04:46.775Z","workflow":"workflow-c6d2s"}
{"level":"warning","msg":"Error updating workflow: Operation cannot be fulfilled on workflows.argoproj.io \"workflow-c6d2s\": the object has been modified; please apply your changes to the latest version and try again Conflict","namespace":"customer-workflows","time":"2024-06-05T14:04:46.787Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Re-applying updates on latest version and retrying update","namespace":"customer-workflows","time":"2024-06-05T14:04:46.787Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Update retry attempt 1 successful","namespace":"customer-workflows","time":"2024-06-05T14:04:46.800Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Workflow update successful","namespace":"customer-workflows","phase":"Pending","resourceVersion":"245969123","time":"2024-06-05T14:04:46.800Z","workflow":"workflow-c6d2s"}
{"key":"customer-workflows/workflow-c6d2s","level":"info","msg":"Workflow processing has been postponed due to max parallelism limit","time":"2024-06-05T14:04:51.785Z"}
{"Phase":"Pending","ResourceVersion":"245969123","level":"info","msg":"Processing workflow","namespace":"customer-workflows","time":"2024-06-05T14:11:46.200Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Task-result reconciliation","namespace":"customer-workflows","numObjs":0,"time":"2024-06-05T14:11:46.206Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Updated phase Pending -\u003e Running","namespace":"customer-workflows","time":"2024-06-05T14:11:46.206Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Updated message Workflow processing has been postponed because too many workflows are already running -\u003e ","namespace":"customer-workflows","time":"2024-06-05T14:11:46.206Z","workflow":"workflow-c6d2s"}
{"level":"warning","msg":"Node was nil, will be initialized as type Skipped","namespace":"customer-workflows","time":"2024-06-05T14:11:46.206Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"was unable to obtain node for , letting display name to be nodeName","namespace":"customer-workflows","time":"2024-06-05T14:11:46.206Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"DAG node workflow-c6d2s initialized Running","namespace":"customer-workflows","time":"2024-06-05T14:11:46.206Z","workflow":"workflow-c6d2s"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-4280457137, taskName task-ec0efb20-7a0f-452f-996e-e8b86bd723ff","time":"2024-06-05T14:11:46.206Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-1995060186, taskName task-15f3d4f5-0e50-4327-83cf-1923b124169a","time":"2024-06-05T14:11:46.206Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-500067944, taskName task-02b979a3-e102-4e26-87ef-91ffe5c799a2","time":"2024-06-05T14:11:46.206Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-500067944, taskName task-02b979a3-e102-4e26-87ef-91ffe5c799a2","time":"2024-06-05T14:11:46.206Z"}
{"level":"info","msg":"All of node workflow-c6d2s.task-02b979a3-e102-4e26-87ef-91ffe5c799a2 dependencies [] completed","namespace":"customer-workflows","time":"2024-06-05T14:11:46.206Z","workflow":"workflow-c6d2s"}
{"level":"warning","msg":"Node was nil, will be initialized as type Skipped","namespace":"customer-workflows","time":"2024-06-05T14:11:46.206Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Pod node workflow-c6d2s-500067944 initialized Pending","namespace":"customer-workflows","time":"2024-06-05T14:11:46.206Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Created pod: workflow-c6d2s.task-02b979a3-e102-4e26-87ef-91ffe5c799a2 (workflow-c6d2s-action-version-7142-task-02b979a3-e102-4e26-87ef-91ffe5c799a2-500067944)","namespace":"customer-workflows","time":"2024-06-05T14:11:46.237Z","workflow":"workflow-c6d2s"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-1995060186, taskName task-15f3d4f5-0e50-4327-83cf-1923b124169a","time":"2024-06-05T14:11:46.237Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-4280457137, taskName task-ec0efb20-7a0f-452f-996e-e8b86bd723ff","time":"2024-06-05T14:11:46.237Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-1995060186, taskName task-15f3d4f5-0e50-4327-83cf-1923b124169a","time":"2024-06-05T14:11:46.237Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-4280457137, taskName task-ec0efb20-7a0f-452f-996e-e8b86bd723ff","time":"2024-06-05T14:11:46.237Z"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"customer-workflows","time":"2024-06-05T14:11:46.237Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"reconcileAgentPod","namespace":"customer-workflows","time":"2024-06-05T14:11:46.237Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Workflow update successful","namespace":"customer-workflows","phase":"Running","resourceVersion":"245995160","time":"2024-06-05T14:11:46.277Z","workflow":"workflow-c6d2s"}
{"Phase":"Running","ResourceVersion":"245995286","level":"info","msg":"Processing workflow","namespace":"customer-workflows","time":"2024-06-05T14:11:51.238Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Task-result reconciliation","namespace":"customer-workflows","numObjs":1,"time":"2024-06-05T14:11:51.240Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-500067944","time":"2024-06-05T14:11:51.240Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"node changed","namespace":"customer-workflows","new.message":"","new.phase":"Running","new.progress":"0/1","nodeID":"workflow-c6d2s-500067944","old.message":"PodInitializing","old.phase":"Pending","old.progress":"0/1","time":"2024-06-05T14:11:51.240Z","workflow":"workflow-c6d2s"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-4280457137, taskName task-ec0efb20-7a0f-452f-996e-e8b86bd723ff","time":"2024-06-05T14:11:51.240Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-1995060186, taskName task-15f3d4f5-0e50-4327-83cf-1923b124169a","time":"2024-06-05T14:11:51.240Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-1995060186, taskName task-15f3d4f5-0e50-4327-83cf-1923b124169a","time":"2024-06-05T14:11:51.240Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-4280457137, taskName task-ec0efb20-7a0f-452f-996e-e8b86bd723ff","time":"2024-06-05T14:11:51.240Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-1995060186, taskName task-15f3d4f5-0e50-4327-83cf-1923b124169a","time":"2024-06-05T14:11:51.240Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-4280457137, taskName task-ec0efb20-7a0f-452f-996e-e8b86bd723ff","time":"2024-06-05T14:11:51.240Z"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"customer-workflows","time":"2024-06-05T14:11:51.240Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"reconcileAgentPod","namespace":"customer-workflows","time":"2024-06-05T14:11:51.240Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Workflow update successful","namespace":"customer-workflows","phase":"Running","resourceVersion":"245995473","time":"2024-06-05T14:11:51.291Z","workflow":"workflow-c6d2s"}
{"Phase":"Running","ResourceVersion":"245995601","level":"info","msg":"Processing workflow","namespace":"customer-workflows","time":"2024-06-05T14:11:56.286Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Task-result reconciliation","namespace":"customer-workflows","numObjs":1,"time":"2024-06-05T14:11:56.287Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-500067944","time":"2024-06-05T14:11:56.287Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"node changed","namespace":"customer-workflows","new.message":"","new.phase":"Succeeded","new.progress":"0/1","nodeID":"workflow-c6d2s-500067944","old.message":"","old.phase":"Running","old.progress":"0/1","time":"2024-06-05T14:11:56.287Z","workflow":"workflow-c6d2s"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-4280457137, taskName task-ec0efb20-7a0f-452f-996e-e8b86bd723ff","time":"2024-06-05T14:11:56.287Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-1995060186, taskName task-15f3d4f5-0e50-4327-83cf-1923b124169a","time":"2024-06-05T14:11:56.288Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-1995060186, taskName task-15f3d4f5-0e50-4327-83cf-1923b124169a","time":"2024-06-05T14:11:56.290Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-1995060186, taskName task-15f3d4f5-0e50-4327-83cf-1923b124169a","time":"2024-06-05T14:11:56.290Z"}
{"level":"info","msg":"All of node workflow-c6d2s.task-15f3d4f5-0e50-4327-83cf-1923b124169a dependencies [task-02b979a3-e102-4e26-87ef-91ffe5c799a2] completed","namespace":"customer-workflows","time":"2024-06-05T14:11:56.290Z","workflow":"workflow-c6d2s"}
{"level":"warning","msg":"Node was nil, will be initialized as type Skipped","namespace":"customer-workflows","time":"2024-06-05T14:11:56.290Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Pod node workflow-c6d2s-1995060186 initialized Pending","namespace":"customer-workflows","time":"2024-06-05T14:11:56.290Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Created pod: workflow-c6d2s.task-15f3d4f5-0e50-4327-83cf-1923b124169a (workflow-c6d2s-action-version-7917-task-15f3d4f5-0e50-4327-83cf-1923b124169a-1995060186)","namespace":"customer-workflows","time":"2024-06-05T14:11:56.318Z","workflow":"workflow-c6d2s"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-4280457137, taskName task-ec0efb20-7a0f-452f-996e-e8b86bd723ff","time":"2024-06-05T14:11:56.318Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-4280457137, taskName task-ec0efb20-7a0f-452f-996e-e8b86bd723ff","time":"2024-06-05T14:11:56.318Z"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"customer-workflows","time":"2024-06-05T14:11:56.318Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"reconcileAgentPod","namespace":"customer-workflows","time":"2024-06-05T14:11:56.318Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Workflow update successful","namespace":"customer-workflows","phase":"Running","resourceVersion":"245995803","time":"2024-06-05T14:11:56.339Z","workflow":"workflow-c6d2s"}
{"action":"deletePod","key":"customer-workflows/workflow-c6d2s-action-version-7142-task-02b979a3-e102-4e26-87ef-91ffe5c799a2-500067944/deletePod","level":"info","msg":"cleaning up pod","time":"2024-06-05T14:11:58.341Z"}
{"Phase":"Running","ResourceVersion":"245995932","level":"info","msg":"Processing workflow","namespace":"customer-workflows","time":"2024-06-05T14:12:01.317Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Task-result reconciliation","namespace":"customer-workflows","numObjs":2,"time":"2024-06-05T14:12:01.319Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-500067944","time":"2024-06-05T14:12:01.319Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-1995060186","time":"2024-06-05T14:12:01.319Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"node changed","namespace":"customer-workflows","new.message":"","new.phase":"Running","new.progress":"0/1","nodeID":"workflow-c6d2s-1995060186","old.message":"PodInitializing","old.phase":"Pending","old.progress":"0/1","time":"2024-06-05T14:12:01.319Z","workflow":"workflow-c6d2s"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-4280457137, taskName task-ec0efb20-7a0f-452f-996e-e8b86bd723ff","time":"2024-06-05T14:12:01.319Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-4280457137, taskName task-ec0efb20-7a0f-452f-996e-e8b86bd723ff","time":"2024-06-05T14:12:01.320Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-4280457137, taskName task-ec0efb20-7a0f-452f-996e-e8b86bd723ff","time":"2024-06-05T14:12:01.320Z"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"customer-workflows","time":"2024-06-05T14:12:01.320Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"reconcileAgentPod","namespace":"customer-workflows","time":"2024-06-05T14:12:01.320Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Workflow update successful","namespace":"customer-workflows","phase":"Running","resourceVersion":"245996166","time":"2024-06-05T14:12:01.333Z","workflow":"workflow-c6d2s"}
{"Phase":"Running","ResourceVersion":"245996323","level":"info","msg":"Processing workflow","namespace":"customer-workflows","time":"2024-06-05T14:12:06.337Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Task-result reconciliation","namespace":"customer-workflows","numObjs":2,"time":"2024-06-05T14:12:06.338Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-500067944","time":"2024-06-05T14:12:06.338Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-1995060186","time":"2024-06-05T14:12:06.338Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"node unchanged","namespace":"customer-workflows","nodeID":"workflow-c6d2s-1995060186","time":"2024-06-05T14:12:06.338Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"node changed","namespace":"customer-workflows","new.message":"PodInitializing","new.phase":"Pending","new.progress":"0/1","nodeID":"workflow-c6d2s-4280457137","old.message":"","old.phase":"Pending","old.progress":"0/1","time":"2024-06-05T14:12:06.338Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"customer-workflows","time":"2024-06-05T14:12:06.339Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"reconcileAgentPod","namespace":"customer-workflows","time":"2024-06-05T14:12:06.339Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Workflow update successful","namespace":"customer-workflows","phase":"Running","resourceVersion":"245996490","time":"2024-06-05T14:12:06.370Z","workflow":"workflow-c6d2s"}
{"action":"deletePod","key":"customer-workflows/workflow-c6d2s-action-version-7917-task-15f3d4f5-0e50-4327-83cf-1923b124169a-1995060186/deletePod","level":"info","msg":"cleaning up pod","time":"2024-06-05T14:12:08.372Z"}
{"Phase":"Running","ResourceVersion":"245996679","level":"info","msg":"Processing workflow","namespace":"customer-workflows","time":"2024-06-05T14:12:11.369Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Task-result reconciliation","namespace":"customer-workflows","numObjs":3,"time":"2024-06-05T14:12:11.371Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-500067944","time":"2024-06-05T14:12:11.371Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-1995060186","time":"2024-06-05T14:12:11.371Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-4280457137","time":"2024-06-05T14:12:11.371Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"node unchanged","namespace":"customer-workflows","nodeID":"workflow-c6d2s-4280457137","time":"2024-06-05T14:12:11.371Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"customer-workflows","time":"2024-06-05T14:12:11.372Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"reconcileAgentPod","namespace":"customer-workflows","time":"2024-06-05T14:12:11.372Z","workflow":"workflow-c6d2s"}
{"action":"terminateContainers","key":"customer-workflows/workflow-c6d2s-action-version-7917-task-ec0efb20-7a0f-452f-996e-e8b86bd723ff-4280457137/terminateContainers","level":"info","msg":"cleaning up pod","time":"2024-06-05T14:12:11.377Z"}
{"level":"info","msg":"Workflow update successful","namespace":"customer-workflows","phase":"Running","resourceVersion":"245996843","time":"2024-06-05T14:12:11.384Z","workflow":"workflow-c6d2s"}
{"Phase":"Running","ResourceVersion":"245997016","level":"info","msg":"Processing workflow","namespace":"customer-workflows","time":"2024-06-05T14:12:16.386Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Task-result reconciliation","namespace":"customer-workflows","numObjs":3,"time":"2024-06-05T14:12:16.387Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-500067944","time":"2024-06-05T14:12:16.387Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-1995060186","time":"2024-06-05T14:12:16.387Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-4280457137","time":"2024-06-05T14:12:16.387Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"node unchanged","namespace":"customer-workflows","nodeID":"workflow-c6d2s-4280457137","time":"2024-06-05T14:12:16.387Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"node changed","namespace":"customer-workflows","new.message":"PodInitializing","new.phase":"Pending","new.progress":"0/1","nodeID":"workflow-c6d2s-2361177406","old.message":"","old.phase":"Pending","old.progress":"0/1","time":"2024-06-05T14:12:16.387Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"customer-workflows","time":"2024-06-05T14:12:16.388Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"reconcileAgentPod","namespace":"customer-workflows","time":"2024-06-05T14:12:16.388Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Running OnExit handler: exit-handler","namespace":"customer-workflows","time":"2024-06-05T14:12:16.388Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Workflow update successful","namespace":"customer-workflows","phase":"Running","resourceVersion":"245997196","time":"2024-06-05T14:12:16.405Z","workflow":"workflow-c6d2s"}
{"action":"deletePod","key":"customer-workflows/workflow-c6d2s-action-version-7917-task-ec0efb20-7a0f-452f-996e-e8b86bd723ff-4280457137/deletePod","level":"info","msg":"cleaning up pod","time":"2024-06-05T14:12:18.411Z"}
{"Phase":"Running","ResourceVersion":"245997390","level":"info","msg":"Processing workflow","namespace":"customer-workflows","time":"2024-06-05T14:12:21.403Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Task-result reconciliation","namespace":"customer-workflows","numObjs":4,"time":"2024-06-05T14:12:21.404Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-500067944","time":"2024-06-05T14:12:21.404Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-1995060186","time":"2024-06-05T14:12:21.404Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-4280457137","time":"2024-06-05T14:12:21.404Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-2361177406","time":"2024-06-05T14:12:21.404Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"node changed","namespace":"customer-workflows","new.message":"","new.phase":"Running","new.progress":"0/1","nodeID":"workflow-c6d2s-2361177406","old.message":"","old.phase":"Running","old.progress":"0/1","time":"2024-06-05T14:12:21.404Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"customer-workflows","time":"2024-06-05T14:12:21.404Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"reconcileAgentPod","namespace":"customer-workflows","time":"2024-06-05T14:12:21.404Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Running OnExit handler: exit-handler","namespace":"customer-workflows","time":"2024-06-05T14:12:21.404Z","workflow":"workflow-c6d2s"}
{"action":"terminateContainers","key":"customer-workflows/workflow-c6d2s-exit-handler-2361177406/terminateContainers","level":"info","msg":"cleaning up pod","time":"2024-06-05T14:12:21.410Z"}
{"level":"info","msg":"https://172.20.0.1:443/api/v1/namespaces/customer-workflows/pods/workflow-c6d2s-exit-handler-2361177406/exec?command=%2Fvar%2Frun%2Fargo%2Fargoexec\u0026command=kill\u0026command=15\u0026command=1\u0026container=wait\u0026stderr=true\u0026stdout=true\u0026tty=false","time":"2024-06-05T14:12:21.410Z"}
{"level":"info","msg":"Workflow update successful","namespace":"customer-workflows","phase":"Running","resourceVersion":"245997561","time":"2024-06-05T14:12:21.423Z","workflow":"workflow-c6d2s"}
{"container":"wait","error":"unable to upgrade connection: container not found (\"wait\")","level":"info","msg":"signaled container","namespace":"customer-workflows","pod":"workflow-c6d2s-exit-handler-2361177406","stderr":null,"stdout":null,"time":"2024-06-05T14:12:21.438Z"}
{"level":"info","msg":"archiving workflow","namespace":"customer-workflows","time":"2024-06-05T14:12:25.441Z","uid":"59086eb4-e239-4e99-ac40-91a4a90f5632","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Queueing Succeeded workflow customer-workflows/workflow-c6d2s for delete in 5m0s due to TTL","time":"2024-06-05T14:12:25.496Z"}
{"action":"killContainers","key":"customer-workflows/workflow-c6d2s-action-version-7917-task-ec0efb20-7a0f-452f-996e-e8b86bd723ff-4280457137/killContainers","level":"info","msg":"cleaning up pod","time":"2024-06-05T14:12:41.377Z"}
{"action":"killContainers","key":"customer-workflows/workflow-c6d2s-exit-handler-2361177406/killContainers","level":"info","msg":"cleaning up pod","time":"2024-06-05T14:12:51.439Z"}
{"level":"info","msg":"Queueing Succeeded workflow customer-workflows/workflow-c6d2s for delete in 2m29s due to TTL","time":"2024-06-05T14:14:56.080Z"}
{"level":"info","msg":"Deleting garbage collected workflow 'customer-workflows/workflow-c6d2s'","time":"2024-06-05T14:17:26.000Z"}
{"level":"info","msg":"Workflow already deleted 'customer-workflows/workflow-c6d2s'","time":"2024-06-05T14:17:26.014Z"}

Logs from in your workflow's wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

tooptoop4 commented 1 month ago

@vermaxik have u tried on v3.5.8?

vermaxik commented 1 month ago

@vermaxik have u tried on v3.5.8?

unfortunately, the issue is still persist in 3.5.8.

tooptoop4 commented 1 month ago

@vermaxik can you change the workflow controller command line to have --loglevel debug and share logs?

Joibel commented 3 weeks ago

Seeing as no-one else is seeing this issue so far I suspect you have some local configuration causing a problem. Your cluster appears quite busy, does this also happen when you're running fewer workflows?

12574 seems the most likely candidate for causing this, but I don't know why you're affected and others aren't, and I'm not sure it's the problem.

reisei commented 3 weeks ago

Hello!

This is logs of the one workflow from ~100 with debug enabled.

{"action":"deletePod","key":"workflows/workflow-21056-k6k42-action-version-7917-task-51e19cea-5aa1-4088-8510-76395472ea67-3451334822/deletePod","level":"info","msg":"cleaning up pod","time":"2024-06-25T14:48:15.878Z"}
{"action":"deletePod","key":"workflows/workflow-21056-k6k42-action-version-7917-task-8c2a4142-1f08-4997-a82e-5edfc2e8d6f0-3182910293/deletePod","level":"info","msg":"cleaning up pod","time":"2024-06-25T14:48:15.878Z"}
{"action":"deletePod","key":"workflows/workflow-21056-k6k42-action-version-7917-task-d6e520c4-8123-4169-b112-2167a2709b12-4278140849/deletePod","level":"info","msg":"cleaning up pod","time":"2024-06-25T14:48:15.878Z"}
{"Phase":"Running","ResourceVersion":"257537302","level":"info","msg":"Processing workflow","namespace":"workflows","time":"2024-06-25T14:48:18.525Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"Task-result reconciliation","namespace":"workflows","numObjs":5,"time":"2024-06-25T14:48:18.538Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result:\n\u0026WorkflowTaskResult{ObjectMeta:{workflow-21056-k6k42-3182910293  workflows  2d8f409a-74e2-4ea4-a5b2-0039bb29783c 257536992 2 2024-06-25 14:48:06 +0000 UTC \u003cnil\u003e \u003cnil\u003e map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:workflow-21056-k6k42] map[] [{argoproj.io/v1alpha1 Workflow workflow-21056-k6k42 30ca6b42-885b-407c-86c1-f89f91d2059c \u003cnil\u003e \u003cnil\u003e}] []  [{argoexec Update argoproj.io/v1alpha1 2024-06-25 14:48:07 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"30ca6b42-885b-407c-86c1-f89f91d2059c\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{},\"f:parameters\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:\u0026Outputs{Parameters:[]Parameter{Parameter{Name:podname,Default:nil,Value:*workflow-21056-k6k42-action-version-7917-task-8c2a4142-1f08-4997-a82e-5edfc2e8d6f0-3182910293,ValueFrom:nil,GlobalName:,Enum:[],Description:nil,},},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:\u0026S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:default/workflow-21056-k6k42/workflow-21056-k6k42-action-version-7917-task-8c2a4142-1f08-4997-a82e-5edfc2e8d6f0-3182910293/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}","namespace":"workflows","time":"2024-06-25T14:48:18.539Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result name:\nworkflow-21056-k6k42-3182910293","namespace":"workflows","time":"2024-06-25T14:48:18.539Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Marking task result complete workflow-21056-k6k42-3182910293","namespace":"workflows","time":"2024-06-25T14:48:18.539Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-21056-k6k42-3182910293","time":"2024-06-25T14:48:18.540Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result:\n\u0026WorkflowTaskResult{ObjectMeta:{workflow-21056-k6k42-1595781171  workflows  7925fcd2-1155-4ab9-bcd2-30903ad93313 257537465 2 2024-06-25 14:48:16 +0000 UTC \u003cnil\u003e \u003cnil\u003e map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:workflow-21056-k6k42] map[] [{argoproj.io/v1alpha1 Workflow workflow-21056-k6k42 30ca6b42-885b-407c-86c1-f89f91d2059c \u003cnil\u003e \u003cnil\u003e}] []  [{argoexec Update argoproj.io/v1alpha1 2024-06-25 14:48:17 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"30ca6b42-885b-407c-86c1-f89f91d2059c\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{},\"f:parameters\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:\u0026Outputs{Parameters:[]Parameter{Parameter{Name:podname,Default:nil,Value:*workflow-21056-k6k42-action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3-1595781171,ValueFrom:nil,GlobalName:,Enum:[],Description:nil,},},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:\u0026S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:default/workflow-21056-k6k42/workflow-21056-k6k42-action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3-1595781171/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}","namespace":"workflows","time":"2024-06-25T14:48:18.540Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result name:\nworkflow-21056-k6k42-1595781171","namespace":"workflows","time":"2024-06-25T14:48:18.540Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Marking task result complete workflow-21056-k6k42-1595781171","namespace":"workflows","time":"2024-06-25T14:48:18.540Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-21056-k6k42-1595781171","time":"2024-06-25T14:48:18.540Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result:\n\u0026WorkflowTaskResult{ObjectMeta:{workflow-21056-k6k42-2728974550  workflows  2b26e8f2-60ec-4a46-9d82-a8135fb1f074 257536296 2 2024-06-25 14:47:56 +0000 UTC \u003cnil\u003e \u003cnil\u003e map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:workflow-21056-k6k42] map[] [{argoproj.io/v1alpha1 Workflow workflow-21056-k6k42 30ca6b42-885b-407c-86c1-f89f91d2059c \u003cnil\u003e \u003cnil\u003e}] []  [{argoexec Update argoproj.io/v1alpha1 2024-06-25 14:47:58 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"30ca6b42-885b-407c-86c1-f89f91d2059c\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{},\"f:parameters\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:\u0026Outputs{Parameters:[]Parameter{Parameter{Name:podname,Default:nil,Value:*workflow-21056-k6k42-action-version-7917-task-df57904b-a342-4892-bb45-5226e7eb3244-2728974550,ValueFrom:nil,GlobalName:,Enum:[],Description:nil,},},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:\u0026S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:default/workflow-21056-k6k42/workflow-21056-k6k42-action-version-7917-task-df57904b-a342-4892-bb45-5226e7eb3244-2728974550/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}","namespace":"workflows","time":"2024-06-25T14:48:18.541Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result name:\nworkflow-21056-k6k42-2728974550","namespace":"workflows","time":"2024-06-25T14:48:18.541Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Marking task result complete workflow-21056-k6k42-2728974550","namespace":"workflows","time":"2024-06-25T14:48:18.541Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-21056-k6k42-2728974550","time":"2024-06-25T14:48:18.541Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result:\n\u0026WorkflowTaskResult{ObjectMeta:{workflow-21056-k6k42-3451334822  workflows  17635c50-a7f1-45a6-b5f8-6b0be6b5e7cb 257537000 2 2024-06-25 14:48:05 +0000 UTC \u003cnil\u003e \u003cnil\u003e map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:workflow-21056-k6k42] map[] [{argoproj.io/v1alpha1 Workflow workflow-21056-k6k42 30ca6b42-885b-407c-86c1-f89f91d2059c \u003cnil\u003e \u003cnil\u003e}] []  [{argoexec Update argoproj.io/v1alpha1 2024-06-25 14:48:07 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"30ca6b42-885b-407c-86c1-f89f91d2059c\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{},\"f:parameters\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:\u0026Outputs{Parameters:[]Parameter{Parameter{Name:podname,Default:nil,Value:*workflow-21056-k6k42-action-version-7917-task-51e19cea-5aa1-4088-8510-76395472ea67-3451334822,ValueFrom:nil,GlobalName:,Enum:[],Description:nil,},},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:\u0026S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:default/workflow-21056-k6k42/workflow-21056-k6k42-action-version-7917-task-51e19cea-5aa1-4088-8510-76395472ea67-3451334822/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}","namespace":"workflows","time":"2024-06-25T14:48:18.541Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result name:\nworkflow-21056-k6k42-3451334822","namespace":"workflows","time":"2024-06-25T14:48:18.541Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Marking task result complete workflow-21056-k6k42-3451334822","namespace":"workflows","time":"2024-06-25T14:48:18.541Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-21056-k6k42-3451334822","time":"2024-06-25T14:48:18.541Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result:\n\u0026WorkflowTaskResult{ObjectMeta:{workflow-21056-k6k42-4278140849  workflows  aa6afaa7-8f90-456b-a5c7-b4430bda2258 257537003 2 2024-06-25 14:48:05 +0000 UTC \u003cnil\u003e \u003cnil\u003e map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:workflow-21056-k6k42] map[] [{argoproj.io/v1alpha1 Workflow workflow-21056-k6k42 30ca6b42-885b-407c-86c1-f89f91d2059c \u003cnil\u003e \u003cnil\u003e}] []  [{argoexec Update argoproj.io/v1alpha1 2024-06-25 14:48:07 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"30ca6b42-885b-407c-86c1-f89f91d2059c\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{},\"f:parameters\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:\u0026Outputs{Parameters:[]Parameter{Parameter{Name:podname,Default:nil,Value:*workflow-21056-k6k42-action-version-7917-task-d6e520c4-8123-4169-b112-2167a2709b12-4278140849,ValueFrom:nil,GlobalName:,Enum:[],Description:nil,},},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:\u0026S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:default/workflow-21056-k6k42/workflow-21056-k6k42-action-version-7917-task-d6e520c4-8123-4169-b112-2167a2709b12-4278140849/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}","namespace":"workflows","time":"2024-06-25T14:48:18.542Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result name:\nworkflow-21056-k6k42-4278140849","namespace":"workflows","time":"2024-06-25T14:48:18.542Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Marking task result complete workflow-21056-k6k42-4278140849","namespace":"workflows","time":"2024-06-25T14:48:18.542Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-21056-k6k42-4278140849","time":"2024-06-25T14:48:18.542Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Skipping artifact GC","namespace":"workflows","time":"2024-06-25T14:48:18.542Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"node changed","namespace":"workflows","new.message":"","new.phase":"Running","new.progress":"0/1","nodeID":"workflow-21056-k6k42-1595781171","old.message":"","old.phase":"Pending","old.progress":"0/1","time":"2024-06-25T14:48:18.543Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Evaluating node workflow-21056-k6k42: template: *v1alpha1.WorkflowStep (root), boundaryID: ","namespace":"workflows","time":"2024-06-25T14:48:18.543Z","workflow":"workflow-21056-k6k42"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Resolving the template","time":"2024-06-25T14:48:18.543Z","tmpl":"*v1alpha1.WorkflowStep (root)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template","time":"2024-06-25T14:48:18.543Z","tmpl":"*v1alpha1.WorkflowStep (root)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template by name: root","time":"2024-06-25T14:48:18.543Z","tmpl":"*v1alpha1.WorkflowStep (root)"}
{"level":"debug","msg":"Executing node workflow-21056-k6k42 of DAG is Running","namespace":"workflows","time":"2024-06-25T14:48:18.544Z","workflow":"workflow-21056-k6k42"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Resolving the template","time":"2024-06-25T14:48:18.544Z","tmpl":"*v1alpha1.DAGTask (action-version-7917-task-51e19cea-5aa1-4088-8510-76395472ea67)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template","time":"2024-06-25T14:48:18.544Z","tmpl":"*v1alpha1.DAGTask (action-version-7917-task-51e19cea-5aa1-4088-8510-76395472ea67)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template by name: action-version-7917-task-51e19cea-5aa1-4088-8510-76395472ea67","time":"2024-06-25T14:48:18.544Z","tmpl":"*v1alpha1.DAGTask (action-version-7917-task-51e19cea-5aa1-4088-8510-76395472ea67)"}
{"level":"debug","msg":"Evaluating node workflow-21056-k6k42.task-8214a347-4a77-43a2-996e-57e4ab9e3da3: template: *v1alpha1.DAGTask (action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3), boundaryID: workflow-21056-k6k42","namespace":"workflows","time":"2024-06-25T14:48:18.545Z","workflow":"workflow-21056-k6k42"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Resolving the template","time":"2024-06-25T14:48:18.545Z","tmpl":"*v1alpha1.DAGTask (action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template","time":"2024-06-25T14:48:18.545Z","tmpl":"*v1alpha1.DAGTask (action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template by name: action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3","time":"2024-06-25T14:48:18.545Z","tmpl":"*v1alpha1.DAGTask (action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3)"}
{"level":"debug","msg":"Executing node workflow-21056-k6k42.task-8214a347-4a77-43a2-996e-57e4ab9e3da3 of Pod is Running","namespace":"workflows","time":"2024-06-25T14:48:18.546Z","workflow":"workflow-21056-k6k42"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Resolving the template","time":"2024-06-25T14:48:18.546Z","tmpl":"*v1alpha1.NodeStatus (root)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template","time":"2024-06-25T14:48:18.547Z","tmpl":"*v1alpha1.NodeStatus (root)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template by name: root","time":"2024-06-25T14:48:18.547Z","tmpl":"*v1alpha1.NodeStatus (root)"}
{"action":"terminateContainers","key":"workflows/workflow-21056-k6k42-action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3-1595781171/terminateContainers","level":"info","msg":"cleaning up pod","time":"2024-06-25T14:48:18.582Z"}
{"level":"debug","msg":"Executing node workflow-21056-k6k42.task-8214a347-4a77-43a2-996e-57e4ab9e3da3 with container template: action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3\n","namespace":"workflows","time":"2024-06-25T14:48:18.586Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Skipped pod workflow-21056-k6k42.task-8214a347-4a77-43a2-996e-57e4ab9e3da3 (workflow-21056-k6k42-1595781171) creation: already exists","namespace":"workflows","podPhase":"Running","time":"2024-06-25T14:48:18.587Z","workflow":"workflow-21056-k6k42"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Resolving the template","time":"2024-06-25T14:48:18.587Z","tmpl":"*v1alpha1.DAGTask (action-version-7917-task-8c2a4142-1f08-4997-a82e-5edfc2e8d6f0)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template","time":"2024-06-25T14:48:18.587Z","tmpl":"*v1alpha1.DAGTask (action-version-7917-task-8c2a4142-1f08-4997-a82e-5edfc2e8d6f0)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template by name: action-version-7917-task-8c2a4142-1f08-4997-a82e-5edfc2e8d6f0","time":"2024-06-25T14:48:18.587Z","tmpl":"*v1alpha1.DAGTask (action-version-7917-task-8c2a4142-1f08-4997-a82e-5edfc2e8d6f0)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Resolving the template","time":"2024-06-25T14:48:18.589Z","tmpl":"*v1alpha1.DAGTask (action-version-7917-task-d6e520c4-8123-4169-b112-2167a2709b12)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template","time":"2024-06-25T14:48:18.590Z","tmpl":"*v1alpha1.DAGTask (action-version-7917-task-d6e520c4-8123-4169-b112-2167a2709b12)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template by name: action-version-7917-task-d6e520c4-8123-4169-b112-2167a2709b12","time":"2024-06-25T14:48:18.590Z","tmpl":"*v1alpha1.DAGTask (action-version-7917-task-d6e520c4-8123-4169-b112-2167a2709b12)"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"workflows","time":"2024-06-25T14:48:18.599Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"reconcileAgentPod","namespace":"workflows","time":"2024-06-25T14:48:18.599Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Log changes patch: {\"status\":{\"conditions\":[{\"status\":\"True\",\"type\":\"PodRunning\"}],\"nodes\":{\"workflow-21056-k6k42-1595781171\":{\"hostNodeName\":\"ip-10-13-167-21.eu-west-1.compute.internal\",\"outputs\":{\"artifacts\":[{\"name\":\"main-logs\",\"s3\":{\"key\":\"default/workflow-21056-k6k42/workflow-21056-k6k42-action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3-1595781171/main.log\"}}],\"exitCode\":\"0\",\"parameters\":[{\"name\":\"podname\",\"value\":\"workflow-21056-k6k42-action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3-1595781171\"}]},\"phase\":\"Running\"}},\"taskResultsCompletionStatus\":{\"workflow-21056-k6k42-1595781171\":true}}}","time":"2024-06-25T14:48:18.604Z"}
{"level":"warning","msg":"Error updating workflow: Operation cannot be fulfilled on workflows.argoproj.io \"workflow-21056-k6k42\": the object has been modified; please apply your changes to the latest version and try again Conflict","namespace":"workflows","time":"2024-06-25T14:48:18.616Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"Re-applying updates on latest version and retrying update","namespace":"workflows","time":"2024-06-25T14:48:18.616Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"Update retry attempt 1 successful","namespace":"workflows","time":"2024-06-25T14:48:18.676Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"Workflow update successful","namespace":"workflows","phase":"Running","resourceVersion":"257537489","time":"2024-06-25T14:48:18.676Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"workflows\", Name:\"workflow-21056-k6k42\", UID:\"30ca6b42-885b-407c-86c1-f89f91d2059c\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"257537489\", FieldPath:\"\"}): type: 'Normal' reason: 'WorkflowNodeRunning' Running node workflow-21056-k6k42.task-8214a347-4a77-43a2-996e-57e4ab9e3da3","time":"2024-06-25T14:48:18.676Z"}
{"Phase":"Running","ResourceVersion":"257537489","level":"info","msg":"Processing workflow","namespace":"workflows","time":"2024-06-25T14:48:23.610Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"Task-result reconciliation","namespace":"workflows","numObjs":5,"time":"2024-06-25T14:48:23.612Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result:\n\u0026WorkflowTaskResult{ObjectMeta:{workflow-21056-k6k42-2728974550  workflows  2b26e8f2-60ec-4a46-9d82-a8135fb1f074 257536296 2 2024-06-25 14:47:56 +0000 UTC \u003cnil\u003e \u003cnil\u003e map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:workflow-21056-k6k42] map[] [{argoproj.io/v1alpha1 Workflow workflow-21056-k6k42 30ca6b42-885b-407c-86c1-f89f91d2059c \u003cnil\u003e \u003cnil\u003e}] []  [{argoexec Update argoproj.io/v1alpha1 2024-06-25 14:47:58 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"30ca6b42-885b-407c-86c1-f89f91d2059c\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{},\"f:parameters\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:\u0026Outputs{Parameters:[]Parameter{Parameter{Name:podname,Default:nil,Value:*workflow-21056-k6k42-action-version-7917-task-df57904b-a342-4892-bb45-5226e7eb3244-2728974550,ValueFrom:nil,GlobalName:,Enum:[],Description:nil,},},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:\u0026S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:default/workflow-21056-k6k42/workflow-21056-k6k42-action-version-7917-task-df57904b-a342-4892-bb45-5226e7eb3244-2728974550/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}","namespace":"workflows","time":"2024-06-25T14:48:23.612Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result name:\nworkflow-21056-k6k42-2728974550","namespace":"workflows","time":"2024-06-25T14:48:23.612Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Marking task result complete workflow-21056-k6k42-2728974550","namespace":"workflows","time":"2024-06-25T14:48:23.612Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-21056-k6k42-2728974550","time":"2024-06-25T14:48:23.612Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result:\n\u0026WorkflowTaskResult{ObjectMeta:{workflow-21056-k6k42-3451334822  workflows  17635c50-a7f1-45a6-b5f8-6b0be6b5e7cb 257537000 2 2024-06-25 14:48:05 +0000 UTC \u003cnil\u003e \u003cnil\u003e map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:workflow-21056-k6k42] map[] [{argoproj.io/v1alpha1 Workflow workflow-21056-k6k42 30ca6b42-885b-407c-86c1-f89f91d2059c \u003cnil\u003e \u003cnil\u003e}] []  [{argoexec Update argoproj.io/v1alpha1 2024-06-25 14:48:07 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"30ca6b42-885b-407c-86c1-f89f91d2059c\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{},\"f:parameters\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:\u0026Outputs{Parameters:[]Parameter{Parameter{Name:podname,Default:nil,Value:*workflow-21056-k6k42-action-version-7917-task-51e19cea-5aa1-4088-8510-76395472ea67-3451334822,ValueFrom:nil,GlobalName:,Enum:[],Description:nil,},},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:\u0026S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:default/workflow-21056-k6k42/workflow-21056-k6k42-action-version-7917-task-51e19cea-5aa1-4088-8510-76395472ea67-3451334822/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}","namespace":"workflows","time":"2024-06-25T14:48:23.612Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result name:\nworkflow-21056-k6k42-3451334822","namespace":"workflows","time":"2024-06-25T14:48:23.612Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Marking task result complete workflow-21056-k6k42-3451334822","namespace":"workflows","time":"2024-06-25T14:48:23.612Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-21056-k6k42-3451334822","time":"2024-06-25T14:48:23.612Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result:\n\u0026WorkflowTaskResult{ObjectMeta:{workflow-21056-k6k42-4278140849  workflows  aa6afaa7-8f90-456b-a5c7-b4430bda2258 257537003 2 2024-06-25 14:48:05 +0000 UTC \u003cnil\u003e \u003cnil\u003e map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:workflow-21056-k6k42] map[] [{argoproj.io/v1alpha1 Workflow workflow-21056-k6k42 30ca6b42-885b-407c-86c1-f89f91d2059c \u003cnil\u003e \u003cnil\u003e}] []  [{argoexec Update argoproj.io/v1alpha1 2024-06-25 14:48:07 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"30ca6b42-885b-407c-86c1-f89f91d2059c\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{},\"f:parameters\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:\u0026Outputs{Parameters:[]Parameter{Parameter{Name:podname,Default:nil,Value:*workflow-21056-k6k42-action-version-7917-task-d6e520c4-8123-4169-b112-2167a2709b12-4278140849,ValueFrom:nil,GlobalName:,Enum:[],Description:nil,},},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:\u0026S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:default/workflow-21056-k6k42/workflow-21056-k6k42-action-version-7917-task-d6e520c4-8123-4169-b112-2167a2709b12-4278140849/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}","namespace":"workflows","time":"2024-06-25T14:48:23.612Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result name:\nworkflow-21056-k6k42-4278140849","namespace":"workflows","time":"2024-06-25T14:48:23.612Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Marking task result complete workflow-21056-k6k42-4278140849","namespace":"workflows","time":"2024-06-25T14:48:23.612Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-21056-k6k42-4278140849","time":"2024-06-25T14:48:23.612Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result:\n\u0026WorkflowTaskResult{ObjectMeta:{workflow-21056-k6k42-3182910293  workflows  2d8f409a-74e2-4ea4-a5b2-0039bb29783c 257536992 2 2024-06-25 14:48:06 +0000 UTC \u003cnil\u003e \u003cnil\u003e map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:workflow-21056-k6k42] map[] [{argoproj.io/v1alpha1 Workflow workflow-21056-k6k42 30ca6b42-885b-407c-86c1-f89f91d2059c \u003cnil\u003e \u003cnil\u003e}] []  [{argoexec Update argoproj.io/v1alpha1 2024-06-25 14:48:07 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"30ca6b42-885b-407c-86c1-f89f91d2059c\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{},\"f:parameters\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:\u0026Outputs{Parameters:[]Parameter{Parameter{Name:podname,Default:nil,Value:*workflow-21056-k6k42-action-version-7917-task-8c2a4142-1f08-4997-a82e-5edfc2e8d6f0-3182910293,ValueFrom:nil,GlobalName:,Enum:[],Description:nil,},},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:\u0026S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:default/workflow-21056-k6k42/workflow-21056-k6k42-action-version-7917-task-8c2a4142-1f08-4997-a82e-5edfc2e8d6f0-3182910293/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}","namespace":"workflows","time":"2024-06-25T14:48:23.613Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result name:\nworkflow-21056-k6k42-3182910293","namespace":"workflows","time":"2024-06-25T14:48:23.613Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Marking task result complete workflow-21056-k6k42-3182910293","namespace":"workflows","time":"2024-06-25T14:48:23.613Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-21056-k6k42-3182910293","time":"2024-06-25T14:48:23.613Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result:\n\u0026WorkflowTaskResult{ObjectMeta:{workflow-21056-k6k42-1595781171  workflows  7925fcd2-1155-4ab9-bcd2-30903ad93313 257537465 2 2024-06-25 14:48:16 +0000 UTC \u003cnil\u003e \u003cnil\u003e map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:workflow-21056-k6k42] map[] [{argoproj.io/v1alpha1 Workflow workflow-21056-k6k42 30ca6b42-885b-407c-86c1-f89f91d2059c \u003cnil\u003e \u003cnil\u003e}] []  [{argoexec Update argoproj.io/v1alpha1 2024-06-25 14:48:17 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"30ca6b42-885b-407c-86c1-f89f91d2059c\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{},\"f:parameters\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:\u0026Outputs{Parameters:[]Parameter{Parameter{Name:podname,Default:nil,Value:*workflow-21056-k6k42-action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3-1595781171,ValueFrom:nil,GlobalName:,Enum:[],Description:nil,},},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:\u0026S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:default/workflow-21056-k6k42/workflow-21056-k6k42-action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3-1595781171/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}","namespace":"workflows","time":"2024-06-25T14:48:23.613Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result name:\nworkflow-21056-k6k42-1595781171","namespace":"workflows","time":"2024-06-25T14:48:23.613Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Marking task result complete workflow-21056-k6k42-1595781171","namespace":"workflows","time":"2024-06-25T14:48:23.613Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-21056-k6k42-1595781171","time":"2024-06-25T14:48:23.613Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Skipping artifact GC","namespace":"workflows","time":"2024-06-25T14:48:23.613Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"node changed","namespace":"workflows","new.message":"","new.phase":"Succeeded","new.progress":"0/1","nodeID":"workflow-21056-k6k42-1595781171","old.message":"","old.phase":"Running","old.progress":"0/1","time":"2024-06-25T14:48:23.613Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Evaluating node workflow-21056-k6k42: template: *v1alpha1.WorkflowStep (root), boundaryID: ","namespace":"workflows","time":"2024-06-25T14:48:23.613Z","workflow":"workflow-21056-k6k42"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Resolving the template","time":"2024-06-25T14:48:23.613Z","tmpl":"*v1alpha1.WorkflowStep (root)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template","time":"2024-06-25T14:48:23.613Z","tmpl":"*v1alpha1.WorkflowStep (root)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template by name: root","time":"2024-06-25T14:48:23.613Z","tmpl":"*v1alpha1.WorkflowStep (root)"}
{"level":"debug","msg":"Executing node workflow-21056-k6k42 of DAG is Running","namespace":"workflows","time":"2024-06-25T14:48:23.614Z","workflow":"workflow-21056-k6k42"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Resolving the template","time":"2024-06-25T14:48:23.614Z","tmpl":"*v1alpha1.DAGTask (action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template","time":"2024-06-25T14:48:23.614Z","tmpl":"*v1alpha1.DAGTask (action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template by name: action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3","time":"2024-06-25T14:48:23.614Z","tmpl":"*v1alpha1.DAGTask (action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Resolving the template","time":"2024-06-25T14:48:23.614Z","tmpl":"*v1alpha1.DAGTask (action-version-7917-task-8c2a4142-1f08-4997-a82e-5edfc2e8d6f0)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template","time":"2024-06-25T14:48:23.614Z","tmpl":"*v1alpha1.DAGTask (action-version-7917-task-8c2a4142-1f08-4997-a82e-5edfc2e8d6f0)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template by name: action-version-7917-task-8c2a4142-1f08-4997-a82e-5edfc2e8d6f0","time":"2024-06-25T14:48:23.614Z","tmpl":"*v1alpha1.DAGTask (action-version-7917-task-8c2a4142-1f08-4997-a82e-5edfc2e8d6f0)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Resolving the template","time":"2024-06-25T14:48:23.614Z","tmpl":"*v1alpha1.DAGTask (action-version-7917-task-d6e520c4-8123-4169-b112-2167a2709b12)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template","time":"2024-06-25T14:48:23.614Z","tmpl":"*v1alpha1.DAGTask (action-version-7917-task-d6e520c4-8123-4169-b112-2167a2709b12)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template by name: action-version-7917-task-d6e520c4-8123-4169-b112-2167a2709b12","time":"2024-06-25T14:48:23.615Z","tmpl":"*v1alpha1.DAGTask (action-version-7917-task-d6e520c4-8123-4169-b112-2167a2709b12)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Resolving the template","time":"2024-06-25T14:48:23.615Z","tmpl":"*v1alpha1.NodeStatus (action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template","time":"2024-06-25T14:48:23.615Z","tmpl":"*v1alpha1.NodeStatus (action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template by name: action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3","time":"2024-06-25T14:48:23.615Z","tmpl":"*v1alpha1.NodeStatus (action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Resolving the template","time":"2024-06-25T14:48:23.615Z","tmpl":"*v1alpha1.NodeStatus (action-version-7917-task-8c2a4142-1f08-4997-a82e-5edfc2e8d6f0)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template","time":"2024-06-25T14:48:23.615Z","tmpl":"*v1alpha1.NodeStatus (action-version-7917-task-8c2a4142-1f08-4997-a82e-5edfc2e8d6f0)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template by name: action-version-7917-task-8c2a4142-1f08-4997-a82e-5edfc2e8d6f0","time":"2024-06-25T14:48:23.615Z","tmpl":"*v1alpha1.NodeStatus (action-version-7917-task-8c2a4142-1f08-4997-a82e-5edfc2e8d6f0)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Resolving the template","time":"2024-06-25T14:48:23.615Z","tmpl":"*v1alpha1.NodeStatus (action-version-7917-task-d6e520c4-8123-4169-b112-2167a2709b12)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template","time":"2024-06-25T14:48:23.615Z","tmpl":"*v1alpha1.NodeStatus (action-version-7917-task-d6e520c4-8123-4169-b112-2167a2709b12)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template by name: action-version-7917-task-d6e520c4-8123-4169-b112-2167a2709b12","time":"2024-06-25T14:48:23.615Z","tmpl":"*v1alpha1.NodeStatus (action-version-7917-task-d6e520c4-8123-4169-b112-2167a2709b12)"}
{"level":"info","msg":"Outbound nodes of workflow-21056-k6k42 set to [workflow-21056-k6k42-1595781171 workflow-21056-k6k42-3182910293 workflow-21056-k6k42-4278140849]","namespace":"workflows","time":"2024-06-25T14:48:23.615Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"node workflow-21056-k6k42 phase Running -\u003e Succeeded","namespace":"workflows","time":"2024-06-25T14:48:23.615Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"node workflow-21056-k6k42 finished: 2024-06-25 14:48:23.615770727 +0000 UTC","namespace":"workflows","time":"2024-06-25T14:48:23.615Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Checking daemoned children of workflow-21056-k6k42","namespace":"workflows","time":"2024-06-25T14:48:23.615Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"workflows","time":"2024-06-25T14:48:23.615Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"reconcileAgentPod","namespace":"workflows","time":"2024-06-25T14:48:23.615Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"Running OnExit handler: exit-handler","namespace":"workflows","time":"2024-06-25T14:48:23.615Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Evaluating node workflow-21056-k6k42.onExit: template: *v1alpha1.WorkflowStep (exit-handler), boundaryID: ","namespace":"workflows","time":"2024-06-25T14:48:23.615Z","workflow":"workflow-21056-k6k42"}
{"level":"warning","msg":"Node was nil, will be initialized as type Skipped","namespace":"workflows","time":"2024-06-25T14:48:23.616Z","workflow":"workflow-21056-k6k42"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Resolving the template","time":"2024-06-25T14:48:23.616Z","tmpl":"*v1alpha1.WorkflowStep (exit-handler)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template","time":"2024-06-25T14:48:23.616Z","tmpl":"*v1alpha1.WorkflowStep (exit-handler)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template by name: exit-handler","time":"2024-06-25T14:48:23.616Z","tmpl":"*v1alpha1.WorkflowStep (exit-handler)"}
{"level":"debug","msg":"Initializing node workflow-21056-k6k42.onExit: template: *v1alpha1.WorkflowStep (exit-handler), boundaryID: ","namespace":"workflows","time":"2024-06-25T14:48:23.616Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"was unable to obtain node for , letting display name to be nodeName","namespace":"workflows","time":"2024-06-25T14:48:23.616Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"Pod node workflow-21056-k6k42-1349795579 initialized Pending","namespace":"workflows","time":"2024-06-25T14:48:23.616Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Executing node workflow-21056-k6k42.onExit with container template: exit-handler\n","namespace":"workflows","time":"2024-06-25T14:48:23.616Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"","namespace":"workflows","needLocation":true,"time":"2024-06-25T14:48:23.616Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Creating Pod: workflow-21056-k6k42.onExit (workflow-21056-k6k42-exit-handler-1349795579)","namespace":"workflows","time":"2024-06-25T14:48:23.620Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"Created pod: workflow-21056-k6k42.onExit (workflow-21056-k6k42-exit-handler-1349795579)","namespace":"workflows","time":"2024-06-25T14:48:23.654Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Log changes patch: {\"status\":{\"conditions\":[{\"status\":\"False\",\"type\":\"PodRunning\"}],\"nodes\":{\"workflow-21056-k6k42\":{\"finishedAt\":\"2024-06-25T14:48:23Z\",\"outboundNodes\":[\"workflow-21056-k6k42-1595781171\",\"workflow-21056-k6k42-3182910293\",\"workflow-21056-k6k42-4278140849\"],\"phase\":\"Succeeded\"},\"workflow-21056-k6k42-1349795579\":{\"displayName\":\"workflow-21056-k6k42.onExit\",\"finishedAt\":null,\"id\":\"workflow-21056-k6k42-1349795579\",\"name\":\"workflow-21056-k6k42.onExit\",\"nodeFlag\":{\"hooked\":true},\"phase\":\"Pending\",\"startedAt\":\"2024-06-25T14:48:23Z\",\"templateName\":\"exit-handler\",\"templateScope\":\"local/workflow-21056-k6k42\",\"type\":\"Pod\"},\"workflow-21056-k6k42-1595781171\":{\"finishedAt\":\"2024-06-25T14:48:17Z\",\"phase\":\"Succeeded\",\"resourcesDuration\":{\"cpu\":0,\"memory\":2}}}}}","time":"2024-06-25T14:48:23.655Z"}
{"level":"info","msg":"Workflow update successful","namespace":"workflows","phase":"Running","resourceVersion":"257537587","time":"2024-06-25T14:48:23.696Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"workflows\", Name:\"workflow-21056-k6k42\", UID:\"30ca6b42-885b-407c-86c1-f89f91d2059c\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"257537587\", FieldPath:\"\"}): type: 'Normal' reason: 'WorkflowNodeSucceeded' Succeeded node workflow-21056-k6k42.task-8214a347-4a77-43a2-996e-57e4ab9e3da3","time":"2024-06-25T14:48:23.696Z"}
{"level":"debug","msg":"Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"workflows\", Name:\"workflow-21056-k6k42\", UID:\"30ca6b42-885b-407c-86c1-f89f91d2059c\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"257537587\", FieldPath:\"\"}): type: 'Normal' reason: 'WorkflowNodeSucceeded' Succeeded node workflow-21056-k6k42","time":"2024-06-25T14:48:23.698Z"}
{"action":"deletePod","key":"workflows/workflow-21056-k6k42-action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3-1595781171/deletePod","level":"info","msg":"cleaning up pod","time":"2024-06-25T14:48:25.698Z"}
{"action":"killContainers","key":"workflows/workflow-21056-k6k42-action-version-7917-task-df57904b-a342-4892-bb45-5226e7eb3244-2728974550/killContainers","level":"info","msg":"cleaning up pod","time":"2024-06-25T14:48:28.615Z"}
{"Phase":"Running","ResourceVersion":"257537600","level":"info","msg":"Processing workflow","namespace":"workflows","time":"2024-06-25T14:48:28.663Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"Task-result reconciliation","namespace":"workflows","numObjs":6,"time":"2024-06-25T14:48:28.682Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result:\n\u0026WorkflowTaskResult{ObjectMeta:{workflow-21056-k6k42-2728974550  workflows  2b26e8f2-60ec-4a46-9d82-a8135fb1f074 257536296 2 2024-06-25 14:47:56 +0000 UTC \u003cnil\u003e \u003cnil\u003e map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:workflow-21056-k6k42] map[] [{argoproj.io/v1alpha1 Workflow workflow-21056-k6k42 30ca6b42-885b-407c-86c1-f89f91d2059c \u003cnil\u003e \u003cnil\u003e}] []  [{argoexec Update argoproj.io/v1alpha1 2024-06-25 14:47:58 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"30ca6b42-885b-407c-86c1-f89f91d2059c\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{},\"f:parameters\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:\u0026Outputs{Parameters:[]Parameter{Parameter{Name:podname,Default:nil,Value:*workflow-21056-k6k42-action-version-7917-task-df57904b-a342-4892-bb45-5226e7eb3244-2728974550,ValueFrom:nil,GlobalName:,Enum:[],Description:nil,},},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:\u0026S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:default/workflow-21056-k6k42/workflow-21056-k6k42-action-version-7917-task-df57904b-a342-4892-bb45-5226e7eb3244-2728974550/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}","namespace":"workflows","time":"2024-06-25T14:48:28.682Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result name:\nworkflow-21056-k6k42-2728974550","namespace":"workflows","time":"2024-06-25T14:48:28.685Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Marking task result complete workflow-21056-k6k42-2728974550","namespace":"workflows","time":"2024-06-25T14:48:28.685Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-21056-k6k42-2728974550","time":"2024-06-25T14:48:28.693Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result:\n\u0026WorkflowTaskResult{ObjectMeta:{workflow-21056-k6k42-3451334822  workflows  17635c50-a7f1-45a6-b5f8-6b0be6b5e7cb 257537000 2 2024-06-25 14:48:05 +0000 UTC \u003cnil\u003e \u003cnil\u003e map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:workflow-21056-k6k42] map[] [{argoproj.io/v1alpha1 Workflow workflow-21056-k6k42 30ca6b42-885b-407c-86c1-f89f91d2059c \u003cnil\u003e \u003cnil\u003e}] []  [{argoexec Update argoproj.io/v1alpha1 2024-06-25 14:48:07 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"30ca6b42-885b-407c-86c1-f89f91d2059c\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{},\"f:parameters\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:\u0026Outputs{Parameters:[]Parameter{Parameter{Name:podname,Default:nil,Value:*workflow-21056-k6k42-action-version-7917-task-51e19cea-5aa1-4088-8510-76395472ea67-3451334822,ValueFrom:nil,GlobalName:,Enum:[],Description:nil,},},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:\u0026S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:default/workflow-21056-k6k42/workflow-21056-k6k42-action-version-7917-task-51e19cea-5aa1-4088-8510-76395472ea67-3451334822/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}","namespace":"workflows","time":"2024-06-25T14:48:28.694Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result name:\nworkflow-21056-k6k42-3451334822","namespace":"workflows","time":"2024-06-25T14:48:28.694Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Marking task result complete workflow-21056-k6k42-3451334822","namespace":"workflows","time":"2024-06-25T14:48:28.694Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-21056-k6k42-3451334822","time":"2024-06-25T14:48:28.694Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result:\n\u0026WorkflowTaskResult{ObjectMeta:{workflow-21056-k6k42-4278140849  workflows  aa6afaa7-8f90-456b-a5c7-b4430bda2258 257537003 2 2024-06-25 14:48:05 +0000 UTC \u003cnil\u003e \u003cnil\u003e map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:workflow-21056-k6k42] map[] [{argoproj.io/v1alpha1 Workflow workflow-21056-k6k42 30ca6b42-885b-407c-86c1-f89f91d2059c \u003cnil\u003e \u003cnil\u003e}] []  [{argoexec Update argoproj.io/v1alpha1 2024-06-25 14:48:07 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"30ca6b42-885b-407c-86c1-f89f91d2059c\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{},\"f:parameters\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:\u0026Outputs{Parameters:[]Parameter{Parameter{Name:podname,Default:nil,Value:*workflow-21056-k6k42-action-version-7917-task-d6e520c4-8123-4169-b112-2167a2709b12-4278140849,ValueFrom:nil,GlobalName:,Enum:[],Description:nil,},},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:\u0026S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:default/workflow-21056-k6k42/workflow-21056-k6k42-action-version-7917-task-d6e520c4-8123-4169-b112-2167a2709b12-4278140849/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}","namespace":"workflows","time":"2024-06-25T14:48:28.694Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result name:\nworkflow-21056-k6k42-4278140849","namespace":"workflows","time":"2024-06-25T14:48:28.694Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Marking task result complete workflow-21056-k6k42-4278140849","namespace":"workflows","time":"2024-06-25T14:48:28.694Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-21056-k6k42-4278140849","time":"2024-06-25T14:48:28.694Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result:\n\u0026WorkflowTaskResult{ObjectMeta:{workflow-21056-k6k42-3182910293  workflows  2d8f409a-74e2-4ea4-a5b2-0039bb29783c 257536992 2 2024-06-25 14:48:06 +0000 UTC \u003cnil\u003e \u003cnil\u003e map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:workflow-21056-k6k42] map[] [{argoproj.io/v1alpha1 Workflow workflow-21056-k6k42 30ca6b42-885b-407c-86c1-f89f91d2059c \u003cnil\u003e \u003cnil\u003e}] []  [{argoexec Update argoproj.io/v1alpha1 2024-06-25 14:48:07 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"30ca6b42-885b-407c-86c1-f89f91d2059c\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{},\"f:parameters\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:\u0026Outputs{Parameters:[]Parameter{Parameter{Name:podname,Default:nil,Value:*workflow-21056-k6k42-action-version-7917-task-8c2a4142-1f08-4997-a82e-5edfc2e8d6f0-3182910293,ValueFrom:nil,GlobalName:,Enum:[],Description:nil,},},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:\u0026S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:default/workflow-21056-k6k42/workflow-21056-k6k42-action-version-7917-task-8c2a4142-1f08-4997-a82e-5edfc2e8d6f0-3182910293/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}","namespace":"workflows","time":"2024-06-25T14:48:28.694Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result name:\nworkflow-21056-k6k42-3182910293","namespace":"workflows","time":"2024-06-25T14:48:28.694Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Marking task result complete workflow-21056-k6k42-3182910293","namespace":"workflows","time":"2024-06-25T14:48:28.694Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-21056-k6k42-3182910293","time":"2024-06-25T14:48:28.694Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result:\n\u0026WorkflowTaskResult{ObjectMeta:{workflow-21056-k6k42-1595781171  workflows  7925fcd2-1155-4ab9-bcd2-30903ad93313 257537465 2 2024-06-25 14:48:16 +0000 UTC \u003cnil\u003e \u003cnil\u003e map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:workflow-21056-k6k42] map[] [{argoproj.io/v1alpha1 Workflow workflow-21056-k6k42 30ca6b42-885b-407c-86c1-f89f91d2059c \u003cnil\u003e \u003cnil\u003e}] []  [{argoexec Update argoproj.io/v1alpha1 2024-06-25 14:48:17 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"30ca6b42-885b-407c-86c1-f89f91d2059c\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{},\"f:parameters\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:\u0026Outputs{Parameters:[]Parameter{Parameter{Name:podname,Default:nil,Value:*workflow-21056-k6k42-action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3-1595781171,ValueFrom:nil,GlobalName:,Enum:[],Description:nil,},},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:\u0026S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:default/workflow-21056-k6k42/workflow-21056-k6k42-action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3-1595781171/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}","namespace":"workflows","time":"2024-06-25T14:48:28.694Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result name:\nworkflow-21056-k6k42-1595781171","namespace":"workflows","time":"2024-06-25T14:48:28.695Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Marking task result complete workflow-21056-k6k42-1595781171","namespace":"workflows","time":"2024-06-25T14:48:28.695Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-21056-k6k42-1595781171","time":"2024-06-25T14:48:28.695Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result:\n\u0026WorkflowTaskResult{ObjectMeta:{workflow-21056-k6k42-1349795579  workflows  b6f80e49-1f52-43d4-89a8-b54467be6e29 257537676 2 2024-06-25 14:48:25 +0000 UTC \u003cnil\u003e \u003cnil\u003e map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:workflow-21056-k6k42] map[] [{argoproj.io/v1alpha1 Workflow workflow-21056-k6k42 30ca6b42-885b-407c-86c1-f89f91d2059c \u003cnil\u003e \u003cnil\u003e}] []  [{argoexec Update argoproj.io/v1alpha1 2024-06-25 14:48:27 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"30ca6b42-885b-407c-86c1-f89f91d2059c\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:\u0026Outputs{Parameters:[]Parameter{},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:\u0026S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:default/workflow-21056-k6k42/workflow-21056-k6k42-exit-handler-1349795579/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}","namespace":"workflows","time":"2024-06-25T14:48:28.695Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result name:\nworkflow-21056-k6k42-1349795579","namespace":"workflows","time":"2024-06-25T14:48:28.695Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Marking task result complete workflow-21056-k6k42-1349795579","namespace":"workflows","time":"2024-06-25T14:48:28.695Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-21056-k6k42-1349795579","time":"2024-06-25T14:48:28.695Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Skipping artifact GC","namespace":"workflows","time":"2024-06-25T14:48:28.695Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"node changed","namespace":"workflows","new.message":"","new.phase":"Running","new.progress":"0/1","nodeID":"workflow-21056-k6k42-1349795579","old.message":"PodInitializing","old.phase":"Pending","old.progress":"0/1","time":"2024-06-25T14:48:28.695Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Evaluating node workflow-21056-k6k42: template: *v1alpha1.WorkflowStep (root), boundaryID: ","namespace":"workflows","time":"2024-06-25T14:48:28.695Z","workflow":"workflow-21056-k6k42"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Resolving the template","time":"2024-06-25T14:48:28.695Z","tmpl":"*v1alpha1.WorkflowStep (root)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template","time":"2024-06-25T14:48:28.695Z","tmpl":"*v1alpha1.WorkflowStep (root)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template by name: root","time":"2024-06-25T14:48:28.695Z","tmpl":"*v1alpha1.WorkflowStep (root)"}
{"level":"debug","msg":"Node workflow-21056-k6k42 already completed","namespace":"workflows","time":"2024-06-25T14:48:28.698Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"workflows","time":"2024-06-25T14:48:28.698Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"reconcileAgentPod","namespace":"workflows","time":"2024-06-25T14:48:28.698Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"Running OnExit handler: exit-handler","namespace":"workflows","time":"2024-06-25T14:48:28.698Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Evaluating node workflow-21056-k6k42.onExit: template: *v1alpha1.WorkflowStep (exit-handler), boundaryID: ","namespace":"workflows","time":"2024-06-25T14:48:28.698Z","workflow":"workflow-21056-k6k42"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Resolving the template","time":"2024-06-25T14:48:28.698Z","tmpl":"*v1alpha1.WorkflowStep (exit-handler)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template","time":"2024-06-25T14:48:28.699Z","tmpl":"*v1alpha1.WorkflowStep (exit-handler)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template by name: exit-handler","time":"2024-06-25T14:48:28.699Z","tmpl":"*v1alpha1.WorkflowStep (exit-handler)"}
{"level":"debug","msg":"Executing node workflow-21056-k6k42.onExit of Pod is Running","namespace":"workflows","time":"2024-06-25T14:48:28.700Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Executing node workflow-21056-k6k42.onExit with container template: exit-handler\n","namespace":"workflows","time":"2024-06-25T14:48:28.700Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Skipped pod workflow-21056-k6k42.onExit (workflow-21056-k6k42-1349795579) creation: already exists","namespace":"workflows","podPhase":"Running","time":"2024-06-25T14:48:28.700Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Log changes patch: {\"status\":{\"conditions\":[{\"status\":\"True\",\"type\":\"PodRunning\"}],\"nodes\":{\"workflow-21056-k6k42-1349795579\":{\"message\":null,\"outputs\":{\"artifacts\":[{\"name\":\"main-logs\",\"s3\":{\"key\":\"default/workflow-21056-k6k42/workflow-21056-k6k42-exit-handler-1349795579/main.log\"}}],\"exitCode\":\"0\"},\"phase\":\"Running\"}},\"taskResultsCompletionStatus\":{\"workflow-21056-k6k42-1349795579\":true}}}","time":"2024-06-25T14:48:28.709Z"}
{"action":"terminateContainers","key":"workflows/workflow-21056-k6k42-exit-handler-1349795579/terminateContainers","level":"info","msg":"cleaning up pod","time":"2024-06-25T14:48:28.710Z"}
{"level":"info","msg":"Workflow update successful","namespace":"workflows","phase":"Running","resourceVersion":"257537693","time":"2024-06-25T14:48:28.744Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"workflows\", Name:\"workflow-21056-k6k42\", UID:\"30ca6b42-885b-407c-86c1-f89f91d2059c\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"257537693\", FieldPath:\"\"}): type: 'Normal' reason: 'WorkflowNodeRunning' Running node workflow-21056-k6k42.onExit","time":"2024-06-25T14:48:28.757Z"}
{"Phase":"Running","ResourceVersion":"257537709","level":"info","msg":"Processing workflow","namespace":"workflows","time":"2024-06-25T14:48:34.231Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"Task-result reconciliation","namespace":"workflows","numObjs":6,"time":"2024-06-25T14:48:34.233Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result:\n\u0026WorkflowTaskResult{ObjectMeta:{workflow-21056-k6k42-1349795579  workflows  b6f80e49-1f52-43d4-89a8-b54467be6e29 257537676 2 2024-06-25 14:48:25 +0000 UTC \u003cnil\u003e \u003cnil\u003e map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:workflow-21056-k6k42] map[] [{argoproj.io/v1alpha1 Workflow workflow-21056-k6k42 30ca6b42-885b-407c-86c1-f89f91d2059c \u003cnil\u003e \u003cnil\u003e}] []  [{argoexec Update argoproj.io/v1alpha1 2024-06-25 14:48:27 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"30ca6b42-885b-407c-86c1-f89f91d2059c\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:\u0026Outputs{Parameters:[]Parameter{},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:\u0026S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:default/workflow-21056-k6k42/workflow-21056-k6k42-exit-handler-1349795579/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}","namespace":"workflows","time":"2024-06-25T14:48:34.233Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result name:\nworkflow-21056-k6k42-1349795579","namespace":"workflows","time":"2024-06-25T14:48:34.233Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Marking task result complete workflow-21056-k6k42-1349795579","namespace":"workflows","time":"2024-06-25T14:48:34.234Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-21056-k6k42-1349795579","time":"2024-06-25T14:48:34.234Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result:\n\u0026WorkflowTaskResult{ObjectMeta:{workflow-21056-k6k42-2728974550  workflows  2b26e8f2-60ec-4a46-9d82-a8135fb1f074 257536296 2 2024-06-25 14:47:56 +0000 UTC \u003cnil\u003e \u003cnil\u003e map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:workflow-21056-k6k42] map[] [{argoproj.io/v1alpha1 Workflow workflow-21056-k6k42 30ca6b42-885b-407c-86c1-f89f91d2059c \u003cnil\u003e \u003cnil\u003e}] []  [{argoexec Update argoproj.io/v1alpha1 2024-06-25 14:47:58 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"30ca6b42-885b-407c-86c1-f89f91d2059c\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{},\"f:parameters\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:\u0026Outputs{Parameters:[]Parameter{Parameter{Name:podname,Default:nil,Value:*workflow-21056-k6k42-action-version-7917-task-df57904b-a342-4892-bb45-5226e7eb3244-2728974550,ValueFrom:nil,GlobalName:,Enum:[],Description:nil,},},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:\u0026S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:default/workflow-21056-k6k42/workflow-21056-k6k42-action-version-7917-task-df57904b-a342-4892-bb45-5226e7eb3244-2728974550/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}","namespace":"workflows","time":"2024-06-25T14:48:34.234Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result name:\nworkflow-21056-k6k42-2728974550","namespace":"workflows","time":"2024-06-25T14:48:34.234Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Marking task result complete workflow-21056-k6k42-2728974550","namespace":"workflows","time":"2024-06-25T14:48:34.234Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-21056-k6k42-2728974550","time":"2024-06-25T14:48:34.234Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result:\n\u0026WorkflowTaskResult{ObjectMeta:{workflow-21056-k6k42-3451334822  workflows  17635c50-a7f1-45a6-b5f8-6b0be6b5e7cb 257537000 2 2024-06-25 14:48:05 +0000 UTC \u003cnil\u003e \u003cnil\u003e map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:workflow-21056-k6k42] map[] [{argoproj.io/v1alpha1 Workflow workflow-21056-k6k42 30ca6b42-885b-407c-86c1-f89f91d2059c \u003cnil\u003e \u003cnil\u003e}] []  [{argoexec Update argoproj.io/v1alpha1 2024-06-25 14:48:07 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"30ca6b42-885b-407c-86c1-f89f91d2059c\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{},\"f:parameters\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:\u0026Outputs{Parameters:[]Parameter{Parameter{Name:podname,Default:nil,Value:*workflow-21056-k6k42-action-version-7917-task-51e19cea-5aa1-4088-8510-76395472ea67-3451334822,ValueFrom:nil,GlobalName:,Enum:[],Description:nil,},},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:\u0026S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:default/workflow-21056-k6k42/workflow-21056-k6k42-action-version-7917-task-51e19cea-5aa1-4088-8510-76395472ea67-3451334822/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}","namespace":"workflows","time":"2024-06-25T14:48:34.234Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result name:\nworkflow-21056-k6k42-3451334822","namespace":"workflows","time":"2024-06-25T14:48:34.234Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Marking task result complete workflow-21056-k6k42-3451334822","namespace":"workflows","time":"2024-06-25T14:48:34.234Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-21056-k6k42-3451334822","time":"2024-06-25T14:48:34.234Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result:\n\u0026WorkflowTaskResult{ObjectMeta:{workflow-21056-k6k42-4278140849  workflows  aa6afaa7-8f90-456b-a5c7-b4430bda2258 257537003 2 2024-06-25 14:48:05 +0000 UTC \u003cnil\u003e \u003cnil\u003e map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:workflow-21056-k6k42] map[] [{argoproj.io/v1alpha1 Workflow workflow-21056-k6k42 30ca6b42-885b-407c-86c1-f89f91d2059c \u003cnil\u003e \u003cnil\u003e}] []  [{argoexec Update argoproj.io/v1alpha1 2024-06-25 14:48:07 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"30ca6b42-885b-407c-86c1-f89f91d2059c\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{},\"f:parameters\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:\u0026Outputs{Parameters:[]Parameter{Parameter{Name:podname,Default:nil,Value:*workflow-21056-k6k42-action-version-7917-task-d6e520c4-8123-4169-b112-2167a2709b12-4278140849,ValueFrom:nil,GlobalName:,Enum:[],Description:nil,},},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:\u0026S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:default/workflow-21056-k6k42/workflow-21056-k6k42-action-version-7917-task-d6e520c4-8123-4169-b112-2167a2709b12-4278140849/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}","namespace":"workflows","time":"2024-06-25T14:48:34.234Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result name:\nworkflow-21056-k6k42-4278140849","namespace":"workflows","time":"2024-06-25T14:48:34.234Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Marking task result complete workflow-21056-k6k42-4278140849","namespace":"workflows","time":"2024-06-25T14:48:34.234Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-21056-k6k42-4278140849","time":"2024-06-25T14:48:34.234Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result:\n\u0026WorkflowTaskResult{ObjectMeta:{workflow-21056-k6k42-3182910293  workflows  2d8f409a-74e2-4ea4-a5b2-0039bb29783c 257536992 2 2024-06-25 14:48:06 +0000 UTC \u003cnil\u003e \u003cnil\u003e map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:workflow-21056-k6k42] map[] [{argoproj.io/v1alpha1 Workflow workflow-21056-k6k42 30ca6b42-885b-407c-86c1-f89f91d2059c \u003cnil\u003e \u003cnil\u003e}] []  [{argoexec Update argoproj.io/v1alpha1 2024-06-25 14:48:07 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"30ca6b42-885b-407c-86c1-f89f91d2059c\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{},\"f:parameters\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:\u0026Outputs{Parameters:[]Parameter{Parameter{Name:podname,Default:nil,Value:*workflow-21056-k6k42-action-version-7917-task-8c2a4142-1f08-4997-a82e-5edfc2e8d6f0-3182910293,ValueFrom:nil,GlobalName:,Enum:[],Description:nil,},},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:\u0026S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:default/workflow-21056-k6k42/workflow-21056-k6k42-action-version-7917-task-8c2a4142-1f08-4997-a82e-5edfc2e8d6f0-3182910293/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}","namespace":"workflows","time":"2024-06-25T14:48:34.234Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result name:\nworkflow-21056-k6k42-3182910293","namespace":"workflows","time":"2024-06-25T14:48:34.234Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Marking task result complete workflow-21056-k6k42-3182910293","namespace":"workflows","time":"2024-06-25T14:48:34.234Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-21056-k6k42-3182910293","time":"2024-06-25T14:48:34.234Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result:\n\u0026WorkflowTaskResult{ObjectMeta:{workflow-21056-k6k42-1595781171  workflows  7925fcd2-1155-4ab9-bcd2-30903ad93313 257537465 2 2024-06-25 14:48:16 +0000 UTC \u003cnil\u003e \u003cnil\u003e map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:workflow-21056-k6k42] map[] [{argoproj.io/v1alpha1 Workflow workflow-21056-k6k42 30ca6b42-885b-407c-86c1-f89f91d2059c \u003cnil\u003e \u003cnil\u003e}] []  [{argoexec Update argoproj.io/v1alpha1 2024-06-25 14:48:17 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"30ca6b42-885b-407c-86c1-f89f91d2059c\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{},\"f:parameters\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:\u0026Outputs{Parameters:[]Parameter{Parameter{Name:podname,Default:nil,Value:*workflow-21056-k6k42-action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3-1595781171,ValueFrom:nil,GlobalName:,Enum:[],Description:nil,},},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:\u0026S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:default/workflow-21056-k6k42/workflow-21056-k6k42-action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3-1595781171/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}","namespace":"workflows","time":"2024-06-25T14:48:34.234Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result name:\nworkflow-21056-k6k42-1595781171","namespace":"workflows","time":"2024-06-25T14:48:34.234Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Marking task result complete workflow-21056-k6k42-1595781171","namespace":"workflows","time":"2024-06-25T14:48:34.234Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-21056-k6k42-1595781171","time":"2024-06-25T14:48:34.234Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Skipping artifact GC","namespace":"workflows","time":"2024-06-25T14:48:34.234Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"node changed","namespace":"workflows","new.message":"","new.phase":"Succeeded","new.progress":"0/1","nodeID":"workflow-21056-k6k42-1349795579","old.message":"","old.phase":"Running","old.progress":"0/1","time":"2024-06-25T14:48:34.235Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Evaluating node workflow-21056-k6k42: template: *v1alpha1.WorkflowStep (root), boundaryID: ","namespace":"workflows","time":"2024-06-25T14:48:34.235Z","workflow":"workflow-21056-k6k42"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Resolving the template","time":"2024-06-25T14:48:34.235Z","tmpl":"*v1alpha1.WorkflowStep (root)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template","time":"2024-06-25T14:48:34.235Z","tmpl":"*v1alpha1.WorkflowStep (root)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template by name: root","time":"2024-06-25T14:48:34.235Z","tmpl":"*v1alpha1.WorkflowStep (root)"}
{"level":"debug","msg":"Node workflow-21056-k6k42 already completed","namespace":"workflows","time":"2024-06-25T14:48:34.235Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"workflows","time":"2024-06-25T14:48:34.235Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"reconcileAgentPod","namespace":"workflows","time":"2024-06-25T14:48:34.235Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"Running OnExit handler: exit-handler","namespace":"workflows","time":"2024-06-25T14:48:34.235Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Evaluating node workflow-21056-k6k42.onExit: template: *v1alpha1.WorkflowStep (exit-handler), boundaryID: ","namespace":"workflows","time":"2024-06-25T14:48:34.235Z","workflow":"workflow-21056-k6k42"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Resolving the template","time":"2024-06-25T14:48:34.235Z","tmpl":"*v1alpha1.WorkflowStep (exit-handler)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template","time":"2024-06-25T14:48:34.235Z","tmpl":"*v1alpha1.WorkflowStep (exit-handler)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template by name: exit-handler","time":"2024-06-25T14:48:34.235Z","tmpl":"*v1alpha1.WorkflowStep (exit-handler)"}
{"level":"debug","msg":"Node workflow-21056-k6k42.onExit already completed","namespace":"workflows","time":"2024-06-25T14:48:34.236Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Task results completion status: map[workflow-21056-k6k42-1349795579:true workflow-21056-k6k42-1595781171:true workflow-21056-k6k42-2728974550:true workflow-21056-k6k42-3182910293:true workflow-21056-k6k42-3451334822:true workflow-21056-k6k42-4278140849:true]","namespace":"workflows","time":"2024-06-25T14:48:34.236Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"Updated phase Running -\u003e Succeeded","namespace":"workflows","time":"2024-06-25T14:48:34.236Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"Marking workflow completed","namespace":"workflows","time":"2024-06-25T14:48:34.236Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"Marking workflow as pending archiving","namespace":"workflows","time":"2024-06-25T14:48:34.236Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"workflows\", Name:\"workflow-21056-k6k42\", UID:\"30ca6b42-885b-407c-86c1-f89f91d2059c\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"257537709\", FieldPath:\"\"}): type: 'Normal' reason: 'WorkflowSucceeded' Workflow completed","time":"2024-06-25T14:48:34.236Z"}
{"level":"debug","msg":"Log changes patch: {\"metadata\":{\"labels\":{\"workflows.argoproj.io/completed\":\"true\",\"workflows.argoproj.io/phase\":\"Succeeded\",\"workflows.argoproj.io/workflow-archiving-status\":\"Pending\"}},\"status\":{\"conditions\":[{\"status\":\"False\",\"type\":\"PodRunning\"},{\"status\":\"True\",\"type\":\"Completed\"}],\"finishedAt\":\"2024-06-25T14:48:34Z\",\"nodes\":{\"workflow-21056-k6k42-1349795579\":{\"finishedAt\":\"2024-06-25T14:48:27Z\",\"phase\":\"Succeeded\",\"resourcesDuration\":{\"cpu\":0,\"memory\":3}}},\"phase\":\"Succeeded\"}}","time":"2024-06-25T14:48:34.237Z"}
{"action":"deletePod","key":"workflows/workflow-21056-k6k42-1340600742-agent/deletePod","level":"info","msg":"cleaning up pod","time":"2024-06-25T14:48:34.242Z"}
{"level":"info","msg":"Workflow update successful","namespace":"workflows","phase":"Succeeded","resourceVersion":"257537770","time":"2024-06-25T14:48:34.256Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"workflows\", Name:\"workflow-21056-k6k42\", UID:\"30ca6b42-885b-407c-86c1-f89f91d2059c\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"257537770\", FieldPath:\"\"}): type: 'Normal' reason: 'WorkflowNodeSucceeded' Succeeded node workflow-21056-k6k42.onExit","time":"2024-06-25T14:48:34.258Z"}
{"level":"info","msg":"Queueing Succeeded workflow workflows/workflow-21056-k6k42 for delete in 5m0s due to TTL","time":"2024-06-25T14:48:34.365Z"}
{"level":"info","msg":"archiving workflow","namespace":"workflows","time":"2024-06-25T14:48:34.462Z","uid":"30ca6b42-885b-407c-86c1-f89f91d2059c","workflow":"workflow-21056-k6k42"}
{"action":"deletePod","key":"workflows/workflow-21056-k6k42-exit-handler-1349795579/deletePod","level":"info","msg":"cleaning up pod","time":"2024-06-25T14:48:36.462Z"}
{"action":"killContainers","key":"workflows/workflow-21056-k6k42-action-version-7917-task-51e19cea-5aa1-4088-8510-76395472ea67-3451334822/killContainers","level":"info","msg":"cleaning up pod","time":"2024-06-25T14:48:38.406Z"}
{"action":"killContainers","key":"workflows/workflow-21056-k6k42-action-version-7917-task-8c2a4142-1f08-4997-a82e-5edfc2e8d6f0-3182910293/killContainers","level":"info","msg":"cleaning up pod","time":"2024-06-25T14:48:38.406Z"}
{"action":"killContainers","key":"workflows/workflow-21056-k6k42-action-version-7917-task-d6e520c4-8123-4169-b112-2167a2709b12-4278140849/killContainers","level":"info","msg":"cleaning up pod","time":"2024-06-25T14:48:38.474Z"}
{"key":"workflows/workflow-21056-k6k42","level":"debug","msg":"Won't process Workflow since it's completed","time":"2024-06-25T14:48:41.479Z"}
{"action":"killContainers","key":"workflows/workflow-21056-k6k42-action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3-1595781171/killContainers","level":"info","msg":"cleaning up pod","time":"2024-06-25T14:48:48.587Z"}
{"action":"killContainers","key":"workflows/workflow-21056-k6k42-exit-handler-1349795579/killContainers","level":"info","msg":"cleaning up pod","time":"2024-06-25T14:48:58.712Z"}
{"level":"info","msg":"Deleting garbage collected workflow 'workflows/workflow-21056-k6k42'","time":"2024-06-25T14:53:35.000Z"}
{"level":"info","msg":"Workflow already deleted 'workflows/workflow-21056-k6k42'","time":"2024-06-25T14:53:35.017Z"}

argoproj / argo-workflows

3.5.6 Logs flooded with `Error updating workflow: Operation cannot be fulfilled on workflows.argoproj.io` #13149