search

argoproj / argo-workflows

Workflow Engine for Kubernetes
https://argo-workflows.readthedocs.io/
Apache License 2.0
14.69k stars 3.15k forks source link

Daemon pod is always succeeded even with a non-zero status code #12911

Open KataaOmar opened 4 months ago

KataaOmar commented 4 months ago

Pre-requisites

What happened/what did you expect to happen?

Daemon pod is always succeeded even with a non-zero status code, I expect to find the daemon step failed, then the workflow as well. image

Version

v3.5.5

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: daemon-step-
  namespace: argo
spec:
  entrypoint: daemon-example
  templates:
  - name: daemon-example
    steps:
    - - name: daemon-pod
        template: daemon-temp

    - - name: client-pod             
        template: client-temp 

  - name: daemon-temp
    daemon: true                       
    container:
      image: alpine
      command: ["/bin/sh", "-c"]
      args:
      - |
        sleep 30
        exit 1

  - name: client-temp
    container:
      image: alpine
      command: ["/bin/sh", "-c"]
      args: ["sleep 600"]

Logs from the workflow controller

workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:00:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:13:57.016Z" level=info msg="Alloc=4104 TotalAlloc=167252 Sys=19301 NumGC=447 Goroutines=18"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:23:00.333Z" level=info msg="Alloc=47122 TotalAlloc=20074390 Sys=142600 NumGC=799 Goroutines=186"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:28:00.333Z" level=info msg="Alloc=52411 TotalAlloc=20105721 Sys=142600 NumGC=801 Goroutines=186"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:00:22.228Z" level=info msg="Alloc=3956 TotalAlloc=74070 Sys=14949 NumGC=190 Goroutines=18"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:01:16.663Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:02:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:03:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:04:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:05:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:05:22.228Z" level=info msg="Alloc=4131 TotalAlloc=74930 Sys=14949 NumGC=192 Goroutines=18"
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:14:33.116Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:15:33.117Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:16:33.116Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:06:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:07:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:08:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:09:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:17:33.116Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:18:33.117Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:18:57.016Z" level=info msg="Alloc=3965 TotalAlloc=168130 Sys=19301 NumGC=450 Goroutines=18"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:10:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:10:22.228Z" level=info msg="Alloc=3972 TotalAlloc=75682 Sys=14949 NumGC=195 Goroutines=18"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:11:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:12:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:28:08.667Z" level=info msg="Processing workflow" Phase=Failed ResourceVersion=843798350 namespace=platform workflow=playwright-brt-publish-tcvvj
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:19:33.117Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:28:08.667Z" level=info msg="Task-result reconciliation" namespace=platform numObjs=8 workflow=playwright-brt-publish-tcvvj
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:28:08.667Z" level=info msg="task-result changed" namespace=platform nodeID=playwright-brt-publish-tcvvj-574790004 workflow=playwright-brt-publish-tcvvj
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:28:08.667Z" level=info msg="task-result changed" namespace=platform nodeID=playwright-brt-publish-tcvvj-1281463502 workflow=playwright-brt-publish-tcvvj
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:28:08.668Z" level=info msg="task-result changed" namespace=platform nodeID=playwright-brt-publish-tcvvj-1981189896 workflow=playwright-brt-publish-tcvvj
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:13:16.663Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:14:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:15:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:15:22.228Z" level=info msg="Alloc=4208 TotalAlloc=76598 Sys=14949 NumGC=197 Goroutines=18"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:28:08.668Z" level=info msg="task-result changed" namespace=platform nodeID=playwright-brt-publish-tcvvj-2640092480 workflow=playwright-brt-publish-tcvvj
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:28:08.668Z" level=info msg="task-result changed" namespace=platform nodeID=playwright-brt-publish-tcvvj-3398277239 workflow=playwright-brt-publish-tcvvj
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:28:08.668Z" level=info msg="task-result changed" namespace=platform nodeID=playwright-brt-publish-tcvvj-3767785617 workflow=playwright-brt-publish-tcvvj
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:20:33.116Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:21:33.117Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:16:16.663Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:22:33.117Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:17:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:23:33.117Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:18:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:23:57.015Z" level=info msg="Alloc=4147 TotalAlloc=169066 Sys=19301 NumGC=452 Goroutines=18"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:19:16.663Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:24:33.117Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:25:33.116Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:26:33.117Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:28:08.668Z" level=info msg="task-result changed" namespace=platform nodeID=playwright-brt-publish-tcvvj-4209023662 workflow=playwright-brt-publish-tcvvj
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:20:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:28:08.668Z" level=info msg="task-result changed" namespace=platform nodeID=playwright-brt-publish-tcvvj-435495160 workflow=playwright-brt-publish-tcvvj
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:20:22.228Z" level=info msg="Alloc=3980 TotalAlloc=77446 Sys=14949 NumGC=200 Goroutines=18"
-fct4n
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:21:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:22:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:28:08.669Z" level=info msg="TaskSet Reconciliation" namespace=platform workflow=playwright-brt-publish-tcvvj
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:27:33.117Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:28:33.116Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:23:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:24:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:25:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:28:57.016Z" level=info msg="Alloc=3978 TotalAlloc=169943 Sys=19301 NumGC=455 Goroutines=18"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:28:08.669Z" level=info msg=reconcileAgentPod namespace=platform workflow=playwright-brt-publish-tcvvj
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:28:08.669Z" level=info msg="Running OnExit handler: exit-handler" namespace=platform workflow=playwright-brt-publish-tcvvj
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:28:08.671Z" level=info msg="Workflow to be dehydrated" Workflow Size=51061
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:25:22.229Z" level=info msg="Alloc=4197 TotalAlloc=78389 Sys=14949 NumGC=202 Goroutines=18"
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:29:33.117Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:30:33.117Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:31:33.117Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:26:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:27:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:28:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:29:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:30:16.662Z" level=info msg="healthz: current pod is not the leader"

workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:30:22.228Z" level=info msg="Alloc=3946 TotalAlloc=79229 Sys=14949 NumGC=205 Goroutines=18"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:31:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:32:16.663Z" level=info msg="healthz: current pod is not the leader"

workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:28:08.671Z" level=info msg="Workflow to be dehydrated" Workflow Size=36561
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:29:14.463Z" level=info msg="Received Workflow Controller config map argo/workflow-controller-configmap update"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:29:14.470Z" level=info msg="Configuration:\nartifactRepository:\n  archiveLogs: true\n  gcs:\n    bucket: argo-artifact-repository-staging\nexecutor:\n  name: \"\"\n  resources: {}\n  securityContext:\n    allowPrivilegeEscalation: false\n    capabilities:\n      drop:\n      - ALL\n    privileged: false\n    runAsGroup: 8737\n    runAsNonRoot: true\n    runAsUser: 8737\n  terminationMessagePolicy: FallbackToLogsOnError\n  volumeMounts:\n  - mountPath: /tmp\n    name: tmp\ninitialDelay: 0s\nmainContainer:\n  name: \"\"\n  resources: {}\n  securityContext:\n    allowPrivilegeEscalation: false\n    capabilities:\n      drop:\n      - ALL\n    privileged: false\n    readOnlyRootFilesystem: true\n    runAsGroup: 8737\n    runAsNonRoot: true\n    runAsUser: 8737\n  terminationMessagePolicy: FallbackToLogsOnError\n  volumeMounts:\n  - mountPath: /tmp\n    name: tmp\nmetricsConfig: {}\nnodeEvents: {}\npersistence:\n  archive: true\n  archiveTTL: 26280h0m0s\n  clusterName: playground\n  connectionPool:\n    maxIdleConns: 100\n  postgresql:\n    database: postgres\n    host: 10.40.33.7\n    passwordSecret:\n      key: password\n      name: argo-postgres-config\n    port: 5432\n    ssl: true\n    sslMode: require\n    tableName: argo_workflows\n    userNameSecret:\n      key: username\n      name: argo-postgres-config\npodSpecLogStrategy: {}\nsso:\n  clientId:\n    key: client-id\n    name: argo-workflows-sso\n  clientSecret:\n    key: client-secret\n    name: argo-workflows-sso\n  issuer: https://staging.argocd.forbes.com/api/dex\n  rbac:\n    enabled: true\n  redirectUrl: https://staging.argo.forbes.com/oauth2/callback\n  scopes:\n  - groups\n  sessionExpiry: 0s\ntelemetryConfig: {}\nworkflowDefaults:\n  metadata:\n    creationTimestamp: null\n  spec:\n    activeDeadlineSeconds: 18000\n    arguments: {}\n    automountServiceAccountToken: false\n    executor:\n      serviceAccountName: argo-read\n    podDisruptionBudget:\n      maxUnavailable: 0\n    podGC:\n      strategy: OnWorkflowSuccess\n    securityContext:\n      runAsGroup: 8737\n      runAsNonRoot: true\n      runAsUser: 8737\n    serviceAccountName: argo-read\n    templateDefaults:\n      inputs: {}\n      metadata: {}\n      outputs: {}\n      volumes:\n      - emptyDir: {}\n        name: tmp\n    ttlStrategy:\n      secondsAfterFailure: 10800\n      secondsAfterSuccess: 900\n  status:\n    finishedAt: null\n    startedAt: null\n"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:29:14.470Z" level=info msg="Persistence configuration enabled"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:29:14.471Z" level=info msg="Node status offloading is disabled"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:29:14.471Z" level=info msg="Workflow archiving is enabled"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:29:14.471Z" level=info executorImage="quay.io/argoproj/argoexec:v3.5.5" executorImagePullPolicy= managedNamespace=
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:30:00.001Z" level=info msg="Running clear-non-running-pods" namespace=argo workflow=clear-non-running-pods
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:30:00.001Z" level=info msg="Running cert-checker-error" namespace=argo workflow=cert-checker-error
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:30:00.001Z" level=info msg="clear-non-running-pods is suspended, skipping execution" namespace=argo workflow=clear-non-running-pods
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:30:00.002Z" level=info msg="cert-checker-error is suspended, skipping execution" namespace=argo workflow=cert-checker-error
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:32:33.116Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:33:00.334Z" level=info msg="Alloc=47471 TotalAlloc=20171583 Sys=142600 NumGC=804 Goroutines=187"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:33:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:33:33.117Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:33:54.948Z" level=info msg="Processing workflow" Phase= ResourceVersion=902005365 namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:33:54.955Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:33:54.955Z" level=info msg="resolved artifact repository" artifactRepositoryRef=default-artifact-repository
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:33:54.955Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:33:54.955Z" level=info msg="Updated phase  -> Running" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:33:54.968Z" level=info msg="Created PDB resource for workflow." namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:33:54.968Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:33:54.968Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:33:54.968Z" level=info msg="Steps node daemon-step-qw2b6 initialized Running" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:33:54.968Z" level=info msg="StepGroup node daemon-step-qw2b6-84771817 initialized Running" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:33:54.969Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:33:54.969Z" level=info msg="Pod node daemon-step-qw2b6-707072755 initialized Pending" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:33:55.021Z" level=info msg="Created pod: daemon-step-qw2b6[0].daemon-pod (daemon-step-qw2b6-daemon-temp-707072755)" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:33:55.021Z" level=info msg="Workflow step group node daemon-step-qw2b6-84771817 not yet completed" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:33:55.021Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:33:55.021Z" level=info msg=reconcileAgentPod namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:33:55.021Z" level=info msg="Workflow to be dehydrated" Workflow Size=1706
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:33:55.032Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=902005371 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:33:57.016Z" level=info msg="Alloc=4175 TotalAlloc=170827 Sys=19301 NumGC=457 Goroutines=18"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:05.022Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=902005371 namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:05.023Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=1 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:05.024Z" level=info msg="task-result changed" namespace=argo nodeID=daemon-step-qw2b6-707072755 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:05.024Z" level=info msg="Node became daemoned" namespace=argo nodeId=daemon-step-qw2b6-707072755 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:05.024Z" level=info msg="node changed" namespace=argo new.message= new.phase=Running new.progress=0/1 nodeID=daemon-step-qw2b6-707072755 old.message= old.phase=Pending old.progress=0/1 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:05.024Z" level=info msg="Step group node daemon-step-qw2b6-84771817 successful" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:05.025Z" level=info msg="node daemon-step-qw2b6-84771817 phase Running -> Succeeded" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:05.025Z" level=info msg="node daemon-step-qw2b6-84771817 finished: 2024-04-08 09:34:05.02501423 +0000 UTC" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:05.025Z" level=info msg="StepGroup node daemon-step-qw2b6-1091576052 initialized Running" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:05.025Z" level=info msg="SG Outbound nodes of daemon-step-qw2b6-707072755 are [daemon-step-qw2b6-707072755]" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:05.025Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:05.025Z" level=info msg="Pod node daemon-step-qw2b6-3790013067 initialized Pending" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:05.064Z" level=info msg="Created pod: daemon-step-qw2b6[1].client-pod (daemon-step-qw2b6-client-temp-3790013067)" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:05.064Z" level=info msg="Workflow step group node daemon-step-qw2b6-1091576052 not yet completed" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:05.064Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:05.064Z" level=info msg=reconcileAgentPod namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:05.064Z" level=info msg="Workflow to be dehydrated" Workflow Size=2620
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:05.079Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=902005503 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:15.064Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=902005503 namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:15.065Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=2 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:15.065Z" level=info msg="task-result changed" namespace=argo nodeID=daemon-step-qw2b6-707072755 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:15.065Z" level=info msg="task-result changed" namespace=argo nodeID=daemon-step-qw2b6-3790013067 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:15.065Z" level=info msg="node changed" namespace=argo new.message= new.phase=Running new.progress=0/1 nodeID=daemon-step-qw2b6-3790013067 old.message= old.phase=Pending old.progress=0/1 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:15.066Z" level=info msg="node unchanged" namespace=argo nodeID=daemon-step-qw2b6-707072755 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:15.066Z" level=info msg="SG Outbound nodes of daemon-step-qw2b6-707072755 are [daemon-step-qw2b6-707072755]" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:15.066Z" level=info msg="Workflow step group node daemon-step-qw2b6-1091576052 not yet completed" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:15.066Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:15.066Z" level=info msg=reconcileAgentPod namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:15.066Z" level=info msg="Workflow to be dehydrated" Workflow Size=2692
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:15.078Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=902005629 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:34:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:25.079Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=902005629 namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:25.080Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=2 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:25.080Z" level=info msg="task-result changed" namespace=argo nodeID=daemon-step-qw2b6-707072755 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:25.080Z" level=info msg="task-result changed" namespace=argo nodeID=daemon-step-qw2b6-3790013067 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:25.080Z" level=info msg="node unchanged" namespace=argo nodeID=daemon-step-qw2b6-3790013067 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:25.080Z" level=info msg="node unchanged" namespace=argo nodeID=daemon-step-qw2b6-707072755 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:25.081Z" level=info msg="SG Outbound nodes of daemon-step-qw2b6-707072755 are [daemon-step-qw2b6-707072755]" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:25.081Z" level=info msg="Workflow step group node daemon-step-qw2b6-1091576052 not yet completed" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:25.081Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:25.081Z" level=info msg=reconcileAgentPod namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:25.081Z" level=info msg="Workflow to be dehydrated" Workflow Size=2693
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:25.095Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=902005749 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:34:33.117Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:35.095Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=902005749 namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:35.097Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=2 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:35.097Z" level=info msg="task-result changed" namespace=argo nodeID=daemon-step-qw2b6-707072755 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:35.097Z" level=info msg="task-result changed" namespace=argo nodeID=daemon-step-qw2b6-3790013067 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:35.097Z" level=info msg="node unchanged" namespace=argo nodeID=daemon-step-qw2b6-3790013067 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:35.097Z" level=info msg="node changed" namespace=argo new.message= new.phase=Succeeded new.progress=0/1 nodeID=daemon-step-qw2b6-707072755 old.message= old.phase=Running old.progress=0/1 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:35.097Z" level=info msg="SG Outbound nodes of daemon-step-qw2b6-707072755 are [daemon-step-qw2b6-707072755]" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:35.098Z" level=info msg="Workflow step group node daemon-step-qw2b6-1091576052 not yet completed" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:35.098Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:35.098Z" level=info msg=reconcileAgentPod namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:35.098Z" level=info msg="Workflow to be dehydrated" Workflow Size=2803
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:35.111Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=902005874 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:45.112Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=902005874 namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:45.113Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=2 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:45.113Z" level=info msg="task-result changed" namespace=argo nodeID=daemon-step-qw2b6-707072755 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:45.113Z" level=info msg="task-result changed" namespace=argo nodeID=daemon-step-qw2b6-3790013067 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:45.113Z" level=info msg="node unchanged" namespace=argo nodeID=daemon-step-qw2b6-3790013067 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:45.113Z" level=info msg="node unchanged" namespace=argo nodeID=daemon-step-qw2b6-707072755 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:45.114Z" level=info msg="SG Outbound nodes of daemon-step-qw2b6-707072755 are [daemon-step-qw2b6-707072755]" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:45.114Z" level=info msg="Workflow step group node daemon-step-qw2b6-1091576052 not yet completed" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:45.114Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:45.114Z" level=info msg=reconcileAgentPod namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:45.114Z" level=info msg="Workflow to be dehydrated" Workflow Size=2803
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:45.124Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=902005874 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:35:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:35:22.228Z" level=info msg="Alloc=4146 TotalAlloc=80063 Sys=14949 NumGC=207 Goroutines=18"
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:35:33.117Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:36:04.662Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=902005874 namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:36:04.663Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=2 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:36:04.663Z" level=info msg="task-result changed" namespace=argo nodeID=daemon-step-qw2b6-3790013067 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:36:04.663Z" level=info msg="task-result changed" namespace=argo nodeID=daemon-step-qw2b6-707072755 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:36:04.663Z" level=info msg="node unchanged" namespace=argo nodeID=daemon-step-qw2b6-3790013067 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:36:04.663Z" level=info msg="node unchanged" namespace=argo nodeID=daemon-step-qw2b6-707072755 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:36:04.663Z" level=info msg="SG Outbound nodes of daemon-step-qw2b6-707072755 are [daemon-step-qw2b6-707072755]" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:36:04.664Z" level=info msg="Workflow step group node daemon-step-qw2b6-1091576052 not yet completed" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:36:04.664Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:36:04.664Z" level=info msg=reconcileAgentPod namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:36:04.664Z" level=info msg="Workflow to be dehydrated" Workflow Size=2803
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:36:04.676Z" level=info msg="Workflow to be dehydrated" Workflow Size=51061
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:36:04.676Z" level=info msg="Workflow to be dehydrated" Workflow Size=36561
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:36:16.663Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:36:33.117Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:37:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:37:33.117Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:00.333Z" level=info msg="Alloc=74797 TotalAlloc=20233272 Sys=142600 NumGC=806 Goroutines=188"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:38:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:20.231Z" level=info msg="Executor plugin updated" name=slack-plugin namespace=argo
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.052Z" level=info msg="Processing qa/bq-validator" cronWorkflow=qa/bq-validator
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.052Z" level=info msg="Processing backend/update-forbesapi" cronWorkflow=backend/update-forbesapi
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.052Z" level=info msg="Processing content-products/bertie-bulletin-2k" cronWorkflow=content-products/bertie-bulletin-2k
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.052Z" level=info msg="Processing frontend/update-simple-site-staging-deployments" cronWorkflow=frontend/update-simple-site-staging-deployments
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.063Z" level=info msg="CronWorkflow qa/bq-validator added" cronWorkflow=qa/bq-validator
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.052Z" level=info msg="Processing argo/clear-non-running-pods" cronWorkflow=argo/clear-non-running-pods
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.064Z" level=info msg="CronWorkflow argo/clear-non-running-pods added" cronWorkflow=argo/clear-non-running-pods
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.052Z" level=info msg="Processing content-products/bertie-bulletin-po" cronWorkflow=content-products/bertie-bulletin-po
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.065Z" level=info msg="Processing argo/cert-checker-error" cronWorkflow=argo/cert-checker-error
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.068Z" level=info msg="CronWorkflow argo/cert-checker-error added" cronWorkflow=argo/cert-checker-error
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.069Z" level=info msg="Processing argo/fastly-google-oauth" cronWorkflow=argo/fastly-google-oauth
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.071Z" level=info msg="CronWorkflow argo/fastly-google-oauth added" cronWorkflow=argo/fastly-google-oauth
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.052Z" level=info msg="Processing profile-portal-dev/staging-profile-api-mongoexport" cronWorkflow=profile-portal-dev/staging-profile-api-mongoexport
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.073Z" level=info msg="Processing argo/gcp-dns-backup" cronWorkflow=argo/gcp-dns-backup
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.075Z" level=info msg="CronWorkflow argo/gcp-dns-backup added" cronWorkflow=argo/gcp-dns-backup
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.075Z" level=info msg="Processing argo/update-simple-site-buckets" cronWorkflow=argo/update-simple-site-buckets
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.085Z" level=info msg="Processing content-products/bertie-bulletin" cronWorkflow=content-products/bertie-bulletin
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.052Z" level=info msg="Processing frontend/update-simple-site-production-deployments" cronWorkflow=frontend/update-simple-site-production-deployments
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.132Z" level=info msg="CronWorkflow content-products/bertie-bulletin-po added" cronWorkflow=content-products/bertie-bulletin-po
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.138Z" level=info msg="Processing argo/cert-sync" cronWorkflow=argo/cert-sync
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.139Z" level=info msg="CronWorkflow content-products/bertie-bulletin added" cronWorkflow=content-products/bertie-bulletin
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.139Z" level=info msg="Processing argo/find-unhealthy-deployments" cronWorkflow=argo/find-unhealthy-deployments
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.132Z" level=info msg="CronWorkflow profile-portal-dev/staging-profile-api-mongoexport added" cronWorkflow=profile-portal-dev/staging-profile-api-mongoexport
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.140Z" level=info msg="Processing argo/cert-checker-warn" cronWorkflow=argo/cert-checker-warn
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.143Z" level=info msg="CronWorkflow argo/find-unhealthy-deployments added" cronWorkflow=argo/find-unhealthy-deployments
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.144Z" level=info msg="Processing mobile/daily-mobile-check" cronWorkflow=mobile/daily-mobile-check
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.153Z" level=info msg="CronWorkflow argo/cert-checker-warn added" cronWorkflow=argo/cert-checker-warn
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.161Z" level=info msg="Processing profile-portal-dev/production-profile-api-mongoexport" cronWorkflow=profile-portal-dev/production-profile-api-mongoexport
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.165Z" level=info msg="CronWorkflow argo/update-simple-site-buckets added" cronWorkflow=argo/update-simple-site-buckets
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.169Z" level=info msg="CronWorkflow mobile/daily-mobile-check added" cronWorkflow=mobile/daily-mobile-check
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.161Z" level=info msg="CronWorkflow argo/cert-sync added" cronWorkflow=argo/cert-sync
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.161Z" level=info msg="CronWorkflow content-products/bertie-bulletin-2k added" cronWorkflow=content-products/bertie-bulletin-2k
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.177Z" level=info msg="CronWorkflow profile-portal-dev/production-profile-api-mongoexport added" cronWorkflow=profile-portal-dev/production-profile-api-mongoexport
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.194Z" level=error msg="cannot validate Workflow: templates.update-forbesapi-staging-deployments.steps[0].update-deloyment templates.update-argocd-appset-deployment-main.steps[2].cicd templates.cicd inputs.parameters.team was not supplied" conditionType=SpecError namespace=backend workflow=update-forbesapi
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.194Z" level=error msg="invalid cron workflow" cronWorkflow=backend/update-forbesapi error="cannot validate Workflow: templates.update-forbesapi-staging-deployments.steps[0].update-deloyment templates.update-argocd-appset-deployment-main.steps[2].cicd templates.cicd inputs.parameters.team was not supplied"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.267Z" level=info msg="CronWorkflow frontend/update-simple-site-staging-deployments added" cronWorkflow=frontend/update-simple-site-staging-deployments
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.286Z" level=info msg="CronWorkflow frontend/update-simple-site-production-deployments added" cronWorkflow=frontend/update-simple-site-production-deployments
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:30.274Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=902005874 namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:30.275Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=2 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:30.276Z" level=info msg="task-result changed" namespace=argo nodeID=daemon-step-qw2b6-3790013067 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:30.276Z" level=info msg="task-result changed" namespace=argo nodeID=daemon-step-qw2b6-707072755 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:30.276Z" level=info msg="node unchanged" namespace=argo nodeID=daemon-step-qw2b6-3790013067 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:30.276Z" level=info msg="node unchanged" namespace=argo nodeID=daemon-step-qw2b6-707072755 workflow=daemon-step-qw2b6
-fct4n
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:30.276Z" level=info msg="SG Outbound nodes of daemon-step-qw2b6-707072755 are [daemon-step-qw2b6-707072755]" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:30.277Z" level=info msg="Workflow step group node daemon-step-qw2b6-1091576052 not yet completed" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:30.277Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:30.277Z" level=info msg=reconcileAgentPod namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:30.277Z" level=info msg="Workflow to be dehydrated" Workflow Size=2803

workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:30.280Z" level=info msg="Workflow to be dehydrated" Workflow Size=36561h-tcvvj-1340600742-agent/deletePod
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:30.291Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=902005874 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:30.309Z" level=info msg="Workflow update successful" namespace=pref-center-dev phase=Running resourceVersion=836386165 workflow=pref-center-
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:38:33.117Z" level=info msg="healthz: current pod is not the leader"

Logs from in your workflow's wait container

time="2024-04-08T13:17:31.231Z" level=info msg="Starting Workflow Executor" version=v3.5.5
time="2024-04-08T13:17:31.235Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2024-04-08T13:17:31.235Z" level=info msg="Executor initialized" deadline="2024-04-08 18:17:29 +0000 UTC" includeScriptOutput=false namespace=argo podName=daemon-step-566zz-daemon-temp-1963735462 templateName=daemon-temp version="&Version{Version:v3.5.5,BuildDate:2024-02-29T20:59:20Z,GitCommit:c80b2e91ebd7e7f604e88442f45ec630380effa0,GitTag:v3.5.5,GitTreeState:clean,GoVersion:go1.21.7,Compiler:gc,Platform:linux/amd64,}"
W0408 13:17:31.249886       1 warnings.go:70] Use tokens from the TokenRequest API or manually created secret-based tokens instead of auto-generated secret-based tokens.
time="2024-04-08T13:17:31.250Z" level=info msg="Starting deadline monitor"
time="2024-04-08T13:18:02.286Z" level=info msg="Main container completed" error="<nil>"
time="2024-04-08T13:18:02.286Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2024-04-08T13:18:02.286Z" level=info msg="No output parameters"
time="2024-04-08T13:18:02.286Z" level=info msg="No output artifacts"
time="2024-04-08T13:18:02.287Z" level=info msg="GCS Save path: /tmp/argo/outputs/logs/main.log, key: daemon-step-566zz/daemon-step-566zz-daemon-temp-1963735462/main.log"
time="2024-04-08T13:18:02.538Z" level=info msg="Save artifact" artifactName=main-logs duration=250.989995ms error="<nil>" key=daemon-step-566zz/daemon-step-566zz-daemon-temp-1963735462/main.log
time="2024-04-08T13:18:02.538Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2024-04-08T13:18:02.538Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
W0408 13:18:02.551995       1 warnings.go:70] Use tokens from the TokenRequest API or manually created secret-based tokens instead of auto-generated secret-based tokens.
W0408 13:18:02.559278       1 warnings.go:70] Use tokens from the TokenRequest API or manually created secret-based tokens instead of auto-generated secret-based tokens.
time="2024-04-08T13:18:02.559Z" level=info msg="Alloc=26115 TotalAlloc=32548 Sys=39781 NumGC=5 Goroutines=11"
W0408 13:18:02.566281       1 warnings.go:70] Use tokens from the TokenRequest API or manually created secret-based tokens instead of auto-generated secret-based tokens.
Stream closed EOF for argo/daemon-step-566zz-daemon-temp-1963735462 (wait)
shuangkun commented 3 months ago

The current design indeed ignores daemon failed. https://github.com/argoproj/argo-workflows/blob/934bf179a703a365a6a5eddfa6edb788492eac79/workflow/controller/operator.go#L1336

KataaOmar commented 3 months ago

@shuangkun Is there a specific purpose for that? And what should we do in this case?

artem-zherdiev-ingio commented 3 months ago

We also have a similar issue, how we can catch daemon failure via hook to send notifications? We have the same in DAG templates.

juliev0 commented 2 months ago

FYI, just added this comment in the PR: https://github.com/argoproj/argo-workflows/pull/12988#issuecomment-2105377768

juliev0 commented 1 month ago

FYI, an explanation for why the existing code is like this and a proposed alternative solution is here.