Failed patching workflowtasksets while role configured properly and no error during execution

[julienteisseire]

Pre-requisites

• [X] I have double-checked my configuration
• [X] I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
• [X] I have searched existing issues and could not find a match for this bug
• [X] I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

Hi,

I submit workflow with http template and 2 http calls with no error.

But while looking at logs of the workflow-controller, I found this error :

time="2024-10-31T10:57:35.216Z" level=debug msg="Patch workflowtasksets/status 403"
time="2024-10-31T10:57:35.217Z" level=warning msg="error updating taskset" error="failed patching taskset: workflowtasksets.argoproj.io \"rinex-ingester-h2xfd\" is forbidden: User \"system:serviceaccount:commanding:workflow-engine-argo-workflows-workflow-controller\" cannot patch resource \"workflowtasksets/status\" in API group \"argoproj.io\" in the namespace \"commanding\"" namespace=commanding workflow=rinex-ingester-h2xfd

But no error during execution / watch, and role properly configured.

Snippet of role configuration :

- apiGroups:
  - argoproj.io
  resources:
  - workflowtasksets/status
  - workflowartifactgctasks/status
  verbs:
  - get
  - list
  - watch
  - create
  - patch

I would like to know why I have this error in workflow-controller logs. Additional question is about CronWorkflows, do I have to understand that step status during workflow execution is checked every 10 seconds ?

time="2024-10-31T10:58:17.967Z" level=debug msg="Syncing all CronWorkflows"
time="2024-10-31T10:58:27.969Z" level=debug msg="Syncing all CronWorkflows"

Thank you workflow.log

Version(s)

v3.5.12

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

Here is the example HTTP template workflow with no private image while logs are with private image, but for you to reproduce.
It does not make any change in my patch error during workflow execution.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: http-template-
  namespace: commanding
  labels:
    workflows.argoproj.io/test: "true"
  annotations:
    workflows.argoproj.io/description: |
      Http template will demostrate http template functionality
    workflows.argoproj.io/version: '>= 3.2.0'
spec:
  entrypoint: main
  serviceAccountName: argo-workflow
  templates:
    - name: main
      steps:
        - - name: good
            template: http
            arguments:
              parameters: [{name: url, value: "https://raw.githubusercontent.com/argoproj/argo-workflows/4e450e250168e6b4d51a126b784e90b11a0162bc/pkg/apis/workflow/v1alpha1/generated.swagger.json"}]
          - name: bad
            template: http
            continueOn:
              failed: true
            arguments:
              parameters: [{name: url, value: "https://raw.githubusercontent.com/argoproj/argo-workflows/thisisnotahash/pkg/apis/workflow/v1alpha1/generated.swagger.json"}]

    - name: http
      inputs:
        parameters:
          - name: url
      http:
       # url: http://dummy.restapiexample.com/api/v1/employees
       url: "{{inputs.parameters.url}}"

Logs from the workflow controller

time="2024-10-31T10:56:45.049Z" level=info msg="Processing workflow" Phase= ResourceVersion=8189 namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:56:45.049Z" level=debug msg="Getting the template by name: main"
time="2024-10-31T10:56:45.049Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-10-31T10:56:45.049Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-10-31T10:56:45.049Z" level=debug msg="Getting the template by name: main" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-10-31T10:56:45.049Z" level=debug msg="Getting the template by name: http-get"
time="2024-10-31T10:56:45.049Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (http-get)"
time="2024-10-31T10:56:45.050Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (http-get)"
time="2024-10-31T10:56:45.050Z" level=debug msg="Getting the template by name: http-get" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (http-get)"
time="2024-10-31T10:56:45.050Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-10-31T10:56:45.050Z" level=debug msg="Getting the template by name: http-get" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (http-get)"
time="2024-10-31T10:56:45.050Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (http-get)"
time="2024-10-31T10:56:45.050Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (http-get)"
time="2024-10-31T10:56:45.050Z" level=debug msg="Getting the template by name: http-get" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (http-get)"
time="2024-10-31T10:56:45.050Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-10-31T10:56:45.050Z" level=debug msg="Getting the template by name: http-put" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (http-get)"
time="2024-10-31T10:56:45.050Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (http-put)"
time="2024-10-31T10:56:45.050Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (http-put)"
time="2024-10-31T10:56:45.050Z" level=debug msg="Getting the template by name: http-put" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (http-put)"
time="2024-10-31T10:56:45.050Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-10-31T10:56:45.050Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-10-31T10:56:45.050Z" level=debug msg="Getting the template by name: http-put" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (http-put)"
time="2024-10-31T10:56:45.050Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (http-put)"
time="2024-10-31T10:56:45.050Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (http-put)"
time="2024-10-31T10:56:45.050Z" level=debug msg="Getting the template by name: http-put" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (http-put)"
time="2024-10-31T10:56:45.050Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-10-31T10:56:45.050Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-10-31T10:56:45.050Z" level=debug msg="Getting the template by name: main" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-10-31T10:56:45.050Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-10-31T10:56:45.050Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-10-31T10:56:45.050Z" level=debug msg="Getting the template by name: main" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-10-31T10:56:45.051Z" level=debug msg="Getting the template by name: http-get" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-10-31T10:56:45.051Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (http-get)"
time="2024-10-31T10:56:45.051Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (http-get)"
time="2024-10-31T10:56:45.051Z" level=debug msg="Getting the template by name: http-get" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (http-get)"
time="2024-10-31T10:56:45.051Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-10-31T10:56:45.051Z" level=debug msg="Getting the template by name: http-put" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (http-get)"
time="2024-10-31T10:56:45.051Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (http-put)"
time="2024-10-31T10:56:45.051Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (http-put)"
time="2024-10-31T10:56:45.051Z" level=debug msg="Getting the template by name: http-put" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (http-put)"
time="2024-10-31T10:56:45.051Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-10-31T10:56:45.051Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-10-31T10:56:45.056Z" level=debug msg="Get configmaps 404"
time="2024-10-31T10:56:45.057Z" level=info msg="resolved artifact repository" artifactRepositoryRef=default-artifact-repository
time="2024-10-31T10:56:45.057Z" level=info msg="Task-result reconciliation" namespace=commanding numObjs=0 workflow=rinex-ingester-h2xfd
time="2024-10-31T10:56:45.057Z" level=debug msg="Skipping artifact GC" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:56:45.057Z" level=info msg="Updated phase  -> Running" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:56:45.057Z" level=debug msg="Evaluating node rinex-ingester-h2xfd: template: *v1alpha1.WorkflowStep (main), boundaryID: , stack depth: 0" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:56:45.057Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:56:45.057Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-10-31T10:56:45.057Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-10-31T10:56:45.057Z" level=debug msg="Getting the template by name: main" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-10-31T10:56:45.057Z" level=debug msg="Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"commanding\", Name:\"rinex-ingester-h2xfd\", UID:\"9ad52e1a-1401-4e94-8fd1-3f37087c8e3e\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"8189\", FieldPath:\"\"}): type: 'Normal' reason: 'WorkflowRunning' Workflow Running"
time="2024-10-31T10:56:45.058Z" level=debug msg="Initializing node rinex-ingester-h2xfd: template: *v1alpha1.WorkflowStep (main), boundaryID: " namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:56:45.058Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:56:45.058Z" level=info msg="Steps node rinex-ingester-h2xfd initialized Running" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:56:45.058Z" level=debug msg="Initializing node rinex-ingester-h2xfd[0]: template: *v1alpha1.WorkflowStep invalid (https://argo-workflows.readthedocs.io/en/release-3.5/templates/), boundaryID: rinex-ingester-h2xfd" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:56:45.058Z" level=info msg="StepGroup node rinex-ingester-h2xfd-392402270 initialized Running" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:56:45.058Z" level=debug msg="Evaluating node rinex-ingester-h2xfd[0].rinex-probe: template: *v1alpha1.WorkflowStep (http-get), boundaryID: rinex-ingester-h2xfd, stack depth: 1" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:56:45.058Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:56:45.058Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (http-get)"
time="2024-10-31T10:56:45.058Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (http-get)"
time="2024-10-31T10:56:45.058Z" level=debug msg="Getting the template by name: http-get" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (http-get)"
time="2024-10-31T10:56:45.058Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-10-31T10:56:45.058Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.NodeStatus (main)"
time="2024-10-31T10:56:45.058Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.NodeStatus (main)"
time="2024-10-31T10:56:45.058Z" level=debug msg="Getting the template by name: main" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.NodeStatus (main)"
time="2024-10-31T10:56:45.058Z" level=debug msg="Initializing node rinex-ingester-h2xfd[0].rinex-probe: template: *v1alpha1.WorkflowStep (http-get), boundaryID: rinex-ingester-h2xfd" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:56:45.058Z" level=info msg="HTTP node rinex-ingester-h2xfd-2643701471 initialized Pending" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:56:45.058Z" level=info msg="Workflow step group node rinex-ingester-h2xfd-392402270 not yet completed" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:56:45.058Z" level=info msg="TaskSet Reconciliation" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:56:45.058Z" level=info msg="Creating TaskSet" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:56:45.058Z" level=debug msg="creating new taskset" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:56:45.071Z" level=debug msg="Create events 201"
time="2024-10-31T10:56:45.071Z" level=debug msg="Create workflowtasksets 201"
time="2024-10-31T10:56:45.071Z" level=info msg=reconcileAgentPod namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:56:45.074Z" level=debug msg="Get secrets 404"
time="2024-10-31T10:56:45.078Z" level=debug msg="Get serviceaccounts 200"
time="2024-10-31T10:56:45.079Z" level=debug msg="Creating Agent pod" namespace=commanding podName=rinex-ingester-h2xfd-1340600742-agent workflow=rinex-ingester-h2xfd
time="2024-10-31T10:56:45.091Z" level=debug msg="Create pods 201"
time="2024-10-31T10:56:45.092Z" level=info msg="Created Agent pod" namespace=commanding podName=rinex-ingester-h2xfd-1340600742-agent workflow=rinex-ingester-h2xfd
time="2024-10-31T10:56:45.092Z" level=info msg=updateAgentPodStatus namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:56:45.092Z" level=info msg=assessAgentPodStatus namespace=commanding podName=rinex-ingester-h2xfd-1340600742-agent
time="2024-10-31T10:56:45.092Z" level=debug msg="Log changes patch: {\"metadata\":{\"annotations\":{\"workflows.argoproj.io/pod-name-format\":\"v2\"},\"labels\":{\"workflows.argoproj.io/completed\":\"false\",\"workflows.argoproj.io/phase\":\"Running\"}},\"status\":{\"artifactGCStatus\":{\"notSpecified\":true},\"artifactRepositoryRef\":{\"artifactRepository\":{\"s3\":{\"accessKeySecret\":{\"key\":\"root-user\",\"name\":\"object-store-minio\"},\"bucket\":\"workflows\",\"endpoint\":\"10.96.58.7:9000\",\"insecure\":true,\"secretKeySecret\":{\"key\":\"root-password\",\"name\":\"object-store-minio\"}}},\"default\":true},\"nodes\":{\"rinex-ingester-h2xfd\":{\"children\":[\"rinex-ingester-h2xfd-392402270\"],\"displayName\":\"rinex-ingester-h2xfd\",\"finishedAt\":null,\"id\":\"rinex-ingester-h2xfd\",\"name\":\"rinex-ingester-h2xfd\",\"phase\":\"Running\",\"startedAt\":\"2024-10-31T10:56:45Z\",\"templateName\":\"main\",\"templateScope\":\"local/rinex-ingester-h2xfd\",\"type\":\"Steps\"},\"rinex-ingester-h2xfd-2643701471\":{\"boundaryID\":\"rinex-ingester-h2xfd\",\"displayName\":\"rinex-probe\",\"finishedAt\":null,\"id\":\"rinex-ingester-h2xfd-2643701471\",\"inputs\":{\"parameters\":[{\"name\":\"url\",\"value\":\"http://commanding-proxy.commanding.svc.cluster.local:7070/rinex/ingester/probe\"}]},\"name\":\"rinex-ingester-h2xfd[0].rinex-probe\",\"phase\":\"Pending\",\"startedAt\":\"2024-10-31T10:56:45Z\",\"templateName\":\"http-get\",\"templateScope\":\"local/rinex-ingester-h2xfd\",\"type\":\"HTTP\"},\"rinex-ingester-h2xfd-392402270\":{\"boundaryID\":\"rinex-ingester-h2xfd\",\"children\":[\"rinex-ingester-h2xfd-2643701471\"],\"displayName\":\"[0]\",\"finishedAt\":null,\"id\":\"rinex-ingester-h2xfd-392402270\",\"name\":\"rinex-ingester-h2xfd[0]\",\"nodeFlag\":{},\"phase\":\"Running\",\"startedAt\":\"2024-10-31T10:56:45Z\",\"templateScope\":\"local/rinex-ingester-h2xfd\",\"type\":\"StepGroup\"}},\"phase\":\"Running\",\"startedAt\":\"2024-10-31T10:56:45Z\"}}"
time="2024-10-31T10:56:45.093Z" level=info msg="Workflow to be dehydrated" Workflow Size=2219
time="2024-10-31T10:56:45.095Z" level=debug msg="Patch workflowtasksets/status 403"
time="2024-10-31T10:56:45.095Z" level=warning msg="error updating taskset" error="failed patching taskset: workflowtasksets.argoproj.io \"rinex-ingester-h2xfd\" is forbidden: User \"system:serviceaccount:commanding:workflow-engine-argo-workflows-workflow-controller\" cannot patch resource \"workflowtasksets/status\" in API group \"argoproj.io\" in the namespace \"commanding\"" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:56:45.105Z" level=debug msg="Update workflows 200"
time="2024-10-31T10:56:45.106Z" level=info msg="Workflow update successful" namespace=commanding phase=Running resourceVersion=8195 workflow=rinex-ingester-h2xfd
time="2024-10-31T10:56:47.954Z" level=debug msg="Syncing all CronWorkflows"
time="2024-10-31T10:56:55.094Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=8195 namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:56:55.094Z" level=info msg="Task-result reconciliation" namespace=commanding numObjs=0 workflow=rinex-ingester-h2xfd
time="2024-10-31T10:56:55.094Z" level=debug msg="Skipping artifact GC" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:56:55.094Z" level=info msg=updateAgentPodStatus namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:56:55.094Z" level=info msg=assessAgentPodStatus namespace=commanding podName=rinex-ingester-h2xfd-1340600742-agent
time="2024-10-31T10:56:55.094Z" level=error msg="was unable to obtain node for rinex-ingester-h2xfd-2166136261" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:56:55.094Z" level=debug msg="Evaluating node rinex-ingester-h2xfd: template: *v1alpha1.WorkflowStep (main), boundaryID: , stack depth: 0" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:56:55.095Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-10-31T10:56:55.095Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-10-31T10:56:55.095Z" level=debug msg="Getting the template by name: main" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-10-31T10:56:55.095Z" level=debug msg="Executing node rinex-ingester-h2xfd of Steps is Running" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:56:55.096Z" level=debug msg="Evaluating node rinex-ingester-h2xfd[0].rinex-probe: template: *v1alpha1.WorkflowStep (http-get), boundaryID: rinex-ingester-h2xfd, stack depth: 1" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:56:55.096Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (http-get)"
time="2024-10-31T10:56:55.096Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (http-get)"
time="2024-10-31T10:56:55.096Z" level=debug msg="Getting the template by name: http-get" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (http-get)"
time="2024-10-31T10:56:55.096Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-10-31T10:56:55.096Z" level=debug msg="Executing node rinex-ingester-h2xfd[0].rinex-probe of HTTP is Pending" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:56:55.096Z" level=info msg="Workflow step group node rinex-ingester-h2xfd-392402270 not yet completed" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:56:55.096Z" level=info msg="TaskSet Reconciliation" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:56:55.096Z" level=info msg="Creating TaskSet" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:56:55.096Z" level=debug msg="creating new taskset" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:56:55.108Z" level=debug msg="Create workflowtasksets 409"
time="2024-10-31T10:56:55.109Z" level=debug msg="patching the exiting taskset" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:56:55.116Z" level=debug msg="Patch workflowtasksets 200"
time="2024-10-31T10:56:55.117Z" level=info msg=reconcileAgentPod namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:56:55.117Z" level=debug msg="Skipped pod creation: already exists" namespace=commanding podName=rinex-ingester-h2xfd-1340600742-agent podPhase=Running workflow=rinex-ingester-h2xfd
time="2024-10-31T10:56:55.117Z" level=info msg=updateAgentPodStatus namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:56:55.117Z" level=info msg=assessAgentPodStatus namespace=commanding podName=rinex-ingester-h2xfd-1340600742-agent
time="2024-10-31T10:56:57.955Z" level=debug msg="Syncing all CronWorkflows"
time="2024-10-31T10:57:01.356Z" level=debug msg="Watch cronworkflows 200"
time="2024-10-31T10:57:05.117Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=8195 namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:05.117Z" level=info msg="Task-result reconciliation" namespace=commanding numObjs=0 workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:05.117Z" level=debug msg="Skipping artifact GC" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:05.117Z" level=info msg=updateAgentPodStatus namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:05.118Z" level=info msg=assessAgentPodStatus namespace=commanding podName=rinex-ingester-h2xfd-1340600742-agent
time="2024-10-31T10:57:05.118Z" level=error msg="was unable to obtain node for rinex-ingester-h2xfd-2166136261" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:05.118Z" level=debug msg="Evaluating node rinex-ingester-h2xfd: template: *v1alpha1.WorkflowStep (main), boundaryID: , stack depth: 0" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:05.118Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-10-31T10:57:05.118Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-10-31T10:57:05.118Z" level=debug msg="Getting the template by name: main" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-10-31T10:57:05.118Z" level=debug msg="Executing node rinex-ingester-h2xfd of Steps is Running" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:05.119Z" level=debug msg="Evaluating node rinex-ingester-h2xfd[0].rinex-probe: template: *v1alpha1.WorkflowStep (http-get), boundaryID: rinex-ingester-h2xfd, stack depth: 1" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:05.119Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (http-get)"
time="2024-10-31T10:57:05.119Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (http-get)"
time="2024-10-31T10:57:05.119Z" level=debug msg="Getting the template by name: http-get" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (http-get)"
time="2024-10-31T10:57:05.119Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-10-31T10:57:05.119Z" level=debug msg="Executing node rinex-ingester-h2xfd[0].rinex-probe of HTTP is Pending" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:05.119Z" level=info msg="Workflow step group node rinex-ingester-h2xfd-392402270 not yet completed" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:05.119Z" level=info msg="TaskSet Reconciliation" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:05.119Z" level=info msg="Creating TaskSet" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:05.119Z" level=debug msg="creating new taskset" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:05.129Z" level=debug msg="Create workflowtasksets 409"
time="2024-10-31T10:57:05.130Z" level=debug msg="patching the exiting taskset" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:05.135Z" level=debug msg="Patch workflowtasksets 200"
time="2024-10-31T10:57:05.136Z" level=info msg=reconcileAgentPod namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:05.136Z" level=debug msg="Skipped pod creation: already exists" namespace=commanding podName=rinex-ingester-h2xfd-1340600742-agent podPhase=Running workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:05.136Z" level=info msg=updateAgentPodStatus namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:05.136Z" level=info msg=assessAgentPodStatus namespace=commanding podName=rinex-ingester-h2xfd-1340600742-agent
time="2024-10-31T10:57:05.136Z" level=debug msg="Log changes patch: {\"status\":{\"conditions\":[{\"status\":\"False\",\"type\":\"PodRunning\"}],\"nodes\":{\"rinex-ingester-h2xfd-2643701471\":{\"finishedAt\":\"2024-10-31T10:57:05Z\",\"outputs\":{\"result\":\"OK\\n\"},\"phase\":\"Succeeded\"}}}}"
time="2024-10-31T10:57:05.136Z" level=info msg="Workflow to be dehydrated" Workflow Size=2508
time="2024-10-31T10:57:05.139Z" level=debug msg="Patch workflowtasksets/status 403"
time="2024-10-31T10:57:05.139Z" level=warning msg="error updating taskset" error="failed patching taskset: workflowtasksets.argoproj.io \"rinex-ingester-h2xfd\" is forbidden: User \"system:serviceaccount:commanding:workflow-engine-argo-workflows-workflow-controller\" cannot patch resource \"workflowtasksets/status\" in API group \"argoproj.io\" in the namespace \"commanding\"" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:05.145Z" level=debug msg="Update workflows 200"
time="2024-10-31T10:57:05.146Z" level=info msg="Workflow update successful" namespace=commanding phase=Running resourceVersion=8234 workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:07.955Z" level=debug msg="Syncing all CronWorkflows"
time="2024-10-31T10:57:14.247Z" level=debug msg="Watch workflowtaskresults 200"
time="2024-10-31T10:57:15.148Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=8234 namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:15.148Z" level=info msg="Task-result reconciliation" namespace=commanding numObjs=0 workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:15.148Z" level=debug msg="Skipping artifact GC" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:15.148Z" level=info msg=updateAgentPodStatus namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:15.148Z" level=info msg=assessAgentPodStatus namespace=commanding podName=rinex-ingester-h2xfd-1340600742-agent
time="2024-10-31T10:57:15.148Z" level=error msg="was unable to obtain node for rinex-ingester-h2xfd-2166136261" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:15.148Z" level=debug msg="Evaluating node rinex-ingester-h2xfd: template: *v1alpha1.WorkflowStep (main), boundaryID: , stack depth: 0" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:15.148Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-10-31T10:57:15.148Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-10-31T10:57:15.148Z" level=debug msg="Getting the template by name: main" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-10-31T10:57:15.149Z" level=debug msg="Executing node rinex-ingester-h2xfd of Steps is Running" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:15.149Z" level=debug msg="Evaluating node rinex-ingester-h2xfd[0].rinex-probe: template: *v1alpha1.WorkflowStep (http-get), boundaryID: rinex-ingester-h2xfd, stack depth: 1" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:15.149Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (http-get)"
time="2024-10-31T10:57:15.149Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (http-get)"
time="2024-10-31T10:57:15.149Z" level=debug msg="Getting the template by name: http-get" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (http-get)"
time="2024-10-31T10:57:15.149Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-10-31T10:57:15.149Z" level=debug msg="Node rinex-ingester-h2xfd[0].rinex-probe already completed" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:15.149Z" level=info msg="Step group node rinex-ingester-h2xfd-392402270 successful" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:15.149Z" level=info msg="node rinex-ingester-h2xfd-392402270 phase Running -> Succeeded" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:15.149Z" level=info msg="node rinex-ingester-h2xfd-392402270 finished: 2024-10-31 10:57:15.149486026 +0000 UTC" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:15.149Z" level=debug msg="Initializing node rinex-ingester-h2xfd[1]: template: *v1alpha1.WorkflowStep invalid (https://argo-workflows.readthedocs.io/en/release-3.5/templates/), boundaryID: rinex-ingester-h2xfd" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:15.149Z" level=info msg="StepGroup node rinex-ingester-h2xfd-325144699 initialized Running" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:15.149Z" level=info msg="SG Outbound nodes of rinex-ingester-h2xfd-2643701471 are [rinex-ingester-h2xfd-2643701471]" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:15.149Z" level=debug msg="Evaluating node rinex-ingester-h2xfd[1].rinex-start: template: *v1alpha1.WorkflowStep (http-put), boundaryID: rinex-ingester-h2xfd, stack depth: 1" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:15.149Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:15.149Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (http-put)"
time="2024-10-31T10:57:15.149Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (http-put)"
time="2024-10-31T10:57:15.149Z" level=debug msg="Getting the template by name: http-put" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (http-put)"
time="2024-10-31T10:57:15.149Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-10-31T10:57:15.150Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-10-31T10:57:15.150Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.NodeStatus (main)"
time="2024-10-31T10:57:15.150Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.NodeStatus (main)"
time="2024-10-31T10:57:15.150Z" level=debug msg="Getting the template by name: main" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.NodeStatus (main)"
time="2024-10-31T10:57:15.150Z" level=debug msg="Initializing node rinex-ingester-h2xfd[1].rinex-start: template: *v1alpha1.WorkflowStep (http-put), boundaryID: rinex-ingester-h2xfd" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:15.150Z" level=info msg="HTTP node rinex-ingester-h2xfd-1776136928 initialized Pending" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:15.150Z" level=info msg="Workflow step group node rinex-ingester-h2xfd-325144699 not yet completed" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:15.150Z" level=info msg="TaskSet Reconciliation" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:15.150Z" level=info msg="Creating TaskSet" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:15.150Z" level=debug msg="creating new taskset" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:15.166Z" level=debug msg="Create workflowtasksets 409"
time="2024-10-31T10:57:15.166Z" level=debug msg="patching the exiting taskset" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:15.175Z" level=debug msg="Patch workflowtasksets 200"
time="2024-10-31T10:57:15.175Z" level=info msg=reconcileAgentPod namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:15.175Z" level=debug msg="Skipped pod creation: already exists" namespace=commanding podName=rinex-ingester-h2xfd-1340600742-agent podPhase=Running workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:15.175Z" level=info msg=updateAgentPodStatus namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:15.175Z" level=info msg=assessAgentPodStatus namespace=commanding podName=rinex-ingester-h2xfd-1340600742-agent
time="2024-10-31T10:57:15.177Z" level=debug msg="Log changes patch: {\"status\":{\"nodes\":{\"rinex-ingester-h2xfd-1776136928\":{\"boundaryID\":\"rinex-ingester-h2xfd\",\"displayName\":\"rinex-start\",\"finishedAt\":null,\"id\":\"rinex-ingester-h2xfd-1776136928\",\"inputs\":{\"parameters\":[{\"name\":\"url\",\"value\":\"http://commanding-proxy.commanding.svc.cluster.local:7070/rinex/ingester/start\"},{\"name\":\"request\",\"value\":\"{\\n  \\\"startRequeststring\\\": \\\"Start Request\\\"\\n}\\n\"}]},\"name\":\"rinex-ingester-h2xfd[1].rinex-start\",\"phase\":\"Pending\",\"startedAt\":\"2024-10-31T10:57:15Z\",\"templateName\":\"http-put\",\"templateScope\":\"local/rinex-ingester-h2xfd\",\"type\":\"HTTP\"},\"rinex-ingester-h2xfd-2643701471\":{\"children\":[\"rinex-ingester-h2xfd-325144699\"],\"finishedAt\":\"2024-10-31T10:57:15Z\"},\"rinex-ingester-h2xfd-325144699\":{\"boundaryID\":\"rinex-ingester-h2xfd\",\"children\":[\"rinex-ingester-h2xfd-1776136928\"],\"displayName\":\"[1]\",\"finishedAt\":null,\"id\":\"rinex-ingester-h2xfd-325144699\",\"name\":\"rinex-ingester-h2xfd[1]\",\"nodeFlag\":{},\"phase\":\"Running\",\"startedAt\":\"2024-10-31T10:57:15Z\",\"templateScope\":\"local/rinex-ingester-h2xfd\",\"type\":\"StepGroup\"},\"rinex-ingester-h2xfd-392402270\":{\"finishedAt\":\"2024-10-31T10:57:15Z\",\"phase\":\"Succeeded\"}}}}"
time="2024-10-31T10:57:15.177Z" level=info msg="Workflow to be dehydrated" Workflow Size=3170
time="2024-10-31T10:57:15.180Z" level=debug msg="Patch workflowtasksets/status 403"
time="2024-10-31T10:57:15.180Z" level=warning msg="error updating taskset" error="failed patching taskset: workflowtasksets.argoproj.io \"rinex-ingester-h2xfd\" is forbidden: User \"system:serviceaccount:commanding:workflow-engine-argo-workflows-workflow-controller\" cannot patch resource \"workflowtasksets/status\" in API group \"argoproj.io\" in the namespace \"commanding\"" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:15.188Z" level=debug msg="Update workflows 200"
time="2024-10-31T10:57:15.189Z" level=info msg="Workflow update successful" namespace=commanding phase=Running resourceVersion=8249 workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:17.959Z" level=debug msg="Syncing all CronWorkflows"
time="2024-10-31T10:57:25.177Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=8249 namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:25.177Z" level=info msg="Task-result reconciliation" namespace=commanding numObjs=0 workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:25.177Z" level=debug msg="Skipping artifact GC" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:25.177Z" level=info msg=updateAgentPodStatus namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:25.177Z" level=info msg=assessAgentPodStatus namespace=commanding podName=rinex-ingester-h2xfd-1340600742-agent
time="2024-10-31T10:57:25.177Z" level=error msg="was unable to obtain node for rinex-ingester-h2xfd-2166136261" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:25.177Z" level=debug msg="Evaluating node rinex-ingester-h2xfd: template: *v1alpha1.WorkflowStep (main), boundaryID: , stack depth: 0" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:25.177Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-10-31T10:57:25.177Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-10-31T10:57:25.177Z" level=debug msg="Getting the template by name: main" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-10-31T10:57:25.177Z" level=debug msg="Executing node rinex-ingester-h2xfd of Steps is Running" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:25.177Z" level=debug msg="Step group node &NodeStatus{ID:rinex-ingester-h2xfd-392402270,Name:rinex-ingester-h2xfd[0],DisplayName:[0],Type:StepGroup,TemplateName:,TemplateRef:nil,Phase:Succeeded,BoundaryID:rinex-ingester-h2xfd,Message:,StartedAt:2024-10-31 10:56:45 +0000 UTC,FinishedAt:2024-10-31 10:57:15 +0000 UTC,PodIP:,Daemoned:nil,Inputs:nil,Outputs:nil,Children:[rinex-ingester-h2xfd-2643701471],OutboundNodes:[],TemplateScope:local/rinex-ingester-h2xfd,ResourcesDuration:ResourcesDuration{},HostNodeName:,MemoizationStatus:nil,EstimatedDuration:0,SynchronizationStatus:nil,Progress:1/2,NodeFlag:&NodeFlag{Hooked:false,Retried:false,},} already marked completed" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:25.177Z" level=info msg="SG Outbound nodes of rinex-ingester-h2xfd-2643701471 are [rinex-ingester-h2xfd-2643701471]" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:25.177Z" level=debug msg="Evaluating node rinex-ingester-h2xfd[1].rinex-start: template: *v1alpha1.WorkflowStep (http-put), boundaryID: rinex-ingester-h2xfd, stack depth: 1" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:25.177Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (http-put)"
time="2024-10-31T10:57:25.177Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (http-put)"
time="2024-10-31T10:57:25.177Z" level=debug msg="Getting the template by name: http-put" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (http-put)"
time="2024-10-31T10:57:25.177Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-10-31T10:57:25.177Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-10-31T10:57:25.178Z" level=debug msg="Executing node rinex-ingester-h2xfd[1].rinex-start of HTTP is Pending" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:25.178Z" level=info msg="Workflow step group node rinex-ingester-h2xfd-325144699 not yet completed" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:25.178Z" level=info msg="TaskSet Reconciliation" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:25.178Z" level=info msg="Creating TaskSet" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:25.178Z" level=debug msg="creating new taskset" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:25.184Z" level=debug msg="Create workflowtasksets 409"
time="2024-10-31T10:57:25.185Z" level=debug msg="patching the exiting taskset" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:25.191Z" level=debug msg="Patch workflowtasksets 200"
time="2024-10-31T10:57:25.191Z" level=info msg=reconcileAgentPod namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:25.191Z" level=debug msg="Skipped pod creation: already exists" namespace=commanding podName=rinex-ingester-h2xfd-1340600742-agent podPhase=Running workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:25.191Z" level=info msg=updateAgentPodStatus namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:25.191Z" level=info msg=assessAgentPodStatus namespace=commanding podName=rinex-ingester-h2xfd-1340600742-agent
time="2024-10-31T10:57:25.192Z" level=debug msg="Log changes patch: {\"status\":{\"nodes\":{\"rinex-ingester-h2xfd-1776136928\":{\"finishedAt\":\"2024-10-31T10:57:25Z\",\"outputs\":{\"result\":\"OK Start Request\\n\"},\"phase\":\"Succeeded\"},\"rinex-ingester-h2xfd-2643701471\":{\"finishedAt\":\"2024-10-31T10:57:25Z\"}}}}"
time="2024-10-31T10:57:25.192Z" level=info msg="Workflow to be dehydrated" Workflow Size=3201
time="2024-10-31T10:57:25.195Z" level=debug msg="Patch workflowtasksets/status 403"
time="2024-10-31T10:57:25.195Z" level=warning msg="error updating taskset" error="failed patching taskset: workflowtasksets.argoproj.io \"rinex-ingester-h2xfd\" is forbidden: User \"system:serviceaccount:commanding:workflow-engine-argo-workflows-workflow-controller\" cannot patch resource \"workflowtasksets/status\" in API group \"argoproj.io\" in the namespace \"commanding\"" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:25.205Z" level=debug msg="Update workflows 200"
time="2024-10-31T10:57:25.206Z" level=info msg="Workflow update successful" namespace=commanding phase=Running resourceVersion=8264 workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:27.960Z" level=debug msg="Syncing all CronWorkflows"
time="2024-10-31T10:57:35.209Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=8264 namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:35.209Z" level=info msg="Task-result reconciliation" namespace=commanding numObjs=0 workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:35.209Z" level=debug msg="Skipping artifact GC" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:35.209Z" level=info msg=updateAgentPodStatus namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:35.210Z" level=info msg=assessAgentPodStatus namespace=commanding podName=rinex-ingester-h2xfd-1340600742-agent
time="2024-10-31T10:57:35.210Z" level=error msg="was unable to obtain node for rinex-ingester-h2xfd-2166136261" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:35.210Z" level=debug msg="Evaluating node rinex-ingester-h2xfd: template: *v1alpha1.WorkflowStep (main), boundaryID: , stack depth: 0" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:35.210Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-10-31T10:57:35.210Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-10-31T10:57:35.210Z" level=debug msg="Getting the template by name: main" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-10-31T10:57:35.210Z" level=debug msg="Executing node rinex-ingester-h2xfd of Steps is Running" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:35.211Z" level=debug msg="Step group node &NodeStatus{ID:rinex-ingester-h2xfd-392402270,Name:rinex-ingester-h2xfd[0],DisplayName:[0],Type:StepGroup,TemplateName:,TemplateRef:nil,Phase:Succeeded,BoundaryID:rinex-ingester-h2xfd,Message:,StartedAt:2024-10-31 10:56:45 +0000 UTC,FinishedAt:2024-10-31 10:57:15 +0000 UTC,PodIP:,Daemoned:nil,Inputs:nil,Outputs:nil,Children:[rinex-ingester-h2xfd-2643701471],OutboundNodes:[],TemplateScope:local/rinex-ingester-h2xfd,ResourcesDuration:ResourcesDuration{},HostNodeName:,MemoizationStatus:nil,EstimatedDuration:0,SynchronizationStatus:nil,Progress:2/2,NodeFlag:&NodeFlag{Hooked:false,Retried:false,},} already marked completed" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:35.211Z" level=info msg="SG Outbound nodes of rinex-ingester-h2xfd-2643701471 are [rinex-ingester-h2xfd-2643701471]" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:35.211Z" level=debug msg="Evaluating node rinex-ingester-h2xfd[1].rinex-start: template: *v1alpha1.WorkflowStep (http-put), boundaryID: rinex-ingester-h2xfd, stack depth: 1" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:35.211Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (http-put)"
time="2024-10-31T10:57:35.211Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (http-put)"
time="2024-10-31T10:57:35.211Z" level=debug msg="Getting the template by name: http-put" base="*v1alpha1.Workflow (namespace=commanding,name=rinex-ingester-h2xfd)" tmpl="*v1alpha1.WorkflowStep (http-put)"
time="2024-10-31T10:57:35.211Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-10-31T10:57:35.211Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-10-31T10:57:35.211Z" level=debug msg="Node rinex-ingester-h2xfd[1].rinex-start already completed" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:35.211Z" level=info msg="Step group node rinex-ingester-h2xfd-325144699 successful" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:35.211Z" level=info msg="node rinex-ingester-h2xfd-325144699 phase Running -> Succeeded" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:35.211Z" level=info msg="node rinex-ingester-h2xfd-325144699 finished: 2024-10-31 10:57:35.211675288 +0000 UTC" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:35.211Z" level=info msg="Outbound nodes of rinex-ingester-h2xfd-1776136928 is [rinex-ingester-h2xfd-1776136928]" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:35.211Z" level=info msg="Outbound nodes of rinex-ingester-h2xfd is [rinex-ingester-h2xfd-1776136928]" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:35.211Z" level=info msg="node rinex-ingester-h2xfd phase Running -> Succeeded" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:35.211Z" level=info msg="node rinex-ingester-h2xfd finished: 2024-10-31 10:57:35.211751288 +0000 UTC" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:35.211Z" level=debug msg="Checking daemoned children of rinex-ingester-h2xfd" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:35.211Z" level=info msg="TaskSet Reconciliation" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:35.211Z" level=info msg=reconcileAgentPod namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:35.211Z" level=debug msg="Task results completion status: map[]" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:35.211Z" level=info msg="Updated phase Running -> Succeeded" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:35.212Z" level=info msg="Marking workflow completed" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:35.212Z" level=debug msg="Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"commanding\", Name:\"rinex-ingester-h2xfd\", UID:\"9ad52e1a-1401-4e94-8fd1-3f37087c8e3e\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"8264\", FieldPath:\"\"}): type: 'Normal' reason: 'WorkflowSucceeded' Workflow completed"
time="2024-10-31T10:57:35.213Z" level=debug msg="Log changes patch: {\"metadata\":{\"labels\":{\"workflows.argoproj.io/completed\":\"true\",\"workflows.argoproj.io/phase\":\"Succeeded\"}},\"status\":{\"conditions\":[{\"status\":\"False\",\"type\":\"PodRunning\"},{\"status\":\"True\",\"type\":\"Completed\"}],\"finishedAt\":\"2024-10-31T10:57:35Z\",\"nodes\":{\"rinex-ingester-h2xfd\":{\"finishedAt\":\"2024-10-31T10:57:35Z\",\"outboundNodes\":[\"rinex-ingester-h2xfd-1776136928\"],\"phase\":\"Succeeded\"},\"rinex-ingester-h2xfd-1776136928\":{\"finishedAt\":\"2024-10-31T10:57:35Z\"},\"rinex-ingester-h2xfd-2643701471\":{\"finishedAt\":\"2024-10-31T10:57:35Z\"},\"rinex-ingester-h2xfd-325144699\":{\"finishedAt\":\"2024-10-31T10:57:35Z\",\"phase\":\"Succeeded\"}},\"phase\":\"Succeeded\"}}"
time="2024-10-31T10:57:35.213Z" level=info msg="Workflow to be dehydrated" Workflow Size=3287
time="2024-10-31T10:57:35.216Z" level=debug msg="Patch workflowtasksets/status 403"
time="2024-10-31T10:57:35.217Z" level=warning msg="error updating taskset" error="failed patching taskset: workflowtasksets.argoproj.io \"rinex-ingester-h2xfd\" is forbidden: User \"system:serviceaccount:commanding:workflow-engine-argo-workflows-workflow-controller\" cannot patch resource \"workflowtasksets/status\" in API group \"argoproj.io\" in the namespace \"commanding\"" namespace=commanding workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:35.217Z" level=info msg="cleaning up pod" action=deletePod key=commanding/rinex-ingester-h2xfd-1340600742-agent/deletePod
time="2024-10-31T10:57:35.223Z" level=debug msg="Create events 201"
time="2024-10-31T10:57:35.229Z" level=debug msg="Delete pods 200"
time="2024-10-31T10:57:35.232Z" level=debug msg="Update workflows 200"
time="2024-10-31T10:57:35.234Z" level=info msg="Workflow update successful" namespace=commanding phase=Succeeded resourceVersion=8280 workflow=rinex-ingester-h2xfd
time="2024-10-31T10:57:35.247Z" level=debug msg="DeleteCollection workflowtaskresults 200"
time="2024-10-31T10:57:37.387Z" level=debug msg="Check the workflow existence"
time="2024-10-31T10:57:37.960Z" level=debug msg="Syncing all CronWorkflows"
time="2024-10-31T10:57:45.233Z" level=debug msg="Won't process Workflow since it's completed" key=commanding/rinex-ingester-h2xfd
time="2024-10-31T10:57:47.962Z" level=debug msg="Syncing all CronWorkflows"
time="2024-10-31T10:57:57.965Z" level=debug msg="Syncing all CronWorkflows"
time="2024-10-31T10:58:06.238Z" level=debug msg="Watch workflowtemplates 200"
time="2024-10-31T10:58:07.966Z" level=debug msg="Syncing all CronWorkflows"
time="2024-10-31T10:58:15.444Z" level=debug msg="Won't process Workflow since it's completed" key=commanding/rinex-ingester-h2xfd
time="2024-10-31T10:58:17.967Z" level=debug msg="Syncing all CronWorkflows"
time="2024-10-31T10:58:27.969Z" level=debug msg="Syncing all CronWorkflows"

Logs from in your workflow's wait container

kubectl logs -n commanding -c wait -l workflows.argoproj.io/workflow=rinex-ingester-6z6mt,workflow.argoproj.io/phase!=Succeeded
error: container wait is not valid for pod rinex-ingester-6z6mt-1340600742-agent

[julienteisseire]

Hi again,

I fixed my problem by setting service account (the same created by workflow.serviceAccount) to workflow-controller in my values.yaml :

controller:
  serviceAccount:
    create: false
    name: "argo-workflow"