search

argoproj / argo-workflows

Workflow Engine for Kubernetes
https://argo-workflows.readthedocs.io/
Apache License 2.0
15.1k stars 3.2k forks source link

DAG task inline another DAG will be cause an error #10235

Open Tan-JiaLiang opened 1 year ago

Tan-JiaLiang commented 1 year ago

Pre-requisites

What happened/what you expected to happen?

When i submit a dag inline dag workflow, the workflow will be in error phase, but the DAG node in Running phase. image

this is the error message image

Version

v3.4.1

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: dag-inline-
spec:
  entrypoint: main
  templates:
    - name: main
      dag:
        tasks:
          - name: inline-dag
            inline:
              dag:
                tasks:
                  - name: say-hello
                    inline:
                      script:
                        image: python:alpine3.6
                        command: [python]
                        source: |
                          print('hello')

Logs from the workflow controller

time="2022-12-15T17:32:15.901Z" level=info msg="Processing workflow" namespace=argo-workflow workflow=dag-inline-gl6j8
time="2022-12-15T17:32:15.901Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo-workflow,name=dag-inline-gl6j8)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2022-12-15T17:32:15.901Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo-workflow,name=dag-inline-gl6j8)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2022-12-15T17:32:15.901Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=argo-workflow,name=dag-inline-gl6j8)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2022-12-15T17:32:15.902Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo-workflow,name=dag-inline-gl6j8)" depth=0 tmpl="*v1alpha1.DAGTask inlined"
time="2022-12-15T17:32:15.902Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo-workflow,name=dag-inline-gl6j8)" depth=0 tmpl="*v1alpha1.DAGTask inlined"
time="2022-12-15T17:32:15.902Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo-workflow,name=dag-inline-gl6j8)" depth=0 tmpl="*v1alpha1.DAGTask inlined"
time="2022-12-15T17:32:15.902Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo-workflow,name=dag-inline-gl6j8)" depth=0 tmpl="*v1alpha1.DAGTask inlined"
time="2022-12-15T17:32:15.902Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo-workflow,name=dag-inline-gl6j8)" depth=0 tmpl="*v1alpha1.DAGTask inlined"
time="2022-12-15T17:32:15.902Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo-workflow,name=dag-inline-gl6j8)" depth=0 tmpl="*v1alpha1.DAGTask inlined"
time="2022-12-15T17:32:15.902Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo-workflow,name=dag-inline-gl6j8)" depth=0 tmpl="*v1alpha1.DAGTask inlined"
time="2022-12-15T17:32:15.902Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo-workflow,name=dag-inline-gl6j8)" depth=0 tmpl="*v1alpha1.DAGTask inlined"
time="2022-12-15T17:32:15.902Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=argo-workflow,name=dag-inline-gl6j8)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2022-12-15T17:32:15.902Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo-workflow,name=dag-inline-gl6j8)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2022-12-15T17:32:15.902Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo-workflow,name=dag-inline-gl6j8)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2022-12-15T17:32:15.902Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=argo-workflow,name=dag-inline-gl6j8)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2022-12-15T17:32:15.910Z" level=info msg="Updated phase  -> Running" namespace=argo-workflow workflow=dag-inline-gl6j8
time="2022-12-15T17:32:15.910Z" level=debug msg="Evaluating node dag-inline-gl6j8: template: *v1alpha1.WorkflowStep (main), boundaryID: " namespace=argo-workflow workflow=dag-inline-gl6j8
time="2022-12-15T17:32:15.910Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo-workflow,name=dag-inline-gl6j8)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2022-12-15T17:32:15.910Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo-workflow,name=dag-inline-gl6j8)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2022-12-15T17:32:15.910Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=argo-workflow,name=dag-inline-gl6j8)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2022-12-15T17:32:15.910Z" level=debug msg="Initializing node dag-inline-gl6j8: template: *v1alpha1.WorkflowStep (main), boundaryID: " namespace=argo-workflow workflow=dag-inline-gl6j8
time="2022-12-15T17:32:15.910Z" level=info msg="DAG node dag-inline-gl6j8 initialized Running" namespace=argo-workflow workflow=dag-inline-gl6j8
time="2022-12-15T17:32:15.910Z" level=info msg="All of node dag-inline-gl6j8.inline-dag dependencies [] completed" namespace=argo-workflow workflow=dag-inline-gl6j8
time="2022-12-15T17:32:15.910Z" level=debug msg="Evaluating node dag-inline-gl6j8.inline-dag: template: *v1alpha1.DAGTask inlined, boundaryID: dag-inline-gl6j8" namespace=argo-workflow workflow=dag-inline-gl6j8
time="2022-12-15T17:32:15.910Z" level=debug msg="Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"argo-workflow\", Name:\"dag-inline-gl6j8\", UID:\"537e587a-3ef6-4474-aabc-5eb50db24f38\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"34230680608\", FieldPath:\"\"}): type: 'Normal' reason: 'WorkflowRunning' Workflow Running"
time="2022-12-15T17:32:15.910Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo-workflow,name=dag-inline-gl6j8)" depth=0 tmpl="*v1alpha1.DAGTask inlined"
time="2022-12-15T17:32:15.910Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo-workflow,name=dag-inline-gl6j8)" depth=0 tmpl="*v1alpha1.DAGTask inlined"
time="2022-12-15T17:32:15.910Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo-workflow,name=dag-inline-gl6j8)" depth=0 tmpl="*v1alpha1.NodeStatus (main)"
time="2022-12-15T17:32:15.910Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo-workflow,name=dag-inline-gl6j8)" depth=0 tmpl="*v1alpha1.NodeStatus (main)"
time="2022-12-15T17:32:15.910Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=argo-workflow,name=dag-inline-gl6j8)" depth=0 tmpl="*v1alpha1.NodeStatus (main)"
time="2022-12-15T17:32:15.910Z" level=debug msg="Initializing node dag-inline-gl6j8.inline-dag: template: *v1alpha1.DAGTask inlined, boundaryID: dag-inline-gl6j8" namespace=argo-workflow workflow=dag-inline-gl6j8
time="2022-12-15T17:32:15.910Z" level=info msg="DAG node dag-inline-gl6j8-2586533071 initialized Running" namespace=argo-workflow workflow=dag-inline-gl6j8
time="2022-12-15T17:32:15.910Z" level=info msg="All of node dag-inline-gl6j8.inline-dag.say-hello dependencies [] completed" namespace=argo-workflow workflow=dag-inline-gl6j8
time="2022-12-15T17:32:15.910Z" level=debug msg="Evaluating node dag-inline-gl6j8.inline-dag.say-hello: template: *v1alpha1.DAGTask inlined, boundaryID: dag-inline-gl6j8-2586533071" namespace=argo-workflow workflow=dag-inline-gl6j8
time="2022-12-15T17:32:15.910Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo-workflow,name=dag-inline-gl6j8)" depth=0 tmpl="*v1alpha1.DAGTask inlined"
time="2022-12-15T17:32:15.910Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo-workflow,name=dag-inline-gl6j8)" depth=0 tmpl="*v1alpha1.DAGTask inlined"
time="2022-12-15T17:32:15.910Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo-workflow,name=dag-inline-gl6j8)" depth=0 tmpl="*v1alpha1.NodeStatus invalid (https://argoproj.github.io/argo-workflows/templates/)"
time="2022-12-15T17:32:15.910Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo-workflow,name=dag-inline-gl6j8)" depth=0 tmpl="*v1alpha1.NodeStatus invalid (https://argoproj.github.io/argo-workflows/templates/)"
time="2022-12-15T17:32:15.910Z" level=error msg="Mark error node" error="task 'dag-inline-gl6j8.inline-dag.say-hello' errored: failed to get a template" namespace=argo-workflow nodeName=dag-inline-gl6j8.inline-dag.say-hello workflow=dag-inline-gl6j8
time="2022-12-15T17:32:15.910Z" level=error msg="Recovered from panic" namespace=argo-workflow r="workflow 'dag-inline-gl6j8' node 'dag-inline-gl6j8.inline-dag.say-hello' uninitialized when marking as Error: [task 'dag-inline-gl6j8.inline-dag.say-hello' errored: failed to get a template]" stack="goroutine 296 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x65\ngithub.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).operate.func2()\n\t/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:190 +0xbc\npanic({0x1d64280, 0xc001b57200})\n\t/usr/local/go/src/runtime/panic.go:844 +0x258\ngithub.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).markNodePhase(0xc001214160, {0xc0025c7710, 0x25}, {0x21508b1, 0x5}, {0xc001b57180?, 0x1, 0x1})\n\t/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:2300 +0xaba\ngithub.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).markNodeError(0xc001214160, {0xc0025c7710, 0x25}, {0x249d460?, 0xc001b570e0})\n\t/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:2412 +0x2fa\ngithub.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).executeDAGTask(0xc001214160, {0x24c4430, 0xc000058028}, 0xc003db1f80, {0xc002fd7630, 0x9})\n\t/go/src/github.com/argoproj/argo-workflows/workflow/controller/dag.go:528 +0x1c58\ngithub.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).executeDAG(0xc001214160, {0x24c4430, 0xc000058028}, {0xc003c1cee0, 0x1b}, 0xc0028ebbc0, {0xc0019e9458, 0x16}, 0xc002f69d40, {0x24c7150, ...}, ...)\n\t/go/src/github.com/argoproj/argo-workflows/workflow/controller/dag.go:244 +0xf05\ngithub.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).executeTemplate(0xc001214160, {0x24c4430, 0xc000058028}, {0xc003c1cee0, 0x1b}, {0x24c7150, 0xc00244b0e0?}, 0x0?, {{0x0, 0x0, ...}, ...}, ...)\n\t/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:1944 +0x240c\ngithub.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).executeDAGTask(0xc001214160, {0x24c4430, 0xc000058028}, 0xc00367b490, {0xc003db5540, 0xa})\n\t/go/src/github.com/argoproj/argo-workflows/workflow/controller/dag.go:518 +0x1408\ngithub.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).executeDAG(0xc001214160, {0x24c4430, 0xc000058028}, {0xc002fd6a60, 0x10}, 0xc002742e80, {0xc0019e8a68, 0x16}, 0xc00262e900, {0x24c71e0, ...}, ...)\n\t/go/src/github.com/argoproj/argo-workflows/workflow/controller/dag.go:244 +0xf05\ngithub.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).executeTemplate(0xc001214160, {0x24c4430, 0xc000058028}, {0xc002fd6a60, 0x10}, {0x24c71e0, 0xc0002255c0?}, 0xc00115e900?, {{0x0, 0x0, ...}, ...}, ...)\n\t/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:1944 +0x240c\ngithub.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).operate(0xc001214160, {0x24c4430, 0xc000058028})\n\t/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:344 +0x16ed\ngithub.com/argoproj/argo-workflows/v3/workflow/controller.(*WorkflowController).processNextItem(0xc00031e000, {0x24c4430, 0xc000058028})\n\t/go/src/github.com/argoproj/argo-workflows/workflow/controller/controller.go:763 +0x759\ngithub.com/argoproj/argo-workflows/v3/workflow/controller.(*WorkflowController).runWorker(0xc000bc0ea0?)\n\t/go/src/github.com/argoproj/argo-workflows/workflow/controller/controller.go:685 +0x9e\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x0?)\n\t/go/pkg/mod/k8s.io/apimachinery@v0.24.3/pkg/util/wait/wait.go:155 +0x3e\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0?, {0x24a2780, 0xc000fe74a0}, 0x1, 0xc000057980)\n\t/go/pkg/mod/k8s.io/apimachinery@v0.24.3/pkg/util/wait/wait.go:156 +0xb6\nk8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0x0?, 0x0?)\n\t/go/pkg/mod/k8s.io/apimachinery@v0.24.3/pkg/util/wait/wait.go:133 +0x89\nk8s.io/apimachinery/pkg/util/wait.Until(0x0?, 0x0?, 0x0?)\n\t/go/pkg/mod/k8s.io/apimachinery@v0.24.3/pkg/util/wait/wait.go:90 +0x25\ncreated by github.com/argoproj/argo-workflows/v3/workflow/controller.(*WorkflowController).Run\n\t/go/src/github.com/argoproj/argo-workflows/workflow/controller/controller.go:319 +0x19c5\n" workflow=dag-inline-gl6j8
time="2022-12-15T17:32:15.911Z" level=info msg="Updated phase Running -> Error" namespace=argo-workflow workflow=dag-inline-gl6j8
time="2022-12-15T17:32:15.911Z" level=info msg="Updated message  -> workflow 'dag-inline-gl6j8' node 'dag-inline-gl6j8.inline-dag.say-hello' uninitialized when marking as Error: [task 'dag-inline-gl6j8.inline-dag.say-hello' errored: failed to get a template]" namespace=argo-workflow workflow=dag-inline-gl6j8
time="2022-12-15T17:32:15.911Z" level=info msg="Marking workflow completed" namespace=argo-workflow workflow=dag-inline-gl6j8
time="2022-12-15T17:32:15.911Z" level=info msg="Marking workflow as pending archiving" namespace=argo-workflow workflow=dag-inline-gl6j8
time="2022-12-15T17:32:15.911Z" level=info msg="Checking daemoned children of " namespace=argo-workflow workflow=dag-inline-gl6j8
time="2022-12-15T17:32:15.911Z" level=debug msg="Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"argo-workflow\", Name:\"dag-inline-gl6j8\", UID:\"537e587a-3ef6-4474-aabc-5eb50db24f38\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"34230680608\", FieldPath:\"\"}): type: 'Warning' reason: 'WorkflowFailed' workflow 'dag-inline-gl6j8' node 'dag-inline-gl6j8.inline-dag.say-hello' uninitialized when marking as Error: [task 'dag-inline-gl6j8.inline-dag.say-hello' errored: failed to get a template]"
time="2022-12-15T17:32:15.911Z" level=debug msg="Log changes patch: {\"metadata\":{\"annotations\":{\"workflows.argoproj.io/pod-name-format\":\"v2\"},\"labels\":{\"workflows.argoproj.io/completed\":\"true\",\"workflows.argoproj.io/phase\":\"Error\",\"workflows.argoproj.io/workflow-archiving-status\":\"Pending\"}},\"status\":{\"artifactGCStatus\":{\"notSpecified\":true},\"artifactRepositoryRef\":{\"artifactRepository\":{\"s3\":{\"accessKeySecret\":{\"key\":\"accesskey\",\"name\":\"argo-workflow-s3-secret\"},\"bucket\":\"argo-workflow-1302248489\",\"endpoint\":\"cos.ap-beijing.myqcloud.com\",\"region\":\"ap-beijing\",\"secretKeySecret\":{\"key\":\"secretkey\",\"name\":\"argo-workflow-s3-secret\"}}},\"configMap\":\"artifact-repositories\",\"key\":\"default-s3-artifact-repository\",\"namespace\":\"argo-workflow\"},\"conditions\":[{\"status\":\"True\",\"type\":\"Completed\"}],\"finishedAt\":\"2022-12-15T09:32:15Z\",\"message\":\"workflow 'dag-inline-gl6j8' node 'dag-inline-gl6j8.inline-dag.say-hello' uninitialized when marking as Error: [task 'dag-inline-gl6j8.inline-dag.say-hello' errored: failed to get a template]\",\"nodes\":{\"dag-inline-gl6j8\":{\"children\":[\"dag-inline-gl6j8-2586533071\"],\"displayName\":\"dag-inline-gl6j8\",\"finishedAt\":null,\"id\":\"dag-inline-gl6j8\",\"message\":\"Workflow operation error\",\"name\":\"dag-inline-gl6j8\",\"phase\":\"Error\",\"startedAt\":\"2022-12-15T09:32:15Z\",\"templateName\":\"main\",\"templateScope\":\"local/dag-inline-gl6j8\",\"type\":\"DAG\"},\"dag-inline-gl6j8-2586533071\":{\"boundaryID\":\"dag-inline-gl6j8\",\"children\":[\"dag-inline-gl6j8-3227980711\"],\"displayName\":\"inline-dag\",\"finishedAt\":null,\"id\":\"dag-inline-gl6j8-2586533071\",\"name\":\"dag-inline-gl6j8.inline-dag\",\"phase\":\"Running\",\"startedAt\":\"2022-12-15T09:32:15Z\",\"templateScope\":\"local/dag-inline-gl6j8\",\"type\":\"DAG\"}},\"phase\":\"Error\",\"startedAt\":\"2022-12-15T09:32:15Z\"}}"
time="2022-12-15T17:32:15.916Z" level=info msg="cleaning up pod" action=deletePod key=argo-workflow/dag-inline-gl6j8-1340600742-agent/deletePod
time="2022-12-15T17:32:15.920Z" level=info msg="Workflow update successful" namespace=argo-workflow phase=Error resourceVersion=34230680622 workflow=dag-inline-gl6j8
time="2022-12-15T17:32:15.920Z" level=debug msg="Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"argo-workflow\", Name:\"dag-inline-gl6j8\", UID:\"537e587a-3ef6-4474-aabc-5eb50db24f38\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"34230680622\", FieldPath:\"\"}): type: 'Normal' reason: 'WorkflowNodeRunning' Running node dag-inline-gl6j8: Workflow operation error"
time="2022-12-15T17:32:15.920Z" level=debug msg="Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"argo-workflow\", Name:\"dag-inline-gl6j8\", UID:\"537e587a-3ef6-4474-aabc-5eb50db24f38\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"34230680622\", FieldPath:\"\"}): type: 'Warning' reason: 'WorkflowNodeError' Error node dag-inline-gl6j8: Workflow operation error"
time="2022-12-15T17:32:15.920Z" level=debug msg="Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"argo-workflow\", Name:\"dag-inline-gl6j8\", UID:\"537e587a-3ef6-4474-aabc-5eb50db24f38\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"34230680622\", FieldPath:\"\"}): type: 'Normal' reason: 'WorkflowNodeRunning' Running node dag-inline-gl6j8.inline-dag"
time="2022-12-15T17:32:16.087Z" level=info msg="archiving workflow" namespace=argo-workflow uid=537e587a-3ef6-4474-aabc-5eb50db24f38 workflow=dag-inline-gl6j8

Logs from in your workflow's wait container

found nothing, i guess the pod was not created.
stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If this is a mentoring request, please provide an update here. Thank you for your contributions.