search

StackStorm / st2

StackStorm (aka "IFTTT for Ops") is event-driven automation for auto-remediation, incident responses, troubleshooting, deployments, and more for DevOps and SREs. Includes rules engine, workflow, 160 integration packs with 6000+ actions (see https://exchange.stackstorm.org) and ChatOps. Installer at https://docs.stackstorm.com/install/index.html
https://stackstorm.com/
Apache License 2.0
6.08k stars 747 forks source link

Orquesta workflow with a sub workflow with "with items" task gets stuck in the running state when empty array is passed for that task #4954

Closed igcherkaev closed 4 years ago

igcherkaev commented 4 years ago

SUMMARY

Orquesta workflow with a sub workflow with "with items" task gets stuck in the running state when empty array is passed for that task

STACKSTORM VERSION

Paste the output of st2 --version:

$ st2 --version
st2 3.2.0, on Python 2.7.5
OS, environment, install method

CentOS 7 install following the docs at: https://docs.stackstorm.com/install/rhel7.html

Steps to reproduce the problem

I have the following two workflows:

wf_orquesta_stuck1.meta.yaml:

---
pack: "playground"
name: "wf_orquesta_stuck1"
description: "Orquesta workflow gets stuck in running bug, st2 v3.2.0"
runner_type: orquesta
enabled: true
entry_point: "workflows/wf_orquesta_stuck1.yaml"
parameters:
  some_array:
    required: false
    default: []
    type: "array"
    description: "Some array."

workflows/wf_orquesta_stuck1.yaml:

---
version: '1.0'
input:
  - some_array
tasks:
  init_task:
    action: core.noop
    next:
      - when: <% succeeded() %>
        do:
          - subflow
  subflow:
    action: playground.wf_orquesta_stuck_sub
    input:
      some_array: <% ctx().some_array %>
    next:
      - when: <% succeeded() %>
        do:
          - finish

  finish:
    action: core.noop

And the sub workflow:

__wf_orquesta_stuck_sub.meta.yaml__:

---
pack: "playground"
name: "wf_orquesta_stuck_sub"
description: "Orquesta sub workflow that gets main workflow stuck in running, st2 v3.2.0"
runner_type: orquesta
enabled: true
entry_point: "workflows/wf_orquesta_stuck_sub.yaml"
parameters:
  some_array:
    required: false
    default: []
    type: "array"
    description: "Some array."

__workflows/wf_orquesta_stuck_sub.yaml__:

---
version: '1.0'
input:
  - some_array

tasks:
  process_some_array:
    with: <% ctx(some_array) %>
    action: core.local
    input:
      cmd: "echo <% item() %>"

Expected Results

The parent workflow to finish when empty array is passed.

Actual Results

The parent workflow gets stuck in the RUNNING state and never finishes.

image

workflow engine logs:

2020-05-19 12:52:23,605 140150526377328 INFO workflows [-] [5ec438f76dcfffe45e24342b] Processing request for workflow execution.
2020-05-19 12:52:23,616 140150526377328 INFO workflows [-] [5ec438f76dcfffe45e24342b] Requesting conductor to start running workflow execution.
2020-05-19 12:52:23,617 140150526377328 INFO workflows [-] [5ec438f76dcfffe45e24342b] Identifying next set (0) of tasks for workflow execution in status "running".
2020-05-19 12:52:23,619 140150526377328 INFO workflows [-] [5ec438f76dcfffe45e24342b] Identified the following set of tasks to execute next: init_task (route 0)
2020-05-19 12:52:23,619 140150526377328 INFO workflows [-] [5ec438f76dcfffe45e24342b] Mark task "init_task", route "0", in conductor as running.
2020-05-19 12:52:23,619 140150526377328 INFO workflows [-] [5ec438f76dcfffe45e24342b] Updating workflow execution from status "requested" to "running".
2020-05-19 12:52:23,654 140150526377328 INFO workflows [-] [5ec438f76dcfffe45e24342b] Requesting execution for task "init_task", route "0".
2020-05-19 12:52:23,655 140150526377328 INFO workflows [-] [5ec438f76dcfffe45e24342b] Processing task execution request for task "init_task", route "0".
2020-05-19 12:52:23,661 140150526377328 INFO workflows [-] [5ec438f76dcfffe45e24342b] Task execution "5ec438f729178af3686e28df" created for task "init_task", route "0".
2020-05-19 12:52:23,773 140150526377328 AUDIT action [-] Action execution requested. LiveAction.id=5ec438f729178af3686e28e0, ActionExecution.id=5ec438f729178af3686e28e1
2020-05-19 12:52:23,773 140150526377328 INFO workflows [-] [5ec438f76dcfffe45e24342b] Action execution "5ec438f729178af3686e28e1" requested for task "init_task", route "0".
2020-05-19 12:52:23,789 140150526377328 INFO workflows [-] [5ec438f76dcfffe45e24342b] Identifying next set (1) of tasks for workflow execution in status "running".
2020-05-19 12:52:23,790 140150526377328 INFO workflows [-] [5ec438f76dcfffe45e24342b] No tasks identified to execute next.
2020-05-19 12:52:23,852 140150526377328 INFO workflows [-] [5ec438f76dcfffe45e24342b] Action execution "5ec438f729178af3686e28e1" for task "init_task" is updated and in "scheduled" state.
2020-05-19 12:52:23,909 140150526377328 INFO workflows [-] [5ec438f76dcfffe45e24342b] Action execution "5ec438f729178af3686e28e1" for task "init_task" is updated and in "running" state.
2020-05-19 12:52:23,967 140150526377328 INFO workflows [-] [5ec438f76dcfffe45e24342b] Action execution "5ec438f729178af3686e28e1" for task "init_task" is updated and in "succeeded" state.
2020-05-19 12:52:23,980 140150526377328 INFO workflows [-] [5ec438f76dcfffe45e24342b] Handling completion of action execution "5ec438f729178af3686e28e1" in status "succeeded" for task "init_task", route "0".
2020-05-19 12:52:24,013 140150526377328 INFO workflows [-] [5ec438f76dcfffe45e24342b] Publish task "init_task", route "0", with status "succeeded" to conductor.
2020-05-19 12:52:24,042 140150526377328 INFO workflows [-] [5ec438f76dcfffe45e24342b] Identifying next set (0) of tasks after completion of task "init_task", route "0".
2020-05-19 12:52:24,045 140150526377328 INFO workflows [-] [5ec438f76dcfffe45e24342b] Identified the following set of tasks to execute next: subflow (route 0)
2020-05-19 12:52:24,046 140150526377328 INFO workflows [-] [5ec438f76dcfffe45e24342b] Mark task "subflow", route "0", in conductor as running.
2020-05-19 12:52:24,085 140150526377328 INFO workflows [-] [5ec438f76dcfffe45e24342b] Requesting execution for task "subflow", route "0".
2020-05-19 12:52:24,086 140150526377328 INFO workflows [-] [5ec438f76dcfffe45e24342b] Processing task execution request for task "subflow", route "0".
2020-05-19 12:52:24,095 140150526377328 INFO workflows [-] [5ec438f76dcfffe45e24342b] Task execution "5ec438f829178af3686e28e2" created for task "subflow", route "0".
2020-05-19 12:52:24,213 140150526377328 AUDIT action [-] Action execution requested. LiveAction.id=5ec438f829178af3686e28e3, ActionExecution.id=5ec438f829178af3686e28e4
2020-05-19 12:52:24,213 140150526377328 INFO workflows [-] [5ec438f76dcfffe45e24342b] Action execution "5ec438f829178af3686e28e4" requested for task "subflow", route "0".
2020-05-19 12:52:24,230 140150526377328 INFO workflows [-] [5ec438f76dcfffe45e24342b] Identifying next set (1) of tasks for workflow execution in status "running".
2020-05-19 12:52:24,230 140150526377328 INFO workflows [-] [5ec438f76dcfffe45e24342b] No tasks identified to execute next.
2020-05-19 12:52:24,386 140150526377328 INFO workflows [-] [5ec438f76dcfffe45e24342b] Action execution "5ec438f829178af3686e28e4" for task "subflow" is updated and in "scheduled" state.
2020-05-19 12:52:24,451 140150526377328 INFO workflows [-] [5ec438f76dcfffe45e24342b] Action execution "5ec438f829178af3686e28e4" for task "subflow" is updated and in "running" state.
2020-05-19 12:52:24,522 140150526377328 INFO workflows [-] [5ec438f829178af3686e28e4] Processing request for workflow execution.
2020-05-19 12:52:24,530 140150526377328 INFO workflows [-] [5ec438f829178af3686e28e4] Requesting conductor to start running workflow execution.
2020-05-19 12:52:24,530 140150526377328 INFO workflows [-] [5ec438f829178af3686e28e4] Identifying next set (0) of tasks for workflow execution in status "running".
2020-05-19 12:52:24,533 140150526377328 INFO workflows [-] [5ec438f829178af3686e28e4] Identified the following set of tasks to execute next: process_some_array (route 0)
2020-05-19 12:52:24,533 140150526377328 INFO workflows [-] [5ec438f829178af3686e28e4] Mark task "process_some_array", route "0", in conductor as running.
2020-05-19 12:52:24,534 140150526377328 INFO workflows [-] [5ec438f829178af3686e28e4] Updating workflow execution from status "requested" to "running".
2020-05-19 12:52:24,573 140150526377328 INFO workflows [-] [5ec438f829178af3686e28e4] Requesting execution for task "process_some_array", route "0".
2020-05-19 12:52:24,574 140150526377328 INFO workflows [-] [5ec438f829178af3686e28e4] Processing task execution request for task "process_some_array", route "0".
2020-05-19 12:52:24,581 140150526377328 INFO workflows [-] [5ec438f829178af3686e28e4] Task execution "5ec438f829178af3686e28e5" created for task "process_some_array", route "0".
2020-05-19 12:52:24,582 140150526377328 INFO workflows [-] [5ec438f829178af3686e28e4] Task "process_some_array", route "0", has no items and succeed by default.
2020-05-19 12:52:24,626 140150526377328 INFO workflows [-] [5ec438f829178af3686e28e4] Publish task "process_some_array", route "0", with status "succeeded" to conductor.
2020-05-19 12:52:24,627 140150526377328 INFO workflows [-] [5ec438f829178af3686e28e4] Updating workflow execution from status "running" to "succeeded".
2020-05-19 12:52:24,663 140150526377328 INFO workflows [-] [5ec438f829178af3686e28e4] Workflow action execution is completed and invoking post run.
2020-05-19 12:52:24,664 140150526377328 INFO utils [-] Invoking post run for action execution 5ec438f829178af3686e28e3.
2020-05-19 12:52:24,666 140150526377328 INFO utils [-] Action execution 5ec438f829178af3686e28e3 runs wf_orquesta_stuck_sub of runner type orquesta.
2020-05-19 12:52:24,684 140150526377328 INFO workflows [-] [5ec438f829178af3686e28e4] Identifying next set (1) of tasks for workflow execution in status "succeeded".
2020-05-19 12:52:24,684 140150526377328 INFO workflows [-] [5ec438f829178af3686e28e4] No tasks identified to execute next.
guzzijones commented 4 years ago

I can confirm I see this on some of my workflows too. As a workaround I check the .length() of the array in a prior task with core.noop action. If the length is 0 then I skip the with_items task.

emptywee commented 4 years ago

Yeah, as a temp workaround it's fine to add empty array tests, but it'd be nice for the engine to handle it properly (mistral did :P)