StackStorm / st2

StackStorm (aka "IFTTT for Ops") is event-driven automation for auto-remediation, incident responses, troubleshooting, deployments, and more for DevOps and SREs. Includes rules engine, workflow, 160 integration packs with 6000+ actions (see https://exchange.stackstorm.org) and ChatOps. Installer at https://docs.stackstorm.com/install/index.html
https://stackstorm.com/
Apache License 2.0
6.08k stars 747 forks source link

Concurrency policy not running if set to one #6044

Open FileMagic opened 1 year ago

FileMagic commented 1 year ago

SUMMARY

If you were to set the concurrency policy of an action to be 1 then it does not run any actions. I believe the following piece of code is responsible:

https://github.com/StackStorm/st2/blob/d374b92f0dbf6b26c5dfdcffd594ecbd0800d635/st2actions/st2actions/policies/concurrency.py#L53C12-L53C34

        count = scheduled + running

        # Mark the execution as scheduled if threshold is not reached or delayed otherwise.
        if count < self.threshold:
            LOG.debug(
                "There are %s instances of %s in scheduled or running status. "
                "Threshold of %s is not reached. Action execution will be scheduled.",
                count,
                target.action,
                self._policy_ref,
            )
            status = action_constants.LIVEACTION_STATUS_REQUESTED
        else:
            action = "delayed" if self.policy_action == "delay" else "canceled"
            LOG.debug(
                "There are %s instances of %s in scheduled or running status. "
                "Threshold of %s is reached. Action execution will be %s.",
                count,
                target.action,
                self._policy_ref,
                action,
            )
            status = self._get_status_for_policy_action(action=self.policy_action)

I think this should be changed to:

        count = scheduled + running

        # Mark the execution as scheduled if threshold is not reached or delayed otherwise.
        if count <= self.threshold:
            LOG.debug(
                "There are %s instances of %s in scheduled or running status. "
                "Threshold of %s is not reached. Action execution will be scheduled.",
                count,
                target.action,
                self._policy_ref,
            )
            status = action_constants.LIVEACTION_STATUS_REQUESTED
        else:
            action = "delayed" if self.policy_action == "delay" else "canceled"
            LOG.debug(
                "There are %s instances of %s in scheduled or running status. "
                "Threshold of %s is reached. Action execution will be %s.",
                count,
                target.action,
                self._policy_ref,
                action,
            )
            status = self._get_status_for_policy_action(action=self.policy_action)

and a unit test should be added.

Steps to reproduce the problem

Set the policy to be 1 for a specific action in stackstorm, and try to run the action using something like with items.

Expected Results

One action running at a time.

Actual Results

No actions run and they stay in the policy_action state.

FileMagic commented 1 year ago

If this seems like an actual bug to anyone else, I can make the PR for it.

arm4b commented 1 year ago

Following this logic, when you set the concurrency to 2, - it really runs only 1 action at a time? Can you confirm that? Do those debug logs also indicate that?

@FileMagic if you're sure there's a bug, - definitely submit a PR 👍

I didn't have a chance to use concurrency for a long time. Did anyone experience this issue too @StackStorm/contributors @StackStorm/maintainers?

Ein-nor commented 1 year ago

For us, the workflows work when I set concurrency to 1. The workflow also uses with_items.

nzlosh commented 1 year ago

Orquesta with-items concurrency is managed internally to Orquesta. Conceptually, Orquesta's concurrency is governed by the scheduling of each item (there is no consideration of global concurrency of the action being called). With action concurrency, it is governed by locking using the coordinator.

This suggests that Orquesta with-items and Action concurrency are cumulative. Orquesta could have a concurrency of 3 items and the action being called have an action concurrency of 1, in which case, 3 actions would be scheduled at a time with only 1 actually running in parallel.

I did a bit of testing and found that when Orquesta has concurrency set to 0 the workflow remains block. https://github.com/StackStorm/orquesta/issues/259

guzzijones commented 1 year ago

to be clear this is in reference to policy not concurrency settings.