cloudposse / atmos

👽 Terraform Orchestration Tool for DevOps. Keep environment configuration DRY with hierarchical imports of configurations, inheritance, and WAY more. Native support for Terraform and Helmfile.
https://atmos.tools
Apache License 2.0
637 stars 86 forks source link

Feature Request: Parallel Steps in Workflows #302

Open osterman opened 1 year ago

osterman commented 1 year ago

Describe the Feature

During cold starts, there are frequently many steps that could be run in parallel, speeding up the provisioning.

Expected Behavior

Support nested blocks of "steps" of type "parallel". When executed, all parallel steps of a step must complete before advancing to the next step.

workflows:
  test-1:
    description: "Test workflow"
    steps:
      - type: parallel
        steps:
        - command: echo Command 1
          name: step1
          type: shell
        - command: echo Command 2
          name: step2
          type: shell

If any step in the parallel execution fails, the next step in the workflow is not run.

Use Case

  1. Cold starts. Bringing up EKS clusters, there are 10-15 services that could be deployed in parallel as they have no dependency on each other.
jamengual commented 1 year ago

should this have a configurable count for how many parallel process to run?

On Thu, Jan 12, 2023 at 7:45 AM Erik Osterman (CEO @ Cloud Posse) < @.***> wrote:

Describe the Feature

During cold starts, there are frequently many steps that could be run in parallel, speeding up the provisioning. Expected Behavior

Support nested blocks of "steps" of type "parallel". When executed, all parallel steps of a step must complete before advancing to the next step.

workflows: test-1: description: "Test workflow" steps:

  • type: parallel steps:
    • command: echo Command 1 name: step1 type: shell
    • command: echo Command 2 name: step2 type: shell

If any step in the parallel execution fails, the next step in the workflow is not run. Use Case

  1. Cold starts. Bringing up EKS clusters, there are 10-15 services that could be deployed in parallel as they have no dependency on each other.

— Reply to this email directly, view it on GitHub https://github.com/cloudposse/atmos/issues/302, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAQ3ERCEAV3VR5BWRVCJTILWSARIXANCNFSM6AAAAAATZMZECY . You are receiving this because you are subscribed to this thread.Message ID: @.***>