Parallel and ordered plugin execution

[ConnorGraham]

All plugins are run sequentially. Unless there is a dependency on one tool to complete before the next runs, we can instead run tools in parallel.

Because tools depending on other tools (e.g. helm needs to wait for terraform to finish) is unique to the user, this needs to be configurable from operations repositories.

Once a weight has been assigned to each plugin, ordering can be managed using Python's built in multiprocessing lib. If the use case expands to become more complex (as parallelism often can), we can explore using a tool like Spotify's Luigi

[mickmcgrath13]

Some additional background from another repo (old private repo where we had bitops issues previously):

@shyamrayaprolu said: Couple of Use Cases #

1. Infrastructure Deployments # In general infrastructure deployments have to follow order like provision the resources first and then deploy code/app on those resource.

2. Application/API Based Deployments # In the case of application/microservice/api deployment we generally deploy multiple components at the same time to save time on overall deployment.

In these scenarios users should have an option to decide how BitOps should deploy code. While performing deployment in order, users can add config to let BitOps execute IAC frameworks as per the config. And if no order is needed users should be able to set config so that Bitops can execute scripts in parallel.

[mickmcgrath13]

Additionally, I'd not call it weight. weight implies to be that it's going to get multiplied to something rather than which order or tier the plugins will run in. How about something verbose like execution_order or something?

[mickmcgrath13]

maybe also consider an alias property (similar to helm) for dependencies/plugins.

[mickmcgrath13]

Reviving this issue/discussion after some BitOps V2 discussions.

BitOps V2 will provide, among other things, a deployments key in the BitOps-level bitops config which will configure the order of deployments:

deployments:
  terraform:
    plugin: terraform
  ansible:
    plugin: ansible

Note In this structure, "plugin" will point to a corresponding config item in the plugins object which is a sibling to deployments in the same BitOps-level bitops config

When we move to be able to specify parallel vs sequential deployment configuration, we'll have to decide:

1. break the existing format, and document the upgrade procedure/requirements
2. maintain the existing format to maintain backward compatibility with the deployments config, and document how it works.

I personally really like the latter, and I would approach the deployments configure like this:

object: serial arrays: parallel nest as you like

For an example, I'll use the following syntax for brevity: Assume deployments: {a,b,c} is an object that looks like this:

deployments: 
  a:
    plugin: terraform
  b:
    plugin: ansible
  c:
    plugin: helm

Backwards Compatibility

so, for this example, "a", "b", and "c" would be deployed in sequence (a, then b, then c)

deployments: {a,b,c}

This is how BitOps will work at launch of V2

Parallel a, b, and c

In this example, "a", "b", and "c" would be deployed all at the same time (i.e. in parallel

deployments: [{a},{b},{c}]

the corresponding full deployment config would look something like this:

deployments: 
  - a:
      plugin: terraform
  - b:
      plugin: ansible
  - c:
      plugin: helm

Parallel and sequential

This example shows running two sets of 3 sequential plugins, each set would run at the same time:

deployments: [{a,b,c},{d,e,f}]

• "a" would go before "b", and "b" would go before "c"
• "d" would go before "e", and "e" would go before "f"
• "a" and "d" would start at the same time

This approach would allow backward compatibilty, but what it would not allow is having a way to "merge" or have one execution in a parallel set "wait for" any execution that is part of another parallel set. For example, in

deployments: [{a,b,c},{d,e,f}, {g}]

there's no way to have "g" wait for "b" (or at least it'd be challenging without some sort of DAG logic). If that's a desirable behavior, we'd likely want to see about leveraging an external execution engine/framework (like luigi as @ConnorGraham mentioned above)

Can we have both (i.e. backwards compatibility AND an external task execution engine/framework)?

[mickmcgrath13]

take a look at this: https://github.com/Illumina/pyflow/tree/master/pyflow

[mickmcgrath13]

maybe also take a look at this: https://dagger.io/