Support parallel execution of tasks

[drewwells]

In a situation where multiple tools are used, it is convenient to allow them to build independently of each other. For instance, I may want to run gulp and go build independently. I could specify that I want these run in parallel in the yaml config:

script:
  - gulp
    parallel: true
  - go build -o api && ./api
    parallel: true
verbose: true

[zabawaba99]

So if one of the parallel processes fails, I would imagine the entire builds fails and everything else is cancels. Is this what you would expect as well @drewwells ?

[drewwells]

I'm not too familiar with the tool, but the screencast looked like it repeated commands even on failure go test failure for instance. So if one failed, it would report the failure but the other command would continue to run. The failed command would be rerun after the next change.

[drewwells]

Possibly that's why the screencast ordered the arguments as they did, go vet failure was preventing go test from running? Parallel execution wouldn't necessarily have that dependency chain since the scripts could be totally unrelated.

[zabawaba99]

Right not if one command fails all subsequent commands do not run. In order to preserve that maybe we can have something like

script
  - parallel
    - gulp
      go build -o api && ./api
    echo "some command that runs after the paralleled commands"

That way, if the group of parallel commands exits successfully then the second command can be executed, otherwise it never gets run.

cc/ @Tonkpils

[Tonkpils]

I actually like the parallel: true approach and explicitly state that any commands with that option will not cause the build to fail.

One thing I thought we discussed is possibly having groups of parallel scripts, which may be a scope creep but would be easier to implement with the parallel: true approach.

[drewwells]

I like that idea of not failing the build for one parallel. Something that is really important to look at is how this interacts with non-parallel tasks. Those should likely run first and fail the build even if parallel ones are coming later. This gives the developer the power to make sure certain things run before any other tasks run.

script:
  -   npm install
  -   somenpmtool
      parallel: true

This should run npm install first before the parallel tasks work. So the developer can create some dependency management even with parallel tasks.

[Tonkpils]

I had a discussion with @zabawaba99 and we came up with a sensible solution for an initial implementation of this feature.

The initial issue with having parallel commands is the order in which the commands happen. Given a file:

script:
  - foo
  - bar
    parallel: true
  - baz
    parallel: true
  - omg

If omg fails, should the entire build stop? What happens with the parallel commands?

We've decided to create another section/label called runners which by default will run the commands in parallel. The runners section would run after the script, or aliased label builders, section.

The names are TBD but this gets the point across.

As an example:

builders:
  - npm install
runners:
  - gulp
  - go build -o foo && ./foo

[zabawaba99]

Couldn't have written it better myself @Tonkpils .

[drewwells]

Yeah, I think sequential ones should all be executed first. If any fail, the whole thing fails.

I like the idea of grouping parallel tasks :+1:

[zabawaba99]

Closed by #49 and #50.

Give it a shot @drewwells and let us know!

[drewwells]

can do, looks good!