Running build jobs in parallel

[kostyay]

Hey Is there any reason why the builds at the moment are synchronous? Do you see any architectural challenges making builds run in go routine pools? Thanks

[fho]

The only reason is that it did not seem worth to implement for our specific usecase so far. :-) I think it would be great if baur would support a commandline option to configure the number of tasks to run in parallel. Most of what is needed for it already exist. The only tricky part to consider is how we handle the output of the tasks in --verbose mode. Either baur shows output of tasks run in parallel interleaved, then it becomes hard to follow or baur buffers the output of parallel executed tasks and shows it when they terminated. If a task hangs, baur won't show it's output until it terminated which will make it harder to analyze the issues.

If baur is used to run tasks that all have a similar workload or on machines that have much more resources available then the average task requires, running them in parallel should be much more efficient.

When baur runs tasks with mixed workloads on multiple machines with low resources it seems impossible to find an efficient limit for the number of parallel executions though. Some tasks require only minimal resources others tasks utilize the whole machine. If multiple tasks are run in parallel that each could utilize the whole machine, running them in parallel will be slower in total then running them sequentially. It might even cause OOM issues.

In our setup we currently parallelize and distribute task execution on multiple machines via Jenkins:

• run baur status to figure out which tasks needs to be run,
• run baur run TASK for each tasks in parallel via jenkins, jenkins schedules them across the available executors on multiple nodes

This has the overhead of discovering and calculating the digest for the same task 2x, though.

[kostyay]

I realize running tasks in parallel could be counter productive and inefficient, however the ability I want is similiar to running make -j <number>.

            Specifies  the  number  of  jobs (commands) to run simultaneously.  If there is more than one -j option, the last one is effective.  If the -j option is given
            without an argument, make will not limit the number of jobs that can run simultaneously.```

I will see if I can create a PR for that :)

[fho]

awesome, thanks :-)

[kostyay]

See PR

[fho]

PR was merged, thanks for your help @kostyay