Concurrent building of sibling nodes?

[Elberet]

I would like to use Broccoli as a build pipeline for a new project, where some tasks are mostly CPU-bound (e.g. processing Docbook-XSL stylesheets with sufficiently large XML inputs).

Given this tree:

I    preprocessing
|\
| A  builder
B |  builder
|/
O    merge results into output

and assuming that both "builder" plugins do return a promise from their build method, will Broccoli build the nodes A and B concurrently?

I'm probably just blind, but I couldn't find any explicit yes or no in the docs... 😅

[rwjblue]

No, A and B will not be ran concurrently. In general, concurrency is added within a specific plugin (by using something like workerpool), but the builder itself does not support building two nodes concurrently.

[Elberet]

Hm, that's unfortunate and means that I'm losing many of the benefits I was hoping to get from Broccoli, since I'd have to code most of the processing logic into the one plugin.

Just to avoid any misunderstanding: concurrency and asynchronicity are considered equivalent here, i.e. the build will be comparable to await I; await A; await B; await O, but not await I; await Promise.all([A, B]); await O ?

[stefanpenner]

As @rwjblue mentioned today, the graph of broccoli nodes are topologically sorted and executed serially. The currently recommended approach to achieve CPU parallelism is to do so within each node via a module such as workerpool. (as @rwjblue described above).

The concurrency you describe is a viable option but we have not yet pursued this capability due to pragmatic reasons.

Conceptually the following could be implemented by having a single workerpool which would be provided by broccoli to it's nodes, to ensure node's share available concurrency gracefully. Once this exists, we could either loosen the serial execution constraint across the board or allow plugins to opt into being scheduled concurrently if broccoli deems safe.

[Elberet]

I believe concurrent node execution would be a nice feature for another major version of Broccoli, but in the meantime, as it happens, my priorities have shifted so it is no longer a requirement for me personally.

Feel free to close this issue, or keep it around as an informal feature request... 😃