Open jamiebuilds opened 6 years ago
I already have a tool for most of the logic here too: https://github.com/jamiebuilds/chunkd
Note: We're gonna need to have some syntax for passing the chunks into tools, we can't use env variables because they evaluate before we see them
What do you mean by "before we can see them"?
Also, am.i right then in assuming this would require custom logic in each tool (or config of tool) to support it? Or do you have something else in mind?
bolt workspaces exec --parallel --chunks -- tool $CHUNK
$CHUNK
is evaluated to ""
outside of the bolt
process, so we just see ""
Also, am.i right then in assuming this would require custom logic in each tool (or config of tool) to support it? Or do you have something else in mind?
Any tool that supports passing globs would support this. ESLint for example
$CHUNK is evaluated to "" outside of the bolt process, so we just see ""
Ah yea, I was thinking about tools with a config.js which would mean bolt could set env vars when it spawns
GNU parallel uses {}
and others: https://www.gnu.org/software/parallel/parallel_tutorial.html#Replacement-strings
There are generally a couple different ways to run tools in a Bolt repo:
When you have a tool that could either run globally or by workspace, you have to make a tradeoff decision:
Right now we don't offer anything in between these two extremes. The different "by workspace" modes don't help at all because they all spawn a new process for every workspace.
But what if we could break workspaces into "chunks" and spawn a smaller number of processes?
This could allow us to increase parallelism of a tool while also reducing the number of processes that it starts up.
CLI
I would focus on just when the
--parallel
flag is on initially.If you wanted to limit the number of items to place in a chunk at a time: