Open MylesBorins opened 4 years ago
Currently there is no way to accomplish this beyond using && or & which is can be platform specific
@MylesBorins - Just to confirm, you are specifically talking about a built-in mechanism for this right? Is the proposal roughly that we should add a run-series
command that works similarly to how npm-run-all (a somewhat popular package that provides some functionality in this space) works?
for "run parallel", i haven't yet found a tool that properly handles nonzero exit codes, and also handles mixing (and prefixing) streaming stdout/stderr output. If npm can solve this that would be awesome.
I think this is a real issue, and our current constraints have led to some great and some really awful things. If we make it more powerful we run the risk of encouraging some even more wild usage.
At work, we have a system for composing "commands" which is not make. Learning from this, it is hard to get right but is ultimately very useful. If we want to really solve this well, which I am not sure we need to do, then I think we should consider preexisting systems with features like work avoidance and DSLs for declaring the commands. Also, to share, we have a tool which uses the stmux
package and is a pretty good DX for the parallel commands to solve the log-interleaving problem.
FWIW I think that we could likely scope this solution to only run in series
that is more or less sugar for npm run one && npm run two && npm run three && npm run four
and significantly reduce duplication and noise in the package.json
Removing Agenda label. A next step would be a making a more complete proposal specifically for series.
As I said in the meeting I think there is also a use case I can see for parallel, although probably in a second RFC: In web development there are often watch tasks that run Typescript, Sass, webpack or whatever. It would be useful to have a way to run them in parallel through npm and if one dies, just kill the entire thing.
I can definitely find this useful, not even parallel, but running multiple scripts in series as an array:
"scripts": {
"build": [ "webpack --verbose", "npm run test", "git add .", "echo 'all done'" ]
}
npm run build
I think run-parallel
should be an addition to yet not implemented run-series
Will run-series
be an easier implementation than run-parallel
as npm
can run one script and in theory it could run multiple?
The array approach is a pretty straightforward way to implement "series", fwiw - I like it.
Not that I like this line of thought, but to toss out another complicated option:
Array
: run in seriesObject
: run in parallel where the names are headings for the grouped logsYou could (although I strongly believe we need to move this all outside of npm
) even allow for nesting:
"scriptfoo": [{
"build1": "thing",
"build2": "otherthing"
}, "test"]
Again, I really dont like the complexity this would add to npm, but if someone were to work on a declarative and simple command runner, one might do it this way.
Uhh, I like this solution a lot @wesleytodd because it does not introduce commands/flags and is straight forward! Through a flag like --no-fail
we could say that parallel or series tasks wouldn't exit on error.
Our scripts are looking like that with npm-run-all
:
This is how it could look like then (sub-tasks would be indented with there title):
*
would just serve as a glob like npm-run-all
handles it :)
Perhaps the array should always be the names of other scripts in the script map? This way the individual tasks remain named scripts that can be run separately. In addition it removes the need to inline npm run
:
{
"scripts": {
"build-all": ["npm run build1", "npm run build2"],
"build1": "...",
"build2": "..."
}
}
and instead just write:
{
"scripts": {
"build-all": ["build1", "build2"],
"build1": "...",
"build2": "..."
}
}
It might be nice to reserve the sweeter array syntax for parallel runs, or even make the series v parallel thing a flag of npm run itself - npm run build-all -j8
:)
This feature could be adapted to be used on workspaces
too.
npm run-parallel lint --workspaces
executes in parallel the lint
script for each project in the monorepo.
I would like to make an argument to allow the dependency graph to be 'statically' extracted. Either by ensure the graph has a static declaration, or if that is not desirable, have an 'export' mode i.e.
npm run build --export-dependency-graph
npm run-export-graph build
or something similar where the graph can be emitted cheaply without actually executing the tasks.
My main motivation for this request is so that an external build scheduler can take this information and optimize the javascript build even futher.At Microsoft we have very large JavaScript repositories as well as mixed repositories where nmake (C++), gradle (java), msbuild (c++ and C#) and javascript projects all live together in a single build graph. Today it is already a struggle to use a single verb per project as the critical build path is too long. We leverage tools like Lage to create finer grained jobs at the 'verb' level. Think scripts called build, bundle, lint, test, test:e2e where lage declares finer dependencies i.e. 'build' depends on the build verb of dependent projects. test depends on the 'build' verb, test:e2e depends on the bundle verb etc etc. We already leverage a library to perform what this RFC promotes just-scripts that performs parallel or sequential subtasks in a single build verb as well as having support for flags to control the subgraph. i.e. wether it should create bundles for all platforms (win32, ios, android, web) or just a single etc.
Lage and Just mentioned in the above paragraph support running locally on a single devbox. We also have robust build engines like BuildXL that can take the graph emitted by Lage (as well as the graphs from other projects in hybrid case like nmake, msbuild, etc) and merge them into a giant graph of work to be executed. The BuildXL build engine supports reliable caching, distributing the build accross multiple machines, handle safety policies (i.e. prevent one JS project from using ../.. to read files from other js projects). It has lots of smarts to make sure to use historical execution data to make sure the build machines are loaded as effeciently as possible. It has support for features like a reliable shared developer cache etc etc.
The finer grained the nodes that the build engine can operate on the better it can optimize itself.. Lage gives a graph with multiple verbs on a given project... But today it cannot peek into the commands of each verb. i.e. of 'build' is serial(parralel(codegen1, codegen2), typescriptcompiler)
that is executed as a unit by the executed npm run build
process in the build graph...
Having a way to cheaply statically get the information of what would happen inside the build
verb would allow a finer grained build graph for the build engine and better machine utilization.
It would be amazing if this functionality could be built-in into npm! I really appreciate that this is being considered!
I think it would be great to get scripts out of the business of directly running sub-steps in their command invocation and instead have them explicitly declare their dependencies, then have the script runner run sub-steps in parallel or serial as required.
So instead of:
"scripts": {
"build": "npm run build:css && npm run build:js && npm run build:html && build:img"
"build:css": ...
we might have scripts declared as objects with explicit dependencies:
"scripts": {
"build": {
"dependencies": [
"build:css",
"build:js",
"build:html",
"build:img",
]
}
"build:css": "...",
"build:html": {
"command": "some-html-build-tool",
// tasks can depend on each other. The runner should figure out the correct order & parallelism
"dependencies": [
"build:css",
]
}
We built and published (today!) https://github.com/google/wireit, which is an attempt at bringing a declarative script-level dependency graph to npm, including parallel execution.
It uses a syntax just like what @justinfagnani showed in the comment above — except in a separate wireit
section of the package.json
. For example:
{
"scripts": {
"build": "wireit",
"bundle": "wireit"
},
"wireit": {
"build": {
"command": "tsc",
"files": ["src/**/*.ts", "tsconfig.json"],
"output": ["lib"]
},
"bundle": {
"command": "rollup -c",
"dependencies": ["build"],
"files": ["rollup.config.json"],
"output": ["dist/bundle.js"]
}
}
}
(So this works by having the actual npm script delegate to the wireit
binary, and the wireit
binary figures out which script was invoked by reading the npm_lifecycle_event
and npm_package_json
environment variables.)
I concur that it would be awesome to have functionality like this built into npm. I'd be interested in starting/contributing to an RFC along these lines.
This would be amazing. npm-run-all
has not been maintained for several years and yet it is the only way to run a series of npm scripts in a controlled, cross-platform way
it would be a good start if myTask
defined in all workspaces, could be run in parallel npm run myTask -ws
with simply adding a --parallel
option.
otherwise an hack and bash/PS7 hack must be done:
WORKSPACES=`jq -r '.workspaces | @sh' package.json | tr -d \\'` && for w in ${WORKSPACES[@]}; do echo \"$w\" && pushd . && cd $w && npm run myTask; popd; done;
not really convinient...
Is this going to happen? It seems like no one is implementing this.
This is massively useful for a number of incredibly common scenarios:
My wish list for npm scripts:
npm-run-all
and concurrently
havenpm run
to support both serial and parallel execution of scripts and workspacesFor example:
{
"scripts": {
// cross-platform operators for commands
"script1": "cmd -a & cmd -b",
"script2": "cmd -c && cmd -d",
"script3": "cmd -e || cmd -f",
"script4": "{ cmd -a & cmd -b } || { cmd -c && cmd -d }",
"script5": "cmd -a | cmd -b",
"script6": "cmd -c 2>&1 outputfile",
"script7": "cmd -d < inputfile",
// cross-platform operators for scripts
"script8": "npm run script1 & npm run script2",
"script9": "{ npm run script1 & npm run script2 } || { npm run script3 && npm run script4 } | format -abc > logfile",
// syntactic sugar and new options
"script10": "npm run script:*",
"script11": "npm run script1 script2 script3 --parallel",
"script12": "npm run lint --workspaces --parallel",
"script13": "npm run build --workspace=a --workspace=b --serial"
}
}
Bonus points for being able to set environment variables in scripts cross-platform and support for json comments as shown above (if that's even possible).
I think as soon as you add running scripts sequentially and concurrently, you quickly end up wanting DAG of scripts where you run common script dependencies only once. You relatively quickly then end up wanting to cache common script dependency results so that subsequent runs are faster.
This is what Wireit does, while maintaining npm as the user interface and package.json as the config file. I think it's a great approach, but short of building those features into npm itself, I think npm could also enable pluggable script runners that can add those features. I propose that in https://github.com/npm/rfcs/issues/691
This feature could be adapted to be used on
workspaces
too.npm run-parallel lint --workspaces
executes in parallel the
lint
script for each project in the monorepo.
@llimllib and I found a fairly clean way to do this using npm query
, JSON parsing the result and xargs:
npm query .workspace | \
node -p 'JSON.parse(fs.readFileSync(0)).map(ws => ws.name).join("\n")' | \
xargs -I {} -P 0 npm run lint --if-present --workspace "{}"
Got the JSON parsing approach from this gist: https://gist.github.com/kristopherjohnson/5065599?permalink_comment_id=4775925#gistcomment-4775925
BTW I just adopted nx / lerna and it works great and it also caches. Example project here: https://github.com/bhouston/template-typescript-monorepo
Motivation ("The Why")
Often an npm script is part of a more complicated pipeline, tools such as grunt + gulp have been created to help manage some of this pipeline. A repo may have multiple steps to a compilation such as clean, build various types of assets (html, css, js), etc. Currently npm itself does not have a mechanism to orchestrate more complex workflow.
Example
Most recently I utilized xargs to accomplish a similar task.
"build": "echo build:clean build:css build:js build:html build:img | xargs -n1 npm run"
if
run-series
were available this could be accomplished in a platform agnostic way via"build": "npm run-series build:clean build:css build:js build:html build:img"
It could also be optimized to run parallel operations
"build": "npm run build:clean && npm run-parallel build:css build:js build:html build:img"
or potentially
"build": "npm run build:clean --then npm run-parallel build:css build:js build:html build:img"
How
Current Behaviour
Currently there is no way to accomplish this beyond using
&&
or&
which is can be platform specificDesired Behaviour
First class support for running multiple scripts in parallel or series
References