[rush] Mode to generate files outside the repo

[nayanshah]

Is this a feature or a bug?

• [x] Feature
• [ ] Bug

We're still in investigation phase so using this thread to start the discussion about overall goal. Individual issues (e.g project specific .rush folders, redirecting node_modules) can be tracked as separate issues.

Note: fixes made as a part of this effort would be helpful in lighting up rush-buildxl integration.

@apostolisms @iclanton @D4N14L @octogonz

Context

When building with BuildXL, it's recommended to mark source code directories as readonly and avoid generating outputs under those directories (details below). In our BuildXL based build system, the entire repo is marked as readonly. There are existing MsBuild & NMake projects which produce outputs in a directory outside the repo. Rush (and in-general all Javascript) builds produce output files inside the source tree which makes it challenging.

Proposal

It is impossible to guarantee that all builds will be able to avoid writing to source tree since the projects control build tasks. But to begin, rush can be updated to support custom output directory, i.e. no orchestrator specific outputs generated inside the repo.

BuildXL's handling of readonly v/s writable directories

An intermediate build step attempting to enumerate files in a directory could get different results depending on when it gets executed. Caching such a process is tricky since it's hard to predict the filesystem at the time of execution of the process. Using the real filesystem would cause unnecessary cache misses since directory state changes over time. Other option is using virtual filesystem (using the inputs & outputs specified in the build dependency graph) which improves caching but is incorrect since that's not what the process actually sees. As a trade-off, BuildXL opted to use the real filesystem for read-only directories and the virtual (build graph based) filesystem for writable directories.

[iclanton]

Rush doesn't currently directly control where a project's build drops its files. Rush invokes whatever command is specified in the build script specified in the project's package.json, and then that build script drops files wherever it wants. Our suggested practice has been that files should all be contained in the project's folder, but we don't actually enforce that.

We could include a feature that provides an env variable during projects' build dictating where build output should go, but it'd still be up to the individual projects to drop files in that folder. @nayanshah - what did you have in mind WRT implementation of this feature?

[nayanshah]

Rush also creates symlinks in node_modules & writes project incremental build states in .rush folder. I'm planning to add some configuration to turn those off.

For build scripts, something along the lines of passing output directory environment variable works.

[trusktr]

What project are you trying to use rush on that shouldn't have a node_modules folder in it? Are you using it on a JavaScript/web project?