Support Rush / Package Managers with Native Dependencies

[octogonz]

We are currently using NVM in our monorepos, but it is frustrating that NVM does not support Windows. Instead, you need to install a separate tool nvm-windows for Windows machines. I like that Volta is a single solution for every platform.

How NVM works

Consider this hypothetical folder layout with NVM:

• ~/git/my-repo/.nvmrc - specifies to use Node.js 14 for all projects in this monorepo
• ~/git/my-repo/apps/app1/package.json
• ~/git/my-repo/apps/app2/package.json
• ~/git/my-repo/libraries/.nvmrc - overrides only lib3 and lib4 to use Node.js 12
• ~/git/my-repo/libraries/lib3/package.json
• ~/git/my-repo/libraries/lib4/package.json

And then I would run npm install -g @microsoft/rush to install my CLI tool. If I run nvm in ~/git/my-repo/apps/app1 it gives me Node 14. Whereas if I run it in ~/git/my-repo/libraries/lib3 it gives me Node 12.

How Volta is different

1. Volta seems to require a setting like "volta": { "node": "12.20.0" } in every single package.json file. In a large monorepo, there could be hundreds of these files. That would be very cumbersome, and difficult to keep up to date.
2. Compared to .nvmrc , storing settings in package.json has the downside that overrides can only happen in paths containing a package.json file. And even if I created a file ~/git/my-repo/libraries/package.json it would not be honored by Volta, since ~/git/my-repo/libraries/lib3/package.json will always take precedence (even if there is no "volta" setting in that file).
3. Also, we generally discourage people from storing tool settings in package.json, because that file has no schema and thus cannot be validated. For example, if someone misspells "voltaa": { "node": "12.20.0" } this mistake must be silently ignored -- there is no way to report an error. Whereas a tool-specific file like .nvmrc is easy to validate.
4. Volta seems to require @microsoft/rush (and any other globally installed tools) to be added as devDependencies to each package.json. Otherwise they run using the default Node.js instead of the specified version. That is confusing and difficult to keep consistent across all projects.

Suggested fix

Here's a couple possible solutions:

1. Without getting into a big design discussion, it seems that maybe Volta could simply provide an "NVM compatibility mode", where it looks for .nvmrc files and ignores the "volta" settings in package.json. I realize this gives up some Volta benefits, but it's still better than using NVM (because it works on Windows, too).

2. Or if we want to improve Volta's design, consider something like this: Instead of specifying settings in package.json, you specify them in .volta.json whose format might be like this:

   ~/git/my-repo/libraries/.volta.json

   {
    "node": "12.12.0",
    // Yarn is not special or handled differently from any other NPM tool
    "packageManager": "pnpm",
    "packageManagerVersion": "1.2.3",
   
    "globalPackages": {
      "@microsoft/rush": "^1.2.3",
      "eslint": "^1.2.3"
    }
   }

[charlespierce]

Hi @octogonz, thanks for the detailed report! There are a couple of interrelated issues here:

1. We can't directly support .nvmrc as the specifier, since it allows for fuzzy versions / version-like values, which would break Volta's goal of reproducibility (for more discussion there, see #905 and the later discussions in #282)
2. For global packages that aren't declared as dependencies, we use the version that they were installed with (see the docs for a discussion of how we pin the version). We do this because arbitrary packages could have native Node modules that only work on one major version of Node. That's also why we can't, in general, switch the Node version with which we call a package.

All that being said, Volta does support workspaces without having to duplicate the Node version in every project, see Workspaces in the docs. It's not as ergonomic as only having files on disk, however since Volta re-evaluates the execution context on every call, we've generally opted to avoid speculative I/O for performance reasons. Additionally, the "extends" entry in package.json can point to any JSON file, it doesn't need to be another package.json. The extends chain also tracks all of the dependencies and devDependencies of the files it reads (even if they aren't named package.json), so if you have @microsoft/rush declared as a dependency of the top level of the monorepo, then all of the subprojects will detect that with the appropriate extends entries.

Separately, we do have an issue #282 to support a volta-specific file in addition to package.json for specifying Volta's configuration. A solution still needs design work to define the specifics and how it would interact with the existing resolution, but we're definitely open to suggestions / recommendations.

[octogonz]

so if you have @microsoft/rush declared as a dependency of the top level of the monorepo, then all of the subprojects will detect that with the appropriate extends entries.

That's not how Rush works. Rush does not permit a package.json file at the root of the monorepo specify to dependencies (because it would introduce phantom dependencies). Conventionally, a Rush repo does not have any package.json file at its root.

Separately, we do have an issue #282 to support a volta-specific file in addition to package.json for specifying Volta's configuration. A solution still needs design work to define the specifics and how it would interact with the existing resolution, but we're definitely open to suggestions / recommendations.

@charlespierce I think issue #282 could meet our needs, if it allowed the Node.js version to be specified in *one place for the entire monorepo.

[charlespierce]

@octogonz I see, thanks for the link to the docs! Based on how it appears to work, it seems like Rush acts as a meta-dependency, similar to the package managers (ironically creating a phantom dependency on itself). Given that, I think the only way to really support it would be to treat it similarly as part of the Platform that Volta determines, as opposed to being "just" another package.

Since it isn't declared anywhere as a dependency of the monorepo, there's no way for Volta to determine that it's supposed to be a local tool (using the project-local Node), instead of a global tool that's coincidentally being called inside a package. So we would need to make that dependency explicit by including it in the settings that define the other meta-dependencies (Node and the package managers).

[octogonz]

(ironically creating a phantom dependency on itself)

The main idea of a "phantom" dependency is that you can do require("some-thing") inside a project folder, and it will succeed, even though "some-thing" is NOT declared in the package.json for that project. This ability to import undeclared dependencies creates all the problems detailed in that article.

Whereas if you do npm install --global some-thing, it does not create a phantom dependency, because Node.js require() does not look in the global folder. But it does look for node_modules in the parent folders. The specific lookup procedure is spelled out in the Node.js docs.

Rush does not create a phantom dependency on itself.

Since it isn't declared anywhere as a dependency of the monorepo, there's no way for Volta to determine that it's supposed to be a local tool (using the project-local Node), instead of a global tool that's coincidentally being called inside a package.

@charlespierce Rush is normally installed via npm install --global @microsoft/rush. Volta should have a general story for how to handle such tools. Package managers like yarn and pnpm are one example of globally installed tools, but it would be unrealistic to assume that is the ONLY valid case.

The NVM way is that people install global packages manually, and they get stored in a special folder, which gets remapped depending on which Node.js version is active. This special folder is outside the Git working directory, so it does not introduce phantom dependencies.

The Volta way seems to be to always declare global packages as devDependencies in a package.json file somewhere. That concept is actually fine with Rush. Our requirement is simply that they must not get installed in a node_modules folder at the root of the monorepo, because that creates phantom dependencies. (A .volta.json file is even nicer, but it's not required to solve this problem.)

So perhaps making Volta work with Rush is maybe not as difficult as it seems. I'm sorry that I don't have time to study Volta more closely to work out the exact details. (I'd be willing to chat offline however, if that would help sort this out. If you don't have Twitter, I'm also reachable in Rush's chat room).

[charlespierce]

Chatted out-of-band a bit with @octogonz today, and the core issue appears to be Volta and Rush not interacting well together. Rush operates in a similar space to the package managers, however it has native dependencies and so we can't treat it the same as our existing package managers and freely swap Node versions. The summary we reached was:

• EITHER Volta needs a mode of operation where you can "reinstall the world" when you install a new Node.js version
• OR we need a way for Rush-on-v12 to work correctly in a repo that wants Node v14
• OR we need to guarantee Rush has no native bindings, so the same installation can be invoked by different Node.js runtimes (and then you model it as a package manager)

Ultimately, supporting package managers with native dependencies (or pseudo-managers like Rush) will likely be something we need to do eventually.

[octogonz]

@charlespierce thanks for taking the time to discuss the requirements for Rush! 🙂🙏

[ramya-rao-a]

Rush operates in a similar space to the package managers, however it has native dependencies

@octogonz This is no longer the case, right? From what I recall the native dependencies came from the Identity package which has now been resolved

[ljharb]

@octogonz nvm does support windows now via WSL2, git bash, and cygwin, fwiw.

[ChangeHow]

Hi, I met the same problem, is there any update?

[octogonz]

This is no longer the case, right? From what I recall the native dependencies came from the Identity package which has now been resolved

Yes, we have been working to eliminate native (node-gyp) dependencies from the main Rush CLI package. (Rush plugins still load them however.)

[ChangeHow]

This is no longer the case, right? From what I recall the native dependencies came from the Identity package which has now been resolved

Yes, we have been working to eliminate native (node-gyp) dependencies from the main Rush CLI package. (Rush plugins still load them however.)

So, the right way for Rush repos is pin node/pnpm/etc. in repo itself? not pin them for entire project right?