RFE: Separate search paths for distribution-packaged NPMs and globally-installed ones

[sgallagher]

Is your feature request related to a problem? Please describe.

When distributing Node.js in a Linux distro (in my case, Fedora), we run into problems when users call npm -g update (most often, this is npm -g update npm which upstream NPM provides as user-feedback when running the CLI).

The problem is that if NPM replaces content in /usr/lib/node_modules/ with the NPM-retrieved versions, it will be replaced the next time the RPM is updated (RPM assumes that it has sole ownership of the contents of /usr/lib/node_modules/*). So the updated files will be clobbered and the results may lead to an installation that has portions of both modules, neither working properly.

(A similar problem arises when one uses npm -g install <NPM> and later someone installs that same NPM from an RPM package due to a packaging dependency.)

Describe the solution you'd like I would like for Node.js to support three search paths instead of its current two. The paths, in order of priority (the first one that provides the module is used):

1. The node_modules path in the current directory.
2. An install location used by npm install --global. (Such as /usr/lib/npm/node_modules/)
3. An install location reserved for distribution packaging, (Such as /usr/lib/node_modules/)

Ideally, those paths would be distro-configurable as a ./configure option, but as long as they are relative to --prefix, that's not a blocker.

Describe alternatives you've considered Fedora currently ships a patched NPM that removes the recommendation to update the global version, but this is just a user-experience hack. (And given how many READMEs out there start out with "Run npm install -g <something>", it's not really solving the problem.

[devsnek]

node doesn't use npm's global install location at all, it doesn't know it exists. If you want to suggest a change to npm i suggest opening a thread on https://npm.community.

[mhdawson]

@devsnek from my understanding of the request this would not be a change to npm as I don't think it involves the installation of packages, instead, it would affect where node looks for installed modules. @sgallagher correct me if I'm wrong on that front.

[mhdawson]

@bastien-roucaries since you mentioned you work on debian/ubuntu I'm wondering if you have a similar problem to the one outlined in this issue?

[bastien-roucaries]

FHS dictate local install should go to /usr/local/.

We use on debian https://github.com/nodejs/node/pull/29782 and we search first /usr/local/share/nodejs then /usr/local/lib/currentarch/nodejs then /usr/local/lib/nodejs then /usr/share/nodejs then /usr/lib/currentarch/nodejs then /usr/lib/nodejs then

you need a patch for npm to allow an option to specify system install dir but from node search path the problem if considered solved

[sam-github]

@sgallagher Have you seen https://github.com/nodejs/node/pull/29782, is it at all related to this?

cc: @gireeshpunathil , who has seen a lot of users have trouble related to node/npm installation and update

[sgallagher]

@devsnek What I'm talking about is this: https://github.com/nodejs/node/blob/master/lib/internal/modules/cjs/loader.js#L1085

I stumbled across https://github.com/nodejs/node/pull/29782 which is similar (it addresses allowing a configure option to change the global search path) but it doesn't include the separation I was looking for between NPM-globally-installed modules and distro-packaged ones.

Though, as I rethink things, in order to make the fewest changes, I think the directories I suggested above were wrong; NPM should continue to install by default to $PREFIX/lib/node_modules and distro packages could go to $PREFIX/lib/nodejs/node_modules or similar. But as long as these paths are configurable, that would be sufficient.

[bastien-roucaries]

We pass the following configure option (for now we do not search /usr/local but we plan to do) --prefix=/usr --arch-triplet=$(DEB_HOST_MULTIARCH) \ --node-relative-path="lib/$(DEB_HOST_MULTIARCH)/nodejs:share/nodejs:lib/nodejs"

[sgallagher]

@sam-github Yes, it's related but solving a larger problem around not having NPM clobber distro packages as well.

[bastien-roucaries]

@sgallagher FHS dictate admin package should go to /usr/local

[devsnek]

lib/node isn't used by npm, it uses lib/npm. rpm -i shouldn't create artifacts that require() can use either.

[sgallagher]

@sgallagher FHS dictate admin package should go to /usr/local

Yes, the NPM installs probably should go to /usr/local, since you mention it. The RPM installs belong in /usr though. We don't need to debate interpretations of the FHS here; as long as it's configurable we can resolve that later.

lib/node isn't used by npm, it uses lib/npm.

> npm install -g grunt
npm WARN checkPermissions Missing write access to /usr/lib/node_modules/grunt/node_modules/coffee-script

rpm -i shouldn't create artifacts that require() can use either.

Can you explain what you mean here? The whole point of using RPM is to install code that can be used by anyone on the system. Or are you saying that it should be expected to have to use npm symlink? I don't think that's a good user-experience.

[devsnek]

sorry typo, i meant lib/node_modules (and btw it can be changed, i use ~/snek/.npm/lib/node_modules)

Can you explain what you mean here? The whole point of using RPM is to install code that can be used by anyone on the system.

back in the day we had .node_modules and such but it was terrible because everything would conflict and be slow so now we really recommend that people avoid those, and they really only exist for historic reasons. now you should use dependencies local to each project. global npm installs aren't supposed to be required (and usually can't be, unless you configure npm oddly), they're supposed to add binaries to your path, like webpack-cli and whatnot. require has no concept of these installs by default, it doesn't know they exist. This is why I said that rpm -i creating globally accessible packages to be required was a bit odd.

[sam-github]

@devsnek

now you should use dependencies local to each project. global npm installs aren't supposed to be required, they're supposed to add binaries to your path, like webpack-cli and whatnot

Doing what you describe here is exactly what doesn't work well for distros.

Allowing these kind of binaries to be packaged for global use, and also for users to install updates to them (or new ones that don't yet have packages), doesn't work, because node reads from one global location, npm writes to that single global location, and everyone clobbers each other.

cc: @nodejs/npm

[devsnek]

if distros feel the need to do this because npm/yarn/pnpm/entropic/etc isn't floating your boat, why doesn't targeting /usr/lib/node work? it's already a global search path (along with .node_modules), and npm doesn't ever touch it.

[sgallagher]

OK, I just re-examined some of our Node packaging tools (written by my predecessor on Fedora Node.js maintenance) and realized I was mistaken. They aren't installed in a way as to make them globally accessible for requires(). The tooling for dependent NPMs automatically creates symlinks into their own paths. So it's behaving like you think it should. So I guess I can withdraw that part of my request.

So, thinking about it more, with the patch from Debian, we could use --node-relative-path=lib/nodejs and then modify the NPM package to use /usr/local/lib/node_modules instead of /usr/lib/node_modules when installing globally.

Then npm install -g won't clobber our RPM-installed packages anymore in that case.

So that might be sufficient for our purposes.

if distros feel the need to do this because npm/yarn/pnpm/entropic/etc isn't floating your boat, why doesn't targeting /usr/lib/node work? it's already a global search path (along with .node_modules), and npm doesn't ever touch it.

Honestly, I didn't know about it, but it seems like maybe we won't need it after all.

[sgallagher]

Just to circle back around on this, I've figured out a way to accomplish this: The Fedora npm package will now drop a configuration file in /etc/npmrc (and a symlink to it from /usr/etc/npmrc)that includes the contentsprefix=/usr/local. The Fedora packages will continue to installnpminto the/usrhierarchy, but as best I can tell from reading the NPM sources, this will result innpm -gcommands working with the/usr/localhierarchy. Since Fedora's default$PATHincludes/usr/localin preference to/usr`, this should work in the majority of cases.

So, I think we can close this ticket, unless anyone sees any serious side-effects to setting prefix=/usr/local in /etc/npmrc.