[BUG] npm install removes packages when workspaces are in use

[Blackclaws]

Is there an existing issue for this?

• [X] I have searched the existing issues

This issue exists in the latest npm version

• [X] I am using the latest npm

Current Behavior

I'm using npm workspaces to link multiple projects together locally for development. As I am not using a pure node js development setup but there are other technologies involved I require that each project has a local node_modules that can be resolved.

I therefore create a parent package.json linking to the subfolders as workspaces. I then run npm install.

Afterwards I symlink the node_modules directory to the workspace children.

It seems that this symlink is what breaks npm install.

On running npm install a second time, dependencies get uninstalled. Running npm install again reinstalls them and so and so forth.

Expected Behavior

Npm installs resolved packages once and further npm install commands are idempotent.

Steps To Reproduce

1. Create a directory and a package.json:

   {
   "name": "testlink",
   "private": true,
   "workspaces": ["./test"]
   }

2. Create a directory test and add a package.json:

   {
   "name": "test",
   "version": "1.0.0",
   "devDependencies": {
   "semantic-release": "^19.0.5"
   }
   }

3. Run npm install in the workspace parent
4. Run ln -s ../node_modules node_modules in the test directory
5. Run npm install in the workspace parent
6. Run npm install in the workspace parent

Environment

• npm: 8.19.2
• Node.js: 16.15.0
• OS Name: Archlinux

[ljharb]

node_modules can’t be a symlink; why do you need that step at all?

[Blackclaws]

I have a dotnet project that consumes npm packages that are restored to node_modules. When I have the project checked out as normal this isn't a problem. When I use the project as part of a workspace arrangement where I want to work on multiple projects simultaneously this only works if the dotnet build has the node_modules folder where it is expected.

[ljharb]

Then if it needs node_modules, it should have a package.json and npm install should be ran there.

[Blackclaws]

Which is what I'm doing. But if you use workspaces then npm install delegates itself to the directory containing the package.json that declares the workspaces. You end up with a node_modules folder there which works fine for nodejs due to how that one resolves dependencies but doesn't work for any other software that wants to consume from a local node_modules.

Clarification: If you run npm install on any package.json that is part of a workspaces arrangement no local node_modules folder is created for those.

[ljharb]

If other software isn’t matching node’s resolution algorithm for files inside node_modules, it’s broken - file an issue there.

[Blackclaws]

If other software isn’t matching node’s resolution algorithm for files inside node_modules, it’s broken - file an issue there.

This isn't helpful in the least. Its not broken, dotnet simply isn't javascript software that should have to care about node_modules or any magic way of how nodejs resolves packages.

As part of the build process of our software I rely on the fact that if we distribute npm packages I can restore them to a node_modules folder local to my project. Which is the default behaviour of npm install. I simply expect that npm install doesn't break in the workspace setting when I symlink node_modules to the main node_modules folder, or alternatively that npm install for workspaces creates a node_modules folder for each of the workspace packages.

The fault lies with npm and the current workspace implementation not with a different software not using nodes module resolution algorithm for packages that it doesn't even natively read.

[PocketlawErik]

@Blackclaws Have you tried using the npm alternative pnpm? 💡

It may offer better support for your use case. 🙂

[fabien]

Okay, so for whatever reason I had a symlink between a workspace's node_modules pointing up to one in the monorepo root. I cannot recall setting up this symlink myself, but who knows. Took me almost a day to figure out what was going on, when I saw npm install install a couple of thousand dependencies, and then removing at least half of them on the next npm install, and this back and forth again.

In the meantime, I went all in on pnpm only to find out that a NextJS app in the workspace failed to run (https://github.com/pnpm/pnpm/issues/2743) so I had to revert back to npm workspaces in the end, while still not knowing why npm install was acting the way it did.

Perhaps some kind of warning could be introduced when symlinked node_modules is found in a workspace?

[Blackclaws]

The fact that npm doesn't deduplicate the node_modules folder if it all references the same one is a bug to begin with imho.