[BUG] npm install removes linked packages, npm link replaces linked package contents

hdodov commented 3 years ago

I have specified a package in my package.json that I have also linked with a local version I use for development. When I run npm i in the parent package, the symlink is removed, the package is installed from npm, and I use the live version instead, not my local development one.

Later, if I run npm link, the package correctly get linked with my local version, but everything is replaced with the downloaded version, meaning that all my local development files are replaced with the contents of the tarball downloaded from npm. This means I have to go in my package folder, clone the repo again, checkout the files from the master branch... Not to mention that if I have any unsaved work, it would get completely erased with no chance to get it back even via Git - because the .git folder is also deleted.

Current Behavior:

running npm i removes linked packages and installs their published versions instead
running npm link <package> replaces the contents of the symlinked package folder with the published package's tarball contents

Expected Behavior:

running npm i should ignore packages in node_modules that are symlinked
running npm link <package> should only create a symlink and leave everything in the symlinked folder intact

Steps To Reproduce:

Install a package from npm in a project
Run npm link in a different, local version of the package
Run npm link <package> in the project
Run npm i
Linked package is removed
Run npm link <package> in the project
Local version is replaced with the published tarball contents

Environment:

OS: Microsoft Windows 10 Pro, 10.0.18363 Build 18363
nvm: 1.1.7
Node: 15.4.0
npm: 7.0.15 (can't install 7.2 due to nvm)

jeremy-coleman commented 3 years ago

Yikes, thats scary.

ljharb commented 3 years ago

@hdodov in what way would nvm prevent npm install -g npm@7? edit: ah, i see you're using nvm-windows, not actual nvm.

darcyclarke commented 3 years ago

@hdodov can you install the latest v7 & try reproducing this? (ie. npm i -g npm@7)

hdodov commented 3 years ago

@darcyclarke I can't reach the point of being able to reproduce. With a fresh install of Node 15.8.0 and npm 7.5.1, I create a symlink by running npm link in the package folder:

added 1 package, and audited 4 packages in 231ms

...which creates the symlink (oblik). However, when I open it, it seems to be broken:

...and indeed if I try to run npm link oblik in a project, I get:

npm ERR! code ENOENT
npm ERR! syscall open
npm ERR! path C:/Program Files/nodejs/node_modules/oblik/package.json
npm ERR! errno -4058
npm ERR! enoent ENOENT: no such file or directory, open 'C:\Program Files\nodejs\node_modules\oblik\package.json'    
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\dodov\AppData\Local\npm-cache\_logs\2021-02-04T07_50_48_106Z-debug.log

Here's the log: 2021-02-04T07_50_48_106Z-debug.log.

I tried to link and unlink several times, but nothing seems to fix it. I guess that's for another issue, though? Let me know if I should open a new one. Maybe there's some issue with nvm for Windows? Although I had no issues with it specifically.

darcyclarke commented 3 years ago

@hdodov good to hear the initial issue seems to be resolved. I have heard of other folks using nvs or volta on Windows for version management (there's also nvm-windows, not sure if that's what you're using already, but I'm not sure how well it's supported) - could you maybe try one of those & see if linking breaks there as well?

hdodov commented 3 years ago

@hdodov good to hear the initial issue seems to be resolved.

@darcyclarke actually, I don't know if it's resolved. That was my point in my previous comment - I couldn't get past the npm link <package> part to see what the npm install after it would do - whether the link would stay or my child project would get erased.

I'll experiment with other version management tools and post the update here. 👍

hdodov commented 3 years ago

I installed Volta 1.0.1 and I didn't have the problems mentioned in my previous comment. I tried to reproduce the issue and it seems that it's half solved. Here's what I tested:

npm link oblik (without the package in my package.json): adds the package as a link
npm i: removes the linked package (maybe not expected?)
npm i oblik: adds the package from npm and lists it in package.json
npm link oblik: changes the package from the downloaded version to the local one via symlink
npm i: changes the package from the linked version to the downloaded one, even though it's listed in package.json (not expected)

Basically, whenever I run a command that installs or uninstalls a package, the linked package is switched with the downloaded version if it's listed in package.json, or the link is simply removed if it isn't listed.

What's different this time is that my child package (the one I'm linking) didn't get erased at any point during the testing, which is wonderful. The only real issue now is that I have to run npm link <package> every time my dependencies change.

OS: Microsoft Windows 10 Pro, 10.0.18363 Build 18363 Volta: 1.0.1 Node: 15.8.0 npm: 7.5.1

tangobravo commented 2 years ago

npm i replacing any links does appear to be the intended default behaviour. With --save you can update package.json so it will persist through npm install operations. https://docs.npmjs.com/cli/v8/commands/npm-link#caveat

ssbarnea commented 2 years ago

As package.json is almost for sure tracked, this cannot reliably be used on CI as it is lightly that the pipeline will fail because a tracked file was modified (a common security measure for ensuring that what you test is what you track).

joepie91 commented 2 years ago

npm i replacing any links does appear to be the intended default behaviour.

That behaviour doesn't make any sense for the intended usecase, though, and this is not how it worked in old npm versions (before npm link was broken) either. The whole point of this functionality is to temporarily substitute a dependency while developing or debugging on multiple packages at once.

So neither "overriding package.json with the substitute" nor "have the substitute be deleted every time you add a completely unrelated dependency" make sense here. The correct behaviour would be to retain the substitute across npm commands, without changing the package.json (and ideally not changing the package-lock.json either).

Edit: This was supposed to be fixed in v7, by the way.

eliot-akira commented 2 years ago

The first part of the originally reported issue is still happening for me, with npm version 8.12.1.

running npm install removes linked packages and installs their published versions instead

Apparently this is now the intended behavior, but it wasn't always like this. I used to have a workflow using npm link to temporarily substitute modules for local development. Running npm install would keep any symlinked modules - I thought that was the whole purpose of npm link.

It was working well for a number of projects and linked modules, but at some point (I seem to recall it started in npm version 5.x) the behavior changed, where npm install would remove and replace linked modules.

(This was never solved, and after a while I had to switch to yarn which works as expected for my local setup. I'm actually trying to move back to using npm fully since it's the canonical package manager for Node.js - and I have issues with the direction yarn is going, but that's another story..)

With the current behavior, every time I run npm install, I have to re-run npm link <package> to manually restore the removed links.

If this is the intended default behavior, would it be possible to add a CLI option to preserve linked modules as before?

vzakharov-rxnt commented 2 years ago

I find it odd the one of the most important development workflows hasn't been fixed for almost 2 years now. I am using Node 16.15.0.

ssbarnea commented 2 years ago

@vzakharov-rxnt While this bug is marked like linking would be broken only on Windows, the reality is that npm link is also broken on platforms fully supporting symlinks too, where npm i does override the links. That is why we switched from npm to yarn on one of our projects (vscode-ansible) last month. We also got some serious performance benefits and much better UX for upgrading dependencies but these were only perks, the linking was key.

vzakharov-rxnt commented 2 years ago

@ssbarnea We had issues with unit tests when dependencies were installed with yarn. Had to switch back to npm. Unit tests are using karma/jasmine.

Vasile-Peste commented 1 year ago

After many years using npm link it's still one of my worst nightmares...

scott-lc commented 1 year ago

@Vasile-Peste - The link package along with a prepare hook has been an excellent alternative for me.

npx link = A safer version of npm link.

{
  "scripts": {
    "prepare": "(test -f link.config.json && npx link) || true"
   ...
 }
}

mryellow commented 1 year ago

npm link A should not invalidate npm link B.

Both packages should exist as symlinks.

pjdon commented 2 months ago

Still getting this with npm version 9.5.0. I get the message removed 1 package when running npm install. When re-running npm link <source-package> it just recreates the symlink, doesn't add a tarball or anything.

tjsr commented 2 months ago

I've been having issues with my dev workflow and this lately. While it seems counter-intuitive to blow away a linked local package, I've decided that the answer is to have an npm postinstall script that then links the packages you want to know about relative to the project.

To help me around this I'm going to write a script/tool in the coming day or so that goes through and looked at linked global packages (either symlink or junction), and calls link on each of either the specified one, or all of them.

npm / cli