search

npm / cli

the package manager for JavaScript
https://docs.npmjs.com/cli/
Other
8.5k stars 3.18k forks source link

[BUG] `npm install` creates unsync `package-lock.json` #6787

Open regseb opened 1 year ago

regseb commented 1 year ago

Is there an existing issue for this?

This issue exists in the latest npm version

Current Behavior

When I run npm install, the generated package-lock.json file isn't synchronized with the package.json file. The npm ci command fails. If I run npm install a second time: the package-lock.json file is modified (and synchronized).

Expected Behavior

npm install creates a package-lock.json file synchronized.

Steps To Reproduce

The directory at the end with the files package.json, package-lock.json, package-save.json, and the directory node_modules/: testcase.zip

Environment

//registry.npmjs.org/:_authToken = (protected)

; node bin location = /usr/bin/node ; node version = v20.5.1 ; npm local prefix = /home/regseb/dev/testcase ; npm version = 10.0.0 ; cwd = /home/regseb/dev/testcase ; HOME = /home/regseb ; Run npm config ls -l to show all defaults.



### Related issues / pull request

- https://github.com/npm/cli/issues/4859
- https://github.com/npm/cli/issues/5854
- https://github.com/npm/cli/issues/6378
- https://github.com/npm/cli/issues/7793
- https://github.com/npm/cli/pull/5301
lukekarrys commented 1 year ago

Triaged and this is happening on both npm 9 and npm 10.

regseb commented 1 year ago

I reproduce the bug with this package too:

{
  "name": "testcase",
  "version": "1.0.0",
  "dependencies": {
    "addons-scanner-utils": "9.3.0",
    "htmlhint": "1.1.4"
  }
}
siemhesda commented 1 year ago

There is a weird thing that happens within the package.json that you may have missed out @regseb

When you do the first npm install, the package.json changes a bit from

{
  "name": "testcase",
  "version": "1.0.0",
  "devDependencies": {
    "addons-linter": "6.13.0",
    "htmlhint": "1.1.4"
  }
}

to

{
  "name": "testcase",
  "version": "1.0.0",
  "devDependencies": {
    "addons-linter": "6.13.0",
    "htmlhint": "1.1.4"
  },
  "dependencies": {
    "npm": "file:.."
  }
}

I am not sure if the second part is expected and should be added by npm. Otherwise, it still isn't the culprit on why the package-lock.json is not synced with package.json on the first npm install. Does this only happen with node-fetch @regseb?

@lukekarrys can you confirm that this issue happens on your end too? (Modification of the package.json)

regseb commented 1 year ago

There is a weird thing that happens within the package.json that you may have missed out @regseb

I tested with this two package.json and they aren't modified after npm install (with npm 10.1.0). And the package.json file in the testcase.zip (from my first message) hasn't been modified.

{
  "name": "testcase",
  "version": "1.0.0",
  "devDependencies": {
    "addons-linter": "6.13.0",
    "htmlhint": "1.1.4"
  }
}
{
  "name": "testcase",
  "version": "1.0.0",
  "dependencies": {
    "addons-scanner-utils": "9.3.0",
    "htmlhint": "1.1.4"
  }
}

Does this only happen with node-fetch @regseb?

This is the first time I've had this problem. I didn't check if I had other dependencies that were also loaded twice with different versions in dependencies and peerDependencies. But I don't think the problem is related to node-fetch.

siemhesda commented 1 year ago

There is a weird thing that happens within the package.json that you may have missed out @regseb

I tested with this two package.json and they aren't modified after npm install (with npm 10.1.0). And the package.json file in the testcase.zip (from my first message) hasn't been modified.

I noticed it is behaving so on my end since I was working on workspaces. In a normal set up it doesn't modify the package.json

Does this only happen with node-fetch @regseb?

This is the first time I've had this problem. I didn't check if I had other dependencies that were also loaded twice with different versions in dependencies and peerDependencies. But I don't think the problem is related to node-fetch.

I asked because all the examples you have given have node-fetch as dependencies or peerDependencies right?

regseb commented 1 year ago

I asked because all the examples you have given have node-fetch as dependencies or peerDependencies right?

I only have one case because addons-scanner-utils is a dependency of addons-linter:

testcase
+-- addons-linter 6.13.0
|   +-- addons-scanner-utils 9.3.0
|       +-- node-fetch 2.6.11 (peer)
+-- htmlhint
    +-- node-fetch ^2.6.2

My second example is just a version with one less intermediate step.

allanpaiste commented 1 year ago

TL;DR: this goes back to 7.0.9 which was the first release that caused the locked versions to be ignored on npm ci. The error Invalid: lock file's ... does not satisfy ... was introduced in 8.4.1 and is indeed correctly thrown as a guard against sneaky upgrades, but sneaky upgrades are actually caused by these two changes from the past (both still cause npm ci to misbehave):

Thus, the last glitch-free npm version is: 7.0.8

(As for the currently implemented installation error - it would serve much better as a test-case in integration test suite rather than user-facing functionality where it just causes confusion due to config-vs-lock being completely valid when it is thrown).

Our steps that get us to the error:

  1. install package X with semver lock on a project Y: ^1.2.3 (new package-lock.json gets generated)
  2. release new version of package X: 1.3.0
  3. run npm ci for project Y.

EXPECTED: install still goes through and installs version 1.2.3 (as package-lock.json indicates) ACTUAL: process errors out and informs that lock file's 1.2.3 does not satisfy 1.3.0

Some history:

This all goes way back to v7.X which got a bug introduced to it of which there have been attempts on fixing it ever since with varying level of success (the change in 8.4.1 being one of them).

The actual issue had been around longer and that specific release (and the error we have all been seeing) just represents an attempt to rein in the whole situation.

The hunt for the cause:

Though I initially focused on tracing down where the error originates from, it became since clear that instead of hunting for a release that does not fail the npm ci with the error - I should instead be searching for behaviour switch that the 8.4.1 code was catching. Little did I know that I'd be going back to the v7.X.

Below then are the findings by just going through the versions one-by-one to identify where the break happened (given in 3 steps where versions represent versions found in: package.json > package-lock.json > node_modules

Thus the issue was somewhere in v7.0.8...v7.0.9.

From there, when doing another round of blind pin-pointing leads to this commit: 0e58e6f.

Starting from that specific commit, you'll start getting newer version installed than what is specified in the lock (which in turn in 8.4.1 was turned into a throw).

Most notably it's something about the change in build-ideal-tree.js

-      // didn't find a parent for it, but we're filling in external
-      // link targets, so go ahead and process it.
-      if (this[_follow] && !link.target.parent && !link.target.fsParent) {
+      // didn't find a parent for it or it has not been seen yet
+      // so go ahead and process it.
+      const unseenLink = (link.target.parent || link.target.fsParent)
+        && !this[_depsSeen].has(link.target)
+      if (this[_follow]
+        && !link.target.parent
+        && !link.target.fsParent
+        || unseenLink) {
         this.addTracker('idealTree', link.target.name, link.target.location)
         this[_depsQueue].push(link.target)
       }

Now when trying out newer version, I bumped into another "checkpoint" where same thing started to happen again (though this time around instead of getting a silent install, one would actually see the infamous error that is stated at the original report of the issue).

The second change was also done to build-ideal-tree.js, this time within the 8.6.0 release.

       .then(tree => {
+        // search the virtual tree for invalid edges, if any are found add their source to
+        // the depsQueue so that we'll fix it later
+        depth({
+          tree,
+          getChildren: (node) => [...node.edgesOut.values()].map(edge => edge.to),
+          filter: node => node,
+          visit: node => {
+            for (const edge of node.edgesOut.values()) {
+              if (!edge.valid) {
+                this[_depsQueue].push(node)
+                break // no need to continue the loop after the first hit
+              }
+            }
+          },
+        })
         // null the virtual tree, because we're about to hack away at it
         // if you want another one, load another copy.

Sadly enough, neither of the changes have proper test coverage which does not really inject confidence to the reader in terms of knowing that the base functionality of the installer remains the same (second change just modifies existing tests which just shadows potential new bug).

As such, would appreciate if someone else would take it from here.

Conclusion:

  1. Reverting the change 0e58e6f will make npm ci behave expectedly up to version 8.5.5.
  2. Reverting the change bd96ae4 (together with the one above) will make npm ci behave expectedly up to version 10.2.0 (latest).

Whatever these two changes claimed to fix - they introduced new issues. So those fixes should be re-introduced with proper test-coverage or re-reported as issues after reverting the broken fixes (I'd rather have those bugs back than the main intent of lock file being broken).

allanpaiste commented 1 year ago

Perhaps @isaacs @nlf @lukekarrys can comment on the changes mentioned above and what they're actually addressing. Could these pieces be re-imagined with provided unit/integration tests that would both guard the pre-7.0.9 functionality and also prove that the changes fix whatever they were supposed to fix.

allanpaiste commented 1 year ago

@siemhesda are you still involved in looking into this issue?

siemhesdagit commented 1 year ago

@siemhesda are you still involved in looking into this issue?

Hi @allanpaiste I have been away for a while but I'm back on this

agdimech commented 5 months ago

I am also seeing this issue occuring when installing @aws/pdk.

Repro:

npm init && npm i && npm ci.

running npm i again resolves the issue and npm ci will work.

Lonli-Lokli commented 5 months ago

It happens in my project too but strangely happens only in Github Action (linux-based) and not on my Windows machine

wesleytodd commented 3 months ago

Hey @wraithgar, we just hit this issue. We did some debugging and are pretty sure the detailed comment above is the cause. I am not sure the conclusion in that is right, but just wanted to flag this as an ongoing issue in latest 9.x (have yet to test in 10.x as the app it happened to is not using 10 yet).

Saibamen commented 1 month ago

Maybe you can find some useful info from my duplicated issue: https://github.com/npm/cli/issues/7793 (also with repro repository, few GitHub Actions and another package-lock.json diff (with better Markdown formatting) and workarounds)

kyle-leonhard commented 1 month ago

Is there a good workaround for this beyond rerunning npm install?

mills-andrew commented 3 weeks ago

Is there a good workaround for this beyond rerunning npm install?

"overrides": {
    "chokidar": "4.0.1"
},

resolved the issue for me. Place in your package.json