[BUG] Platform-specific optional dependencies not being included in `package-lock.json` when reinstalling with `node_modules` present

[JustinChristensen]

Is there an existing issue for this?

• [X] I have searched the existing issues

This issue exists in the latest npm version

• [X] I am using the latest npm

Current Behavior

[user@host:foo] $ npm -v
8.8.0
[user@host:foo] $ node
Welcome to Node.js v16.14.2.
Type ".help" for more information.
> process.arch
'arm64'

I'm working on a team that utilizes a mix of x64-based and m1-based macs, and has CI build processes that uses musl. We're seeing that npm is skipping platform-specific optional dependencies for packages such as @swc/core as a result of the package-lock.json file being generated without all of them included. In our case, this then causes linting to throw an exception, because one of our eslint plugins depends on @swc, which depends on having the platform specific @swc package also installed.

There seems to be at least two stages of cause to this. Firstly, when installing @swc/core from a clean slate working directory npm generates a package-lock.json with all of the optional dependencies for @swc/core listed:

[user@host:foo] $ npm install @swc/core
[user@host:foo] $ grep 'node_modules/@swc/core-*' package-lock.json
    "node_modules/@swc/core": {
    "node_modules/@swc/core-android-arm-eabi": {
    "node_modules/@swc/core-android-arm64": {
    "node_modules/@swc/core-darwin-arm64": {
    "node_modules/@swc/core-darwin-x64": {
    "node_modules/@swc/core-freebsd-x64": {
    "node_modules/@swc/core-linux-arm-gnueabihf": {
    "node_modules/@swc/core-linux-arm64-gnu": {
    "node_modules/@swc/core-linux-arm64-musl": {
    "node_modules/@swc/core-linux-x64-gnu": {
    "node_modules/@swc/core-linux-x64-musl": {
    "node_modules/@swc/core-win32-arm64-msvc": {
    "node_modules/@swc/core-win32-ia32-msvc": {
    "node_modules/@swc/core-win32-x64-msvc": {

And it only installs the platform specific package:

[user@host:foo] $ ls -l node_modules/@swc/
total 0
drwxr-xr-x  22 user  staff  704 Apr 29 15:39 core
drwxr-xr-x   6 user  staff  192 Apr 29 15:39 core-darwin-arm64

If I then remove my package-lock.json, leave my node_modules directory as-is, and then reinstall, I get:

[user@host:foo] $ rm -rf package-lock.json
[user@host:foo] $ npm install
[user@host:foo] $ grep 'node_modules/@swc/core-*' package-lock.json
    "node_modules/@swc/core": {
    "node_modules/@swc/core-darwin-arm64": {

That is, it then generates a package-lock.json with only the platform-specific dependency that was installed on this machine, and not with the other optional dependencies that should also be listed.

If you delete both node_modules AND package-lock.json, and then re-run npm install, it generates the correct lockfile with all of those optional dependencies listed.

The problem is that then, If the package-lock.json with the missing optional platform-specific dependencies gets checked into git and an x64 user pulls it down, or vice-versa, npm fails to detect that your platform's optional dependencies are missing in the lockfile and just silently skips installing the platform-specific dependency. For example, when I've got a package-lock.json that only contains the x64 @swc package because of the above problem (generated by my coworker on his x64 machine):

[user@host:foo] $ node
Welcome to Node.js v16.14.2.
Type ".help" for more information.
> process.arch
'arm64'
>
[user@host:foo] $ grep 'node_modules/@swc/core-*' package-lock.json
    "node_modules/@swc/core": {
    "node_modules/@swc/core-darwin-x64": {
[user@host:foo] $ ls
package-lock.json package.json

And I then install:

[user@host:foo] $ npm install
added 1 package in 341ms

1 package is looking for funding
  run `npm fund` for details
[user@host:foo] $ ls node_modules/@swc/
core

You can see that it fails to install the arm64 dependency or warn me in any way that the package-lock.json is missing my platform's dependency.

So yeah, two problems:

1. npm is generating an inconsistent package-lock.json when node_modules has your platform-specific dependency installed.
2. When installing from this inconsistent package-lock.json, npm fails to try to correct the problem by comparing the optional dependencies to what's listed upstream

Expected Behavior

1. npm should preserve the full set of platform-specific optional deps for a package like @swc when rebuilding package-lock.json from an existing node_modules tree
2. npm install should warn if the package-lock.json becomes inconsistent because of the first case

Steps To Reproduce

See above.

Environment

• npm: 8.8.0
• Node.js:
• OS Name: OSX
• System Model Name: Macbook Pro

[user@host:foo] $ npm -v
8.8.0
[user@host:foo] $ node -v
v16.14.2
[user@host:foo] $ uname -a
Darwin host.foo.com. 21.3.0 Darwin Kernel Version 21.3.0: Wed Jan  5 21:37:58 PST 2022; root:xnu-8019.80.24~20/RELEASE_ARM64_T8101 arm64

[user@host] $ npm config ls
; "user" config from /Users/user/.npmrc
; node bin location = /Users/user/.nvm/versions/node/v16.14.2/bin/node
; node version = v16.14.2
; npm local prefix = /Users/user/Development/foo
; npm version = 8.8.0
; cwd = /Users/user/Development/foo
; HOME = /Users/user
; Run `npm config ls -l` to show all defaults.

[schjetne]

I am at my wits end here. I've been trying to figure out why my build fails for 6 hours now and then I found this discussion and what I'm dealing with is an NPM bug that's existed for more than a year? Jeeez. How is this not fixed yet? I was supposed to release before my Christmas vacation but that's not looking so great right now.

[smogg]

@schjetne it's not pretty, but npm i @rollup/rollup-linux-x64-gnu --save-optional (mentioned above) fixed CI deployments for me. Enjoy your holiday break!

[schjetne]

@schjetne it's not pretty, but npm i @rollup/rollup-linux-x64-gnu --save-optional (mentioned above) fixed CI deployments for me. Enjoy your holiday break!

Yeah I wish that worked for me but that breaks the packages that depend on it… So making that optional just moves the problem elsewhere.

[ccc1129]

创建项目时，请使用“npm create vite@4.4.0”或“npm create vite@4.0.0”而不是“npm create vite@latest”以确保其正常工作。 我有同样的问题并尝试了这个命令但它不起作用

确实有用，确实是vite的版本问题

[Jrubzjeknf]

This is beyond silly. Why is a package-lock.json generation checking the currently installed node_modules? It should be always be consistent.

This even happens when running npm i --package-lock-only (after deleting the package-lock.json), and the docs even say that it doesn't check installed node_modules. The command is not abiding its own docs.

This is a major bug that needs resolution. Any package with platform specific optional dependencies will break the package-lock.json if it ever needs to be regenerated on a developer's machine.

[SalvatorePreviti]

please :(

[vinkky]

I'm getting Cannot find module @rollup/rollup-linux-arm64-musl, but it's already added to optional dependencies... :/

[schjetne]

The way I "solved" this in the end was adding the platform specific packages I needed to devDependencies, instead of making them optional. It's not a cure, just a bandaid, but it's what I had to do to get this built without errors. Obviously this will break the moment we try to deploy to a different pipeline or we change the architecture we build on, but for now this works.

Very annoying bug.

[fajar-apri-alaska]

+1 getting this error also after updating rollup package.

[VittorioAltagracia]

'+' 1 got this issue setting up testing stage in our pipeline for a new projet. Before reading through the entire thread here, I apparently solved it, or maybe it is a bandaid. Workaround: Ran npm update and it installed what I was missing. And what a miracle it worked. Package I was missing: @rollup/rollup-linux-x64-gnu

I can't believe this nasty bug hasn't been fixed yet.
I was also thinking that maybe the very latest version of npm 10.2.5 wouldn't have this issue, but haven't got to updating it yet. My npm version is 10.2.3

[yuschick]

Running npm update and generating a new package-lock file that way resolved this issue for me.

[amandaguthrie]

+1. Had the same issue building via GitHub actions on Ubuntu. Missing package: @rollup/rollup-linux-x64-gnu. Temporarily removed package-lock.json from the repository so I didn't have to tweak the action to account for this. Thanks to everyone in thread for the helpful notes.

[ionutVizitiu]

Running npm update worked for us as well

[danyalutsevich]

for thouse who bumped into this problem with vite project this might be possible solution

https://medium.com/@fael-atom/struggling-with-vite-and-mui-42f3f5e0658d

[mariodujic]

Running npm update and generating a new package-lock file that way resolved this issue for me.

This worked for me.

[4skl]

I was having this issue (Cannot find module @rollup/rollup-linux-x64-musl) cause using an node alpine devcontainer on windows and did the npm create vue@latest... Solved it by re-building my devcontainer with the top command, running npm i inside, then going back to my initial command and re-building my devcontainer. TLDR : If you create a vite project on windows and want to open it on a devcontainer using npm ci, you'll need to run at least 1 time npm i on your devcontainer using top as a waiter for example to install requiered node packages using exec for your vite project on the devcontainer ✅; this will install the required packages for the unix based devcontainer

[roysharon]

Still happening for me, with vite and rollup. I'm using Mac M1, and the build always fails because it requires @rollup/rollup-darwin-x64 instead of @rollup/rollup-darwin-arm64. Tried deleting node_modules and package-lock.json, doing npm update and npm i. No go.

Debugging the requiring file I saw that node:process has arch=x64 instead of arm64. Reinstalled node for Mac with arm. Again deleted node_modules and package-lock.json and did npm i. Still no go.

Workaround: in the dev script in package.json I changed it from vite dev to arch -arm64 vite dev and now it works fine. Writing here just in case some gets the same problem in the future.

[cristidraghici]

npm i -O @rollup/rollup-linux-x64-gnu did the trick for my project. This added the following to the package.json:

{
  ...
  "optionalDependencies": {
    "@rollup/rollup-linux-x64-gnu": "^4.9.6"
  }
}

[perjo927]

npm i -O @rollup/rollup-linux-x64-gnu did the trick for my project. This added the following to the package.json:

{
  ...
  "optionalDependencies": {
    "@rollup/rollup-linux-x64-gnu": "^4.9.6"
  }
}

Thanks, this worked for me!

[Oguzhanst]

Some libraries working with the python.

Heres exact solution,

1. Try to download python
2. Open your file directory's on terminal
3. Delete your modules manually or write rmdir /s /q node_modules inside of terminal,
4. then you can just type npm install
5. If its doesnt work you need the do delete package.json and try to put with npm init -y
6. Try again.

[ericblade]

fwiw, it appears that the mere presence of optional dependencies seems to trigger this -- or at least, it triggers an error that tells me to come here for more details.

I'm trying to figure out a way to prevent my Docker build from installing cypress, as the docker image does not need to perform testing, so I threw cypress in optionalDependencies, but now it's complaining

 > [12/14] RUN npm run build:
0.547
0.547 > public@0.0.0 build
0.547 > tsc && vite build
0.547
3.514 /usr/src/app/public/node_modules/rollup/dist/native.js:87
3.514           throw new Error(
3.514                 ^
3.514
3.514 Error: Cannot find module @rollup/rollup-linux-x64-musl. npm has a bug related to optional dependencies (https://github.com/npm/cli/issues/4828). Please try `npm i` again after removing both package-lock.json and node_modules directory.

update: using npm install --omit=optional appears to be the direct cause of this -- it completely ignores the platform specific packages for rollup, both on linux and windows. There doesn't need to be anything IN my package's optionalDependencies.

Ultimately, it seems that omit=optional should probably only affect this package's optional dependencies, not all packages optionals.

[jweaver85]

npm i -O @rollup/rollup-linux-x64-gnu did the trick for my project. This added the following to the package.json:

{
  ...
  "optionalDependencies": {
    "@rollup/rollup-linux-x64-gnu": "^4.9.6"
  }
}

This works as a workaround on github action runners (ubuntu-latest). Thanks for sharing 🎸

[jdugas]

I ran into this issue with NX where the developers build the package lock on a windows machine, but the Jenkins build occurs on a linux server. The workaround was roughly the same, but wanted to share in case others run into the same issue. I added the optional dependencies for nx to support linux and windows.

"optionalDependencies": { "@nx/nx-linux-x64-gnu": "18.0.1", "@nx/nx-win32-x64-msvc": "18.0.1" }

In the windows machine I ran npm install to update the package-lock.json. I was then able to continue using npm ci on the jenkins server.

[rasimoner]

npm i -O @rollup/rollup-linux-x64-gnu did the trick for my project. This added the following to the package.json:

{
  ...
  "optionalDependencies": {
    "@rollup/rollup-linux-x64-gnu": "^4.9.6"
  }
}

thank you so much, it works for me :)

[samuk10]

both created with same command, tried to change npm version too. This ocurred on kubuntu 23.10 and 22.04 $ ng new angular-portfolio

this not work:

ls -ltra /home/samuel/external/Projetos/Angular/angular-portfolio
drwxrwxrwx  28 samuel samuel   4096 fev 14 00:46 ..
drwxrwxr-x   2 samuel samuel   4096 fev 14 00:46 .vscode
-rw-rw-r--   1 samuel samuel    273 fev 14 00:46 tsconfig.spec.json
-rw-rw-r--   1 samuel samuel    903 fev 14 00:46 tsconfig.json
-rw-rw-r--   1 samuel samuel    263 fev 14 00:46 tsconfig.app.json
drwxrwxr-x   4 samuel samuel   4096 fev 14 00:46 src
-rw-rw-r--   1 samuel samuel   1070 fev 14 00:46 README.md
-rw-rw-r--   1 samuel samuel   1048 fev 14 00:46 package.json
-rw-rw-r--   1 samuel samuel    548 fev 14 00:46 .gitignore
-rw-rw-r--   1 samuel samuel    274 fev 14 00:46 .editorconfig
-rw-rw-r--   1 samuel samuel   2821 fev 14 00:46 angular.json
-rw-rw-r--   1 samuel samuel 444022 fev 14 00:48 package-lock.json
drwxrwxr-x 557 samuel samuel  20480 fev 14 00:48 node_modules
drwxrwxr-x   6 samuel samuel   4096 fev 14 00:48 .
drwxrwxr-x   8 samuel samuel   4096 fev 14 00:48 .git

this work:

ls -ltra /home/samuel/angular-proj/angular-portfolio
drwxrwxr-x   6 samuel samuel   4096 fev 14 00:40 ..
drwxrwxr-x   2 samuel samuel   4096 fev 14 00:40 .vscode
-rw-rw-r--   1 samuel samuel    273 fev 14 00:40 tsconfig.spec.json
-rw-rw-r--   1 samuel samuel    903 fev 14 00:40 tsconfig.json
-rw-rw-r--   1 samuel samuel    263 fev 14 00:40 tsconfig.app.json
drwxrwxr-x   4 samuel samuel   4096 fev 14 00:40 src
-rw-rw-r--   1 samuel samuel   1070 fev 14 00:40 README.md
-rw-rw-r--   1 samuel samuel   1048 fev 14 00:40 package.json
-rw-rw-r--   1 samuel samuel    548 fev 14 00:40 .gitignore
-rw-rw-r--   1 samuel samuel    274 fev 14 00:40 .editorconfig
-rw-rw-r--   1 samuel samuel   2821 fev 14 00:40 angular.json
-rw-rw-r--   1 samuel samuel 444022 fev 14 00:41 package-lock.json
drwxrwxr-x 557 samuel samuel  20480 fev 14 00:41 node_modules
drwxrwxr-x   8 samuel samuel   4096 fev 14 00:42 .git
drwxrwxr-x   3 samuel samuel   4096 fev 14 00:42 .angular
drwxrwxr-x   7 samuel samuel   4096 fev 14 00:42 .

also copied working folder to not working and not worked

[Harshwardhan18]

I have faced a very similar kind of issue (@rollup/darwin) while trying to start vest-react server. I was using node 20.9.

I basically deleted the package-lock file, node_modules and changed the node version to v16 and then installed (npm i) all the packages once again. SOLVED!!!

[kassiansun]

We need a more elegant way to handle this, yarn solved this issue by locking all optional dependencies in yarn.lock. If the work-around is to delete package-lock.json, then there's really no point to introduce this locking mechanism.

[lovell]

For those who hadn't seen, this issue is now the number one most upvoted open bug in npm. Thank you to everyone that has added their 👍 to the original post to help with visibility and therefore likelihood of a fix.

[sverrejb]

Currently facing this problem on a Sveltekit project building on Netlify. Adding @rollup/rollup-linux-x64-gnu as an optional dependency did not fix the problem.

[Andrew-Pulley]

I'm also facing this problem on a SvelteKit project both building and/or running dev in Docker. Added @rollup/rollup-linux-x64-gnu as an optional dependency (as well as an explicit dependency and dev dependency), modified my Dockerfile to remove node_modules and package-lock.json before installing, building or running dev.

[sverrejb]

Deleting my lock file and local node_modules, then running npm i resulted in a new lockfile (why?) that worked when building in CI. The fragility of the web eco system never ceases to amaze me.

[ljharb]

@sverrejb this has nothing to do with fragility, and what do you mean why - npm install always makes a lockfile by default.

[sverrejb]

Yes, but why is it different after deleting my node_modules? Shouldn't my lockfile dictate which dependencies are downloaded, not the other way around? I'll admit I am not very knowledgeable about how npm works under the hood.

[ljharb]

@sverrejb if you were having trouble with your lockfile, and deleting it and recreating it fixed it, then clearly something was incorrect in your previous lockfile. That certainly could have been caused by an npm bug (or many, over time), or by having an invalid node_modules when you first created the lockfile, or by someone manually editing the lockfile, or all of those.

[psnavega]

Works when I delete node_modules generated first and then running again npm install inside the container. I do no why but works. Using container there's no definitive solution( I guess).

[zappys]

Deleting your lock file is bad practice. Note that when deleting the package-lock file, the versions of packages you are using in production might be slightly different than the ones you are using when developing, and there ways to introduce bugs you are not testing for.

[cklim24]

Deleting your lock file is bad practice. Note that when deleting the package-lock file, the versions of packages you are using in production might be slightly different than the ones you are using when developing, and there ways to introduce bugs you are not testing for.

[InstantlyMoist]

Same issue here, currently trying to build a barebones Nitro server in dev mode.

node                          | 
node                          |  ERROR  Cannot find module @rollup/rollup-linux-x64-musl. npm has a bug related to optional dependencies (https://github.com/npm/cli/issues/4828). Please try npm i again after removing both package-lock.json and node_modules directory.
node                          |
node                          |   at requireWithFriendlyError (node_modules/rollup/dist/native.js:87:9)
node                          |   at Object.<anonymous> (node_modules/rollup/dist/native.js:96:76)
node                          |   at Module._compile (node:internal/modules/cjs/loader:1376:14)
node                          |   at Module._extensions..js (node:internal/modules/cjs/loader:1435:10)
node                          |   at Module.load (node:internal/modules/cjs/loader:1207:32)
node                          |   at Module._load (node:internal/modules/cjs/loader:1023:12)
node                          |   at cjsLoader (node:internal/modules/esm/translators:356:17)
node                          |   at ModuleWrap.<anonymous> (node:internal/modules/esm/translators:305:7)
node                          |   at ModuleJob.run (node:internal/modules/esm/module_job:218:25)
node                          |   at async ModuleLoader.import (node:internal/modules/esm/loader:329:24)
node                          |
node                          |
node                          |
node                          |  ERROR  Cannot find module @rollup/rollup-linux-x64-musl. npm has a bug related to optional dependencies (https://github.com/npm/cli/issues/4828). Please try npm i again after removing both package-lock.json and node_modules directory.

[eudoroolivares2016]

I found that this behavior at least for the sharp package to work as expected in different platform environments (including aws lambda) after I updated the npm version > 10 if anyone is here for issues around that package

[sbaechler]

The problem seems to be that running npm install prunes the lockfile. Optional dependencies that are not installed for the current architecture are removed.

Leaving those in the lockfile would fix this issue.

[InstantlyMoist]

i mean even installing them manually afterwards doesn't work. running within a docker container

[steven-muniz]

npm i -O @rollup/rollup-linux-x64-gnu did the trick for my project. This added the following to the package.json:

{
  ...
  "optionalDependencies": {
    "@rollup/rollup-linux-x64-gnu": "^4.9.6"
  }
}

thank you so much, it works for me :)

I also did this, resolved my problem! Thanks so much!

[jbouhier]

Just got that bug. Issue is still present

[JanPodmajersky]

We struggled with this for few hours but this https://github.com/rollup/rollup/issues/5194#issuecomment-1824156669 fixed it on our end, we use yarn, so we added this to package.json

"resolutions": {
  "rollup": "npm:@rollup/wasm-node"
}

However the rationale is unknown 🤔 , but it works 🎉 , follow this for more details: https://github.com/rollup/rollup/discussions/5378

[striderwhite]

Even after removing node_modules and package-lock.json then re-running npm i I still get the same error. Specifically for rollup (I am on mac M2)

Error: Cannot find module @rollup/rollup-darwin-x64. npm has a bug related to optional dependencies (https://github.com/npm/cli/issues/4828). Please try `npm i` again after removing both package-lock.json and node_modules directory.

Brutal!

[MoustafaMohsen]

same for me, I still have an issue with rollup , I'm on mac m1

[ludndev]

Temporary Fix for Rollup Issue Without Sacrificing Lock Files

You can try installing Rollup using the following command:

npm i g @rollup/rollup-linux-x64-gnu

This workaround has worked for me and might provide a temporary solution until a more permanent fix is implemented.

Please note that while this workaround may resolve the immediate issue, it's important not to sacrifice the integrity of your lock files. Be sure to keep track of any changes made and revert them once a proper solution is available.

If anyone else has encountered this issue or has alternative solutions, feel free to share your experiences and suggestions here.

[atrofimov-sc]

I'm constantly struggling with biome, esbuild and turbo all of which have optional dependencies on platform specific binaries. I have an M1 mac and an Intel Mac, and switching the same codebase between the two constantly gets me into trouble where the I get foo-x64 or foo-arm aren't installed.

[schjetne]

This has been a known bug for ages. It doesn't seem like something that should be this hard to fix. Is there an effort to fix it or is it just going to be people posting in this issue to confirm they too have this bug?

[ludndev]

Temporary Fix for Rollup Issue Without Sacrificing Lock Files

You can try installing Rollup using the following command:

npm i g @rollup/rollup-linux-x64-gnu

This workaround has worked for me and might provide a temporary solution until a more permanent fix is implemented.

Please note that while this workaround may resolve the immediate issue, it's important not to sacrifice the integrity of your lock files. Be sure to keep track of any changes made and revert them once a proper solution is available.

If anyone else has encountered this issue or has alternative solutions, feel free to share your experiences and suggestions here.

I found another solution on the internet.

    "optionalDependencies": {
        "@rollup/rollup-linux-x64-gnu": "4.9.5"
    }