[BUG] EMFILE error in environment with low file descriptors limit

rschick commented 8 months ago

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior

EMFILE error may be thrown during npm install, depending on the allowed file descriptors and the state of the cache before the install. An example error:

Error: EMFILE: too many open files, open '/Users/<user>/.npm/_cacache/index-v5/64/ee/136420e5adf6592619d25b411c7849220f30364ed8ba96dea19887a5d1f2'

Expected Behavior

npm install should succeed.

Steps To Reproduce

Create a project directory and add package.json with the following:
```
{
"name": "nuxt-app",
"devDependencies": {
"nuxt": "^3.7.0"
}
}
```

(The error happens reliably with nuxt, but it's not related to nuxt, you can use other packages and get the same result.)

Set a low file descriptor limit using ulimit -Hn 128 (It's possible to get the error with a higher ulimit, but using a low value helps to reliably reproduce the error)
Delete the global cacache, eg: rm -rf /<home-dir>/.npm/_cacache/
Delete the project's node_modules folder: rm -rf node_modules
Run npm install
EMFILE error is thrown

(Reset ulimit using ulimit -Hn unlimited)

Environment

npm: 10.4.0
Node: 20.11.0
OS: MacOS 14.1.2
platform: Macbook Pro 2019 (Intel)

rschick commented 8 months ago

I'm not familiar with the npm codebase, but I was able to fix this issue by adding retries here: https://github.com/npm/cacache/blob/main/lib/put.js#L69. I'd be happy to submit a PR if you think this is a viable approach.

tamoreton commented 7 months ago

I'm not familiar with the npm codebase, but I was able to fix this issue by adding retries here: https://github.com/npm/cacache/blob/main/lib/put.js#L69. I'd be happy to submit a PR if you think this is a viable approach.

What about updating the library to use graceful-fs instead of fs? I think that essentially does the same thing – adds a backoff to prevent EMFILE errors.

I don't know if there is a performance reason cacache is using the lower-level fs API – it does have ~29 million weekly downloads and needs to be "really fast".

k2s commented 7 months ago

I fixed this issue by using fs-extra package instead of fs/promises. It uses graceful-fs internally and provides patched functions with support for promises. This fixes EMFILE, also EAGAIN and potential problems with antivirus software.

There is also example of custom implementation using only p-limit package https://github.com/prismicio/slice-machine/pull/1192.

npm / cacache