yarn upgrade gives ENFILE: file table overflow, open

[neiled]

Do you want to request a feature or report a bug?

Bug

What is the current behavior?

If the current behavior is a bug, please provide the steps to reproduce.

No other node instances running.

Installed yarn

brew install yarn

Ran

Yarn

in existing project - ALL OK Running

yarn upgrade

• Gave me:

CError: ENFILE: file table overflow, open '/usr/local/Cellar/yarn/0.18.1/libexec/lib/node_modules/yarn/lib/errors.js' at Object.fs.openSync (fs.js:558:18) at Object.fs.readFileSync (fs.js:468:33) at Object.Module._extensions..js (module.js:579:20) at Module.load (module.js:488:32) at tryModuleLoad (module.js:447:12) at Function.Module._load (module.js:439:3) at Module.require (module.js:498:17) at require (internal/module.js:20:19) at _load_errors (/usr/local/Cellar/yarn/0.18.1/libexec/lib/node_modules/yarn/lib/cli/index.js:42:20) at config.init.then.catch.err (/usr/local/Cellar/yarn/0.18.1/libexec/lib/node_modules/yarn/lib/cli/index.js:417:34)

What is the expected behavior?

Please mention your node.js, yarn and operating system version.

yarn --version 0.18.1

MacOS: 10.12.2 (16C67)

Node: 7.4.0

Package file this failed on:

{
  "name": "xxx",
  "version": "1.0.0",
  "description": "A Vue.js project",
  "author": "neiled",
  "private": true,
  "scripts": {
    "dev": "node build/dev-server.js",
    "build": "node build/build.js",
    "unit": "karma start test/unit/karma.conf.js --single-run",
    "e2e": "node test/e2e/runner.js",
    "test": "npm run unit && npm run e2e",
    "lint": "eslint --ext .js,.vue src test/unit/specs test/e2e/specs",
    "dev-server": "nodemon -r dotenv/config -r babel-register ./server/server.js"
  },
  "dependencies": {
    "async-busboy": "^0.3.3",
    "aws-sdk": "^2.7.21",
    "axios": "^0.15.3",
    "bcrypt": "^1.0.1",
    "dotenv": "^2.0.0",
    "knex": "^0.12.6",
    "koa": "^2.0.0-alpha.7",
    "koa-bodyparser": "^3.2.0",
    "koa-convert": "^1.2.0",
    "koa-cors": "0.0.16",
    "koa-generic-session": "^1.11.4",
    "koa-jwt": "^1.2.0",
    "koa-logger": "^2.0.0",
    "koa-passport": "^2.2.2",
    "koa-router": "^7.0.1",
    "lodash": "^4.17.4",
    "objection": "^0.6.2",
    "passport": "^0.3.2",
    "passport-local": "^1.0.0",
    "pg": "^6.1.1",
    "uuid": "^3.0.1",
    "vee-validate": "^2.0.0-beta.18",
    "vue": "^2.1.4",
    "vue-router": "^2.1.1"
  },
  "devDependencies": {
    "autoprefixer": "^6.5.3",
    "babel-cli": "^6.18.0",
    "babel-core": "^6.20.0",
    "babel-eslint": "^7.1.1",
    "babel-loader": "^6.2.9",
    "babel-plugin-transform-runtime": "^6.15.0",
    "babel-preset-es2015": "^6.18.0",
    "babel-preset-es2017": "^6.16.0",
    "babel-preset-stage-2": "^6.18.0",
    "babel-register": "^6.18.0",
    "chai": "^3.5.0",
    "chalk": "^1.1.3",
    "chromedriver": "^2.25.3",
    "connect-history-api-fallback": "^1.3.0",
    "cross-spawn": "^5.0.1",
    "css-loader": "^0.26.1",
    "eslint": "^3.12.0",
    "eslint-config-standard": "^6.2.1",
    "eslint-friendly-formatter": "^2.0.6",
    "eslint-loader": "^1.6.1",
    "eslint-plugin-html": "^1.7.0",
    "eslint-plugin-promise": "^3.4.0",
    "eslint-plugin-standard": "^2.0.1",
    "eventsource-polyfill": "^0.9.6",
    "express": "^4.14.0",
    "extract-text-webpack-plugin": "^1.0.1",
    "file-loader": "^0.9.0",
    "function-bind": "^1.1.0",
    "html-webpack-plugin": "^2.24.1",
    "http-proxy-middleware": "^0.17.3",
    "inject-loader": "^3.0.0-beta2",
    "isparta-loader": "^2.0.0",
    "json-loader": "^0.5.4",
    "karma": "^1.3.0",
    "karma-coverage": "^1.1.1",
    "karma-mocha": "^1.3.0",
    "karma-phantomjs-launcher": "^1.0.2",
    "karma-sinon-chai": "^1.2.4",
    "karma-sourcemap-loader": "^0.3.7",
    "karma-spec-reporter": "0.0.26",
    "karma-webpack": "^1.8.0",
    "lolex": "^1.5.2",
    "mocha": "^3.2.0",
    "nightwatch": "^0.9.9",
    "opn": "^4.0.2",
    "ora": "^0.4.0",
    "phantomjs-prebuilt": "^2.1.14",
    "selenium-server": "3.0.1",
    "semver": "^5.3.0",
    "shelljs": "^0.7.5",
    "sinon": "^1.17.6",
    "sinon-chai": "^2.8.0",
    "url-loader": "^0.5.7",
    "vue-loader": "^10.0.2",
    "vue-style-loader": "^1.0.0",
    "vue-template-compiler": "^2.1.4",
    "webpack": "^1.14.0",
    "webpack-dev-middleware": "^1.8.4",
    "webpack-hot-middleware": "^2.13.2",
    "webpack-merge": "^2.0.0"
  },
  "engines": {
    "node": ">= 4.0.0",
    "npm": ">= 3.0.0"
  }
}

[benjajaja]

yarn

gave me

yarn install v0.17.8
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
error ENFILE: file table overflow, open '/Users/bgrosse/Library/Caches/Yarn/npm-source-map-0.2.0-dab73fbcfc2ba819b4de03bd6f6eaa48164b3f9d/.yarn-metadata.json'
error ENFILE: file table overflow, open '/Users/bgrosse/Library/Caches/Yarn/npm-fsevents-1.0.14-558e8cc38643d8ef40fe45158486d0d25758eee4/.yarn-metadata.json'
error ENFILE: file table overflow, open '/Users/bgrosse/Library/Caches/Yarn/npm-bcrypt-pbkdf-1.0.0-3ca76b85241c7170bf7d9703e7b9aa74630040d4/.yarn-metadata.json'
error ENFILE: file table overflow, open '/Users/bgrosse/Library/Caches/Yarn/npm-ecc-jsbn-0.1.1-0fc73a9ed5f0d53c38193398523ef7e543777505/.yarn-metadata.json'
error ENFILE: file table overflow, open '/Users/bgrosse/Library/Caches/Yarn/npm-jodid25519-1.0.2-06d4912255093419477d425633606e0e90782967/.yarn-metadata.json'
error ENFILE: file table overflow, open '/Users/bgrosse/Library/Caches/Yarn/npm-jsbn-0.1.0-650987da0dd74f4ebf5a11377a2aa2d273e97dfd/.yarn-metadata.json'
error ENFILE: file table overflow, open '/Users/bgrosse/Library/Caches/Yarn/npm-tweetnacl-0.14.3-3da382f670f25ded78d7b3d1792119bca0b7132d/.yarn-metadata.json'
███████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 178/1242^

and it just hangs there forever.

Running yarn cache clean fixed the issue for me.

[joeporpeglia]

I'm frequently running into this issue working on a project with a ton of npm dependencies. I see that the onUnexpectedError handler in the cli module attempts to open more files. Could that be problematic if the unexpected error is a file table overflow?

[BYK]

I think this is an OS X issue: https://github.com/reactioncommerce/reaction-cli/issues/11#issuecomment-246740634

LMK if increasing the limit fixes your problem.

[joeporpeglia]

Increasing the limit definitely fixes the file table overflow issue, but it seems like yarn could still avoid hanging when this issue comes up. I modified the onUnexpectedError method to return before opening those additional files and it looks like it fixes the hanging problem and spits out the correct "file table overflow" message before exiting.

[BYK]

@joeporpeglia - ah, that's great! Can you submit a PR?

[joeporpeglia]

I'd be happy to! Should we try to fail gracefully here or just spit out the file table overflow message? It doesn't seem like we can write out to yarn-error.log, which is the typical flow for onUnexpectedError.

[BYK]

@joeporpeglia oh yeah, that's tricky. I'd say put the code writing the error log in a try-catch block and report the error from that on the console along with the original one. Special casing for tabl overflow makes sense but there may be other errors like ENOSPC, EROFS etc. that would prevent us from writing the file.

[joeporpeglia]

@BYK I must have had my machine in a very specific state the other day because I'm having trouble reproducing the issue now 😞. If you think it's still worth wrapping those IO operations in onUnexpectedError in a try/catch I can PR the changes: https://github.com/yarnpkg/yarn/compare/master...joeporpeglia:handle-file-table-overflow.

[chambi-GI]

Try this sudo launchctl limit maxfiles 16384 16384 && ulimit -n 16384