EMFILE, too many open files

valotas commented 9 years ago

I am not sure if it has to do with karma-jspm itself or karma but I'm getting the following

[23:36:35] Using gulpfile ~/Projects/xxx/gulpfile.js
[23:36:35] Starting 'test'...
[23:36:36] 'test' errored after 1.18 s
[23:36:36] Error: EMFILE, too many open files '/xxx/src/test/webapp'
    at Error (native)
    at Object.fs.readdirSync (fs.js:761:18)
    at GlobSync._readdir (/xxx/node_modules/karma/node_modules/glob/sync.js:275:41)
    at GlobSync._processReaddir (/xxx/node_modules/karma/node_modules/glob/sync.js:135:22)
    at GlobSync._process (/xxx/node_modules/karma/node_modules/glob/sync.js:130:10)
    at new GlobSync (/xxx/node_modules/karma/node_modules/glob/sync.js:46:10)
    at new Glob (/xxx/node_modules/karma/node_modules/glob/glob.js:111:12)
    at /xxx/node_modules/karma/lib/file-list.js:163:14
    at Array.map (native)
    at List._refresh (/xxx/node_modules/karma/lib/file-list.js:155:37)
    at List.refresh (/xxx/node_modules/karma/lib/file-list.js:254:27)
    at Server._start (/xxx/node_modules/karma/lib/server.js:173:12)
    at invoke (/xxx/node_modules/karma/node_modules/di/lib/injector.js:75:15)
    at Server.start (/xxx/node_modules/karma/lib/server.js:97:18)
    at Gulp.<anonymous> (/xxx/gulpfile.js:25:10)
    at module.exports (/xxx/node_modules/gulp/node_modules/orchestrator/lib/runTask.js:34:7)

My assumption is that karma can not handle the size of jspm_packages

lgvo commented 8 years ago

Actually I believe is not a bug at all. You probably don't have enough limit to open files on your user. (Maybe karma could use something like https://github.com/isaacs/node-graceful-fs to avoid that)

To test, you can change it running the command: ulimit -u 10240

You can add it at your .profile or config in the /etc/security/limits.conf (at least in linux):

For all users, add these lines:

*    soft    nofile        10240
*    hard    nofile        10240

Only for your user (valotas), add these lines:

valotas    soft    nofile        10240
valotas    hard    nofile        10240

Hope it help.

maxwellpeterson-wf commented 8 years ago

Yes, this is an issue with the size of your project/number of dependencies it is using and your OS settings as @lgvo has explained. StackOverflow backs this up. Thanks for the in-depth explanation!

togakangaroo commented 8 years ago

I am having the exact same thing happen on windows 8.1. My karma was working just fine until I upgraded to node 4.1.1 (previously was node 0.latest). Now its telling me too many open files.

lares83 commented 8 years ago

I'm having the same issue too, on Windows 10 and node 4.1.1 and I don't know how to fix it?

sateffen commented 8 years ago

We're having a similar issue, but we can't explain it:

Our dev-team uses only windows 7 laptops, and our project is somewhat big (around 1500-2000 js files that get loaded). That should be no problem, and wasn't up till now.

One of my co-workers got a new laptop to replace his old, weak one, with a new, very strong one. The new and pretty strong one has windows 7 64bit system as well, but much stronger hardware, but gets this EMFILE error.

All other, even the old dev machines don't get this error. At our dev machines phantomjs keeps crushing, but that's another "problem". On our build servers (pretty weak Linux machines, but they a build process + testing may take some more minutes, don't care) everything works fine: karma, phantomjs, everything.

So there is only ONE computer, the most powerful, that has the "EMFILE" error, but all other, with the same windows 7 64bit system, don't have this problem.

We use nodejs version 4.1.1 on all tested systems (now), but the different behaviour stays the same.

togakangaroo commented 8 years ago

@lgvo what do you think? Seems like this is a common issue on windows now, so it can't really be tied to what you cited.

lgvo commented 8 years ago

@togakangaroo I do think it is just the same kind of issue. But I don't use Windows, maybe you can try look if others libraries have the same problem when you upgrade to node 4.x.

I also didn't upgrade my projects to node 4, some backend libs aren't working properly when I tried. But I'm using the lastest iojs 3.x without any problems.

You can try use iojs too, it have almost the same features that the node 4. I was using node 0.x before and migrate to iojs 3.x to easily upgrade to node 4.x when I find it more stable.

togakangaroo commented 8 years ago

@lgvo hmm, I have noticed zero issues with any other library. I need to upgrade since jspm bundle needs a newer version of node to properly bundle es6 features for production (ie I hate the idea of transforming generators, arrow functions, and const when the latest browsers already support them). Io.js is a non-starter since, not only is it a step backwards, but it would affect all other developers on my project, and all other projects using the same build server.

It is strange though, I've got another small project that has very few dependencies where this is still happening. Is there anything in karma-jspm that you know of that might cause this?

Seems like I might have to spend some time setting up a small issue repro. We can discuss then.

lgvo commented 8 years ago

@togakangaroo if it' s only karma-jspm it must be related to the glob dependency I think, try to use karma-jspm locally (using npm link) and update the glob library and see if it helps. About transpiling ES6 features you can watch the https://github.com/getify/es-feature-tests We are talking about that on https://github.com/getify/es-feature-tests/issues/9. But if you have to support more browser I don't think you can get a production solution on that maybe use the browser support only for your workflow. For backend I create some scripts that work, maybe if I get some time I make a pull request on that.

sateffen commented 8 years ago

Just in case: We don't use karma-jspm, we use tried plain karma and grunt-karma as runners. We have some setup with requirejs, but currently I'm not at work, so I can't tell the exact setup, but it's pretty much like here: https://karma-runner.github.io/0.8/plus/RequireJS.html

Gu1 commented 8 years ago

Hey. I have the same problem here. EMFILE errors with karma/karma-jspm since I switched to nodejs 4.x

togakangaroo commented 8 years ago

Just to update everyone... I replicated this in a simple project with just karma, jspm, karma-jspm, and a single test. However, when I upgraded everything to latest, dumped node_modules and re-installed everything started working. Still investigating whether I can get my actual project back to running.

lgvo commented 8 years ago

Looks like that since 4.0.5 glob has support to use graceful-fs Issue.

zckrs commented 8 years ago

Same issue :-/

wzhao-c commented 8 years ago

I have same issue since upgraded to node v5

xtreemrage commented 8 years ago

Same issue, but that can't be the solution to edit a system file... When will this be fixed?

m-j commented 8 years ago

I get the same issue on Windows 8.1. I believe karama tries to "eat" all files at once and it hits system limit, therefore I believe it's an actual bug.

Has aynone tried global patching it? https://github.com/isaacs/node-graceful-fs

Some more info. I believe problem may be due karma serving node_modules. I ran few commands through cygwin:

$ ls -R . | wc -l
  26428

$ ls -R node_modules | wc -l
  19589

$ ls -R client | wc -l
   3386

(Client is where my code and jspm_packages sit)

valotas commented 8 years ago

I also believe that karma tries to "eat" all the files at once. It can be a general karma problem as angular has the same problem

rdehuyss commented 8 years ago

+1

I'm trying to get the tests in https://github.com/AngularClass/NG6-starter/tree/jspm to work.

C:\Users\bkdevconstructiv\tmp\NG6-starter>systeminfo
...
OS Name:                   Microsoft Windows 8.1 Enterprise
OS Version:                6.3.9600 N/A Build 9600

C:\Users\bkdevconstructiv\tmp\NG6-starter>node --version
v5.4.1

C:\Users\bkdevconstructiv\tmp\NG6-starter>karma --version
Karma version: 0.12.37

My package.json:

{
  "name": "angular-jspm-starter",
  "version": "0.0.1",
  "description": "starter",
  "dependencies": {},
  "scripts": {
    "test": "karma start",
    "postinstall": "jspm install"
  },
  "keywords": [
    "angular",
    "jspm",
    "es6"
  ],
  "author": "AngularClass",
  "license": "ISC",
  "jspm": {
    "directories": {
      "baseURL": "client"
    },
    "configFile": "jspm.config.js",
    "dependencies": {
      "angular": "github:angular/bower-angular@^1.4.0",
      "angular-mocks": "npm:angular-mocks@^1.4.0",
      "angular-ui-router": "github:angular-ui/ui-router@^0.2.15",
      "capaj/systemjs-hot-reloader": "github:capaj/systemjs-hot-reloader@^0.5.0",
      "css": "github:systemjs/plugin-css@^0.1.11",
      "normalize.css": "github:necolas/normalize.css@^3.0.3",
      "text": "github:systemjs/plugin-text@^0.0.2"
    },
    "devDependencies": {
      "babel": "npm:babel-core@^5.8.24",
      "babel-runtime": "npm:babel-runtime@^5.8.24",
      "core-js": "npm:core-js@^1.1.4"
    }
  },
  "devDependencies": {
    "browser-sync": "^2.7.6",
    "chai": "^3.5.0",
    "chokidar-socket-emitter": "^0.3.0",
    "gulp": "^3.9.0",
    "gulp-html-replace": "^1.5.0",
    "gulp-ng-annotate": "^1.0.0",
    "gulp-rename": "^1.2.2",
    "gulp-template": "^3.0.0",
    "gulp-uglify": "^1.2.0",
    "jspm": "^0.16.25",
    "karma": "^0.12.36",
    "karma-chai": "^0.1.0",
    "karma-chrome-launcher": "^0.1.12",
    "karma-jspm": "^2.0.2",
    "karma-mocha": "^0.1.10",
    "karma-mocha-reporter": "^1.0.2",
    "mocha": "^2.4.4",
    "rimraf": "^2.4.3",
    "yargs": "^3.10.0"
  }
}

The error I have:

ERROR [karma]: { [Error: EMFILE: too many open files, open 'C:\Users\bkdevconstructiv\tmp\NG6-starter\client\jspm_packages\npm\util@0.10.3.js']
  errno: -4066,
  code: 'EMFILE',
  syscall: 'open',
  path: 'C:\\Users\\bkdevconstructiv\\tmp\\NG6-starter\\client\\jspm_packages\\npm\\util@0.10.3.js' }
Error: EMFILE: too many open files, open 'C:\Users\bkdevconstructiv\tmp\NG6-starter\client\jspm_packages\npm\util@0.10.3.js'
    at Error (native)

fabioelia commented 8 years ago

Take a look @ https://github.com/Workiva/karma-jspm/pull/115 JSPM noCache resolved the EMFILE issue

balazsmeszegeto commented 8 years ago

Updating karma (old version: 0.12.36, new version: 0.13.19) fixes the EMFILE issue, at least for me.

Reason, I think, because the newer version uses graceful-fs@4.1.3, instead of node-fs. graceful-fs uses a resource-pool, thus only allows to open a limited amount of files (file openings are being queued)

As a side note, I think that karma's preprocessor should act more clever. It should not send unlimited files at the same time to the preprocessor, but rather implement a smart solution. Ie. define a reasonable limit for parallel running.

AurelienRibon commented 8 years ago

Issue with jspm_packages fixed in pull request 126 https://github.com/Workiva/karma-jspm/pull/126

mribichich commented 8 years ago

Same thing here, I'm in Windows 10, with karma 0.12.x I had the EMFILE issue, and after updating to 0.13.19, it was solved

bjanderson commented 8 years ago

I'm having the same issue. I've read that karma-jspm@2.0.3 supposedly fixes this, or at least allows you to set noPackagesCache:true in karma.config, which doesn't work for me. I also see that the latest version is karma-jspm@2.1.0, but neither of these are available in npm. The latest valid install target is 2.0.2. How long does it usually take for npm to sync up with github?

I'm having the same problem with a project at work, so I created a project to try and figure it out. It's basically a bare-bones foundation project that started failing as soon as I jspm installed google's material-design-icons. *Note to future: that is a work in progress, so hopefully it will work when this issue gets fixed.

For the time being, I've had to stop using karma and just run mocha straight from npm (project). But I would much rather use karma if at all possible.

autoZhangHui commented 8 years ago

@balazsmeszegeto
Updating karma (old version: 0.12, new version: 0.13.22) and fixes the EMFILE issue. Thx for your advice!

nguillaumin commented 8 years ago

It's still happening for us with Karma 0.13.22 and karma-jspm 2.1.0 (observed on Mac & Linux)

eyalellenbogen commented 8 years ago

same here. windows 10. karma 0.13.22 running through gulp. Only when trying 5 karma instances simultaneously. 4 simultaneous instances pass tests fine.

ldunca commented 8 years ago

I have the same issue :( karma 0.13.22 and karma-jspm 2.1.0

0Ky commented 8 years ago

Windows Server 2012 R2 ~ Same issue. Someone please find a solution for this error.

events.js:154
      throw er; // Unhandled 'error' event
      ^
Error: EMFILE: too many open files, open 'C:\projectc1\fswrite.html'
    at Error (native)

Thanks.

mattfricker commented 8 years ago

I was having the same issue on Ubuntu 14.04. Tried the recommended methods above with no luck.

What ended up fixing the issue for me was deleting my jspm_packages folder and then running jspm install again. After this, my tests now run without the error.

svi3c commented 8 years ago

I've got the same issue. My stack trace:

[project-root]/node_modules/karma/node_modules/glob/sync.js:328
        throw er
        ^

Error: EMFILE: too many open files, scandir '[project-root]/src/app/todos'
    at Error (native)
    at Object.fs.readdirSync (fs.js:856:18)
    at GlobSync._readdir ([project-root]/node_modules/karma/node_modules/glob/sync.js:275:41)
    at GlobSync._processReaddir ([project-root]/node_modules/karma/node_modules/glob/sync.js:135:22)
    at GlobSync._process ([project-root]/node_modules/karma/node_modules/glob/sync.js:130:10)
    at new GlobSync ([project-root]/node_modules/karma/node_modules/glob/sync.js:46:10)
    at new Glob ([project-root]/node_modules/karma/node_modules/glob/glob.js:124:12)
    at [project-root]/node_modules/karma/lib/file-list.js:161:14
    at Array.map (native)
    at List._refresh ([project-root]/node_modules/karma/lib/file-list.js:153:37)
    at List.refresh ([project-root]/node_modules/karma/lib/file-list.js:252:27)
    at Server._start ([project-root]/node_modules/karma/lib/server.js:178:12)
    at invoke ([project-root]/node_modules/di/lib/injector.js:75:15)
    at Server.start ([project-root]/node_modules/karma/lib/server.js:102:18)
    at Object.<anonymous> (/home/sven/.IntelliJIdea2016.1/config/plugins/js-karma/js_reporter/karma-intellij/lib/intellijServer.js:12:13)
    at Module._compile (module.js:413:34)

What I found out: When I move this line to the bottom of the function and replace unshift() by push(), it works for me. So instead of having

files === [ '/home/sven/dev/projects/angular2-setup-poc/jspm_packages/system.src.js',
  '/home/sven/dev/projects/angular2-setup-poc/jspm_packages/system-polyfills.src.js',
  '/home/sven/dev/projects/angular2-setup-poc/node_modules/karma-jspm/src/adapter.js',
  '/home/sven/dev/projects/angular2-setup-poc/jspm.test.js',
  '/home/sven/dev/projects/angular2-setup-poc/jspm.config.js',
  '/home/sven/dev/projects/angular2-setup-poc/jspm_packages/**/*',
  '/home/sven/dev/projects/angular2-setup-poc/src/app/todos/*.spec.ts',
  '/home/sven/dev/projects/angular2-setup-poc/src/**/*.ts' ]

I get

files === [ '/home/sven/dev/projects/angular2-setup-poc/jspm_packages/system.src.js',
  '/home/sven/dev/projects/angular2-setup-poc/jspm_packages/system-polyfills.src.js',
  '/home/sven/dev/projects/angular2-setup-poc/node_modules/karma-jspm/src/adapter.js',
  '/home/sven/dev/projects/angular2-setup-poc/jspm.test.js',
  '/home/sven/dev/projects/angular2-setup-poc/jspm.config.js',
  '/home/sven/dev/projects/angular2-setup-poc/src/app/todos/*.spec.ts',
  '/home/sven/dev/projects/angular2-setup-poc/src/**/*.ts',
  '/home/sven/dev/projects/angular2-setup-poc/jspm_packages/**/*' ]

But I have no clue how this could provoke the error. Note: As you can see, no graceful-fs is used. glob is used and this uses node-fs. My relevant deps in the package.json:

{
  "jspm": "^0.17.0-beta.13",
  "karma": "^0.13.22",
  "karma-jspm": "2.1.0"
}

wuliupo commented 8 years ago

Thanks @svi3c ! After update the karma to new version, it is solved. Run below command under your project folder (or as global, -g)

npm install karma@0.13.22 npm install karma-jspm@2.1.0 npm install jspm@0.17.0-beta.13

cAhmad commented 8 years ago

Thanks @svi3c you fixed the too many open files for me!

karma@0.13.22 karma-jspm@2.1.1 jspm@0.16.2

maxwellpeterson-wf commented 8 years ago

Fixed in #148

akulaggarwal commented 7 years ago

thank you @wuliupo and @svi3c. was happening for me on my mac

computmaxer / karma-jspm

EMFILE, too many open files #97