search

oleics / node-filewalker

Fast and rock-solid asynchronous traversing of directories and files for node.js
67 stars 8 forks source link

stream event: Error: EMFILE: too many open files #11

Open strarsis opened 8 years ago

strarsis commented 8 years ago

When attaching a handler to stream event, recursively traversing a larger amount of files in a folder using node-filewalker, an error message occurs (usually at the same point (12114 files)):

[...]
Error: EMFILE: too many open files[...]
[...]

The ReadStream is even closed in the stream event handler, but this doesn't have an effect.

Windows 10 Home Premium x64, node v4.4.5, node-filewalker v0.1.3

oleics commented 8 years ago

Did you try the maxPending option?

strarsis commented 8 years ago

Wouldn't it be a good feature if node-filewalker determines/find out retry what the limit currently is? Could node-graceful-fs be integrated?

oleics commented 8 years ago

This was/is in fact the next feature that should be integrated:

https://github.com/oleics/node-filewalker/blob/58a0bff/lib/filewalker.js#L92

Linking detectedMaxOpen and maxPending if a flag is enabled should do the trick ;)

Using graceful-fs (which is a great node-module) could have negative impacts on performance.