search

ruipgil / scraperjs

A complete and versatile web scraper.
MIT License
3.7k stars 188 forks source link

OSX Dynamic Scraper Error: spawn EMFILE #40

Open vdraceil opened 9 years ago

vdraceil commented 9 years ago

This happens to me more frequently (in OSX) when using DynamicScraper. Is this possibly because of routing too many URLs? ... may be, too many Phantom instances are getting created.

child_process.js:958
    throw errnoException(process._errno, 'spawn');
          ^
Error: spawn EMFILE
    at errnoException (child_process.js:1011:11)
    at ChildProcess.spawn (child_process.js:958:11)
    at exports.spawn (child_process.js:746:9)
    at spawn (/Users/kannanv/Personal/Workplace/helloworld/node_modules/scraperjs/node_modules/phantom/node_modules/win-spawn/index.js:54:10)
    at startPhantomProcess (/Users/kannanv/Personal/Workplace/helloworld/node_modules/scraperjs/node_modules/phantom/phantom.js:17:12)
    at Server.<anonymous> (/Users/kannanv/Personal/Workplace/helloworld/node_modules/scraperjs/node_modules/phantom/phantom.js:115:14)
    at Server.emit (events.js:92:17)
    at net.js:1056:10
    at process._tickCallback (node.js:442:13)
ruipgil commented 9 years ago

Yes, it looks like the problem may be related with it. Try increasing the ulimit: https://github.com/gruntjs/grunt/issues/788#issuecomment-18088252