search

ContentMine / quickscrape

A scraping command line tool for the modern web
MIT License
259 stars 43 forks source link

warning: possible EventEmitter memory leak detected. 11 listeners added. #52

Open chreman opened 9 years ago

chreman commented 9 years ago

was scraping a short list of urls (18), 10 went through, at the 11th came this warning. quickscrape then continued to the end of the list

(node) warning: possible EventEmitter memory leak detected. 11 listeners added. Use emitter.setMaxListeners() to increase limit. Trace at growListenerTree (/home/workshop/.nvm/v0.10.24/lib/node_modules/quickscrape/node_modules/thresher/node_modules/eventemitter2/lib/eventemitter2.js:206:23) at EventEmitter.on (/home/workshop/.nvm/v0.10.24/lib/node_modules/quickscrape/node_modules/thresher/node_modules/eventemitter2/lib/eventemitter2.js:366:24) at Thresher.scrape (/home/workshop/.nvm/v0.10.24/lib/node_modules/quickscrape/node_modules/thresher/lib/thresher.js:68:11) at processUrl (/home/workshop/.nvm/v0.10.24/lib/node_modules/quickscrape/bin/quickscrape.js:181:7) at null._onTimeout (/home/workshop/.nvm/v0.10.24/lib/node_modules/quickscrape/bin/quickscrape.js:204:5) at Timer.listOnTimeout as ontimeout (node) warning: possible EventEmitter memory leak detected. 11 listeners added. Use emitter.setMaxListeners() to increase limit. Trace at growListenerTree (/home/workshop/.nvm/v0.10.24/lib/node_modules/quickscrape/node_modules/thresher/node_modules/eventemitter2/lib/eventemitter2.js:206:23) at EventEmitter.on (/home/workshop/.nvm/v0.10.24/lib/node_modules/quickscrape/node_modules/thresher/node_modules/eventemitter2/lib/eventemitter2.js:366:24) at Thresher.scrape (/home/workshop/.nvm/v0.10.24/lib/node_modules/quickscrape/node_modules/thresher/lib/thresher.js:72:11) at processUrl (/home/workshop/.nvm/v0.10.24/lib/node_modules/quickscrape/bin/quickscrape.js:181:7) at null._onTimeout (/home/workshop/.nvm/v0.10.24/lib/node_modules/quickscrape/bin/quickscrape.js:204:5) at Timer.listOnTimeout as ontimeout

blahah commented 9 years ago

This is very strange - I explicitly set the max number of listeners to be unlimited for all the event emitters. I can't immediately see the cause but will keep investigating.

petermr commented 8 years ago

I just got the same error - after ca 20 downloads:

info: processing URL: http://dx.doi.org/10.1080/14780887.2016.1183065
(node) warning: possible EventEmitter memory leak detected. 11 listeners added. Use emitter.setMaxListeners() to increase limit.
Trace
    at Request.addListener (events.js:160:15)
    at Request.init (/Users/pm286/.nvm/v0.10.38/lib/node_modules/quickscrape/node_modules/thresher/node_modules/request/request.js:667:8)
    at Redirect.onResponse (/Users/pm286/.nvm/v0.10.38/lib/node_modules/quickscrape/node_modules/thresher/node_modules/request/lib/redirect.js:149:11)
    at Request.onRequestResponse (/Users/pm286/.nvm/v0.10.38/lib/node_modules/quickscrape/node_modules/thresher/node_modules/request/request.js:1108:22)
    at ClientRequest.emit (events.js:95:17)
    at HTTPParser.parserOnIncomingClient [as onIncoming] (http.js:1689:21)
    at HTTPParser.parserOnHeadersComplete [as onHeadersComplete] (http.js:122:23)
    at Socket.socketOnData [as ondata] (http.js:1584:20)
    at TCP.onread (net.js:528:27)
(node) warning: possible EventEmitter memory leak detected. 11 listeners added. Use emitter.setMaxListeners() to increase limit.
Trace
    at Request.addListener (events.js:160:15)
    at Request.start (/Users/pm286/.nvm/v0.10.38/lib/node_modules/quickscrape/node_modules/thresher/node_modules/request/request.js:1001:8)
    at Request.end (/Users/pm286/.nvm/v0.10.38/lib/node_modules/quickscrape/node_modules/thresher/node_modules/request/request.js:1531:10)
    at end (/Users/pm286/.nvm/v0.10.38/lib/node_modules/quickscrape/node_modules/thresher/node_modules/request/request.js:734:14)
    at Object._onImmediate (/Users/pm286/.nvm/v0.10.38/lib/node_modules/quickscrape/node_modules/thresher/node_modules/request/request.js:748:7)
    at processImmediate [as _immediateCallback] (timers.js:354:15)
^Clocalhost:xref pm286$
chartgerink commented 8 years ago

Got it too just now (note, still running 0.4.6; just haven't updated the machine I run quickscrape on continuously)

When I restart, it just picks up where it left off without errors. This has only happened in taylor francis papers for me. Where did it happen for you?

petermr commented 8 years ago

was a T+F paper for me.

On Tue, May 31, 2016 at 6:07 PM, Chris Hartgerink notifications@github.com wrote:

Got it too just now (note, still running 0.4.6; just haven't updated the machine I run quickscrape on continuously)

When I restart, it just picks up where it left off without errors. This has only happened in taylor francis papers for me. Where did it happen for you?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ContentMine/quickscrape/issues/52#issuecomment-222754135, or mute the thread https://github.com/notifications/unsubscribe/AAsxS7d4IQVS2TY4EC--W2j0tPxOSgkUks5qHGq2gaJpZM4FWr-A .

Peter Murray-Rust Reader in Molecular Informatics Unilever Centre, Dep. Of Chemistry University of Cambridge CB2 1EW, UK +44-1223-763069

petermr commented 8 years ago

Fuller details:

localhost:xref pm286$ quickscrape -o tf -r ../1080.txt -s ../../../../journal-scrapers/scrapers/tf.json -i 15
info: quickscrape 0.4.7 launched with...
info: - URLs from file: undefined
info: - Scraper: /Users/pm286/workspace/journal-scrapers/scrapers/tf.json
info: - Rate limit: 15 per minute
info: - Log level: info
info: urls to scrape: 10
info: processing URL: http://dx.doi.org/10.1080/00224545.2016.1184125
info: [scraper]. URL rendered. http://www.tandfonline.com/doi/full/10.1080/00224545.2016.1184125.
(node) warning: possible EventEmitter memory leak detected. 11 listeners added. Use emitter.setMaxListeners() to increase limit.
Trace
    at Request.addListener (events.js:160:15)
    at Request.start (/Users/pm286/.nvm/v0.10.38/lib/node_modules/quickscrape/node_modules/thresher/node_modules/request/request.js:1001:8)
    at Request.end (/Users/pm286/.nvm/v0.10.38/lib/node_modules/quickscrape/node_modules/thresher/node_modules/request/request.js:1531:10)
    at end (/Users/pm286/.nvm/v0.10.38/lib/node_modules/quickscrape/node_modules/thresher/node_modules/request/request.js:734:14)
    at Object._onImmediate (/Users/pm286/.nvm/v0.10.38/lib/node_modules/quickscrape/node_modules/thresher/node_modules/request/request.js:748:7)
    at processImmediate [as _immediateCallback] (timers.js:354:15)
(node) warning: possible EventEmitter memory leak detected. 11 listeners added. Use emitter.setMaxListeners() to increase limit.
Trace
    at Request.addListener (events.js:160:15)
    at Request.start (/Users/pm286/.nvm/v0.10.38/lib/node_modules/quickscrape/node_modules/thresher/node_modules/request/request.js:1001:8)
    at Request.end (/Users/pm286/.nvm/v0.10.38/lib/node_modules/quickscrape/node_modules/thresher/node_modules/request/request.js:1531:10)
    at end (/Users/pm286/.nvm/v0.10.38/lib/node_modules/quickscrape/node_modules/thresher/node_modules/request/request.js:734:14)
    at Object._onImmediate (/Users/pm286/.nvm/v0.10.38/lib/node_modules/quickscrape/node_modules/thresher/node_modules/request/request.js:748:7)
    at processImmediate [as _immediateCallback] (timers.js:354:15)
(node) warning: possible EventEmitter memory leak detected. 11 listeners added. Use emitter.setMaxListeners() to increase limit.
Trace
    at Request.addListener (events.js:160:15)
    at Request.start (/Users/pm286/.nvm/v0.10.38/lib/node_modules/quickscrape/node_modules/thresher/node_modules/request/request.js:1001:8)
    at Request.end (/Users/pm286/.nvm/v0.10.38/lib/node_modules/quickscrape/node_modules/thresher/node_modules/request/request.js:1531:10)
    at end (/Users/pm286/.nvm/v0.10.38/lib/node_modules/quickscrape/node_modules/thresher/node_modules/request/request.js:734:14)
    at Object._onImmediate (/Users/pm286/.nvm/v0.10.38/lib/node_modules/quickscrape/node_modules/thresher/node_modules/request/request.js:748:7)
    at processImmediate [as _immediateCallback] (timers.js:354:15)
(node) warning: possible EventEmitter memory leak detected. 11 listeners added. Use emitter.setMaxListeners() to increase limit.
Trace
    at Request.addListener (events.js:160:15)
    at Request.init (/Users/pm286/.nvm/v0.10.38/lib/node_modules/quickscrape/node_modules/thresher/node_modules/request/request.js:667:8)
    at Redirect.onResponse (/Users/pm286/.nvm/v0.10.38/lib/node_modules/quickscrape/node_modules/thresher/node_modules/request/lib/redirect.js:149:11)
    at Request.onRequestResponse (/Users/pm286/.nvm/v0.10.38/lib/node_modules/quickscrape/node_modules/thresher/node_modules/request/request.js:1108:22)
    at ClientRequest.emit (events.js:95:17)
    at HTTPParser.parserOnIncomingClient [as onIncoming] (http.js:1689:21)
    at HTTPParser.parserOnHeadersComplete [as onHeadersComplete] (http.js:122:23)
    at Socket.socketOnData [as ondata] (http.js:1584:20)
    at TCP.onread (net.js:528:27)
(node) warning: possible EventEmitter memory leak detected. 11 listeners added. Use emitter.setMaxListeners() to increase limit.
Trace
    at Request.addListener (events.js:160:15)
    at Request.init (/Users/pm286/.nvm/v0.10.38/lib/node_modules/quickscrape/node_modules/thresher/node_modules/request/request.js:667:8)
    at Redirect.onResponse (/Users/pm286/.nvm/v0.10.38/lib/node_modules/quickscrape/node_modules/thresher/node_modules/request/lib/redirect.js:149:11)
    at Request.onRequestResponse (/Users/pm286/.nvm/v0.10.38/lib/node_modules/quickscrape/node_modules/thresher/node_modules/request/request.js:1108:22)
    at ClientRequest.emit (events.js:95:17)
    at HTTPParser.parserOnIncomingClient [as onIncoming] (http.js:1689:21)
    at HTTPParser.parserOnHeadersComplete [as onHeadersComplete] (http.js:122:23)
    at Socket.socketOnData [as ondata] (http.js:1584:20)
    at TCP.onread (net.js:528:27)
(node) warning: possible EventEmitter memory leak detected. 11 listeners added. Use emitter.setMaxListeners() to increase limit.
Trace
    at Request.addListener (events.js:160:15)
    at Request.init (/Users/pm286/.nvm/v0.10.38/lib/node_modules/quickscrape/node_modules/thresher/node_modules/request/request.js:667:8)
    at Redirect.onResponse (/Users/pm286/.nvm/v0.10.38/lib/node_modules/quickscrape/node_modules/thresher/node_modules/request/lib/redirect.js:149:11)
    at Request.onRequestResponse (/Users/pm286/.nvm/v0.10.38/lib/node_modules/quickscrape/node_modules/thresher/node_modules/request/request.js:1108:22)
    at ClientRequest.emit (events.js:95:17)
    at HTTPParser.parserOnIncomingClient [as onIncoming] (http.js:1689:21)
    at HTTPParser.parserOnHeadersComplete [as onHeadersComplete] (http.js:122:23)
    at Socket.socketOnData [as ondata] (http.js:1584:20)
    at TCP.onread (net.js:528:27)

[HANGS at this stage]

with URLs in file:

http://dx.doi.org/10.1080/00224545.2016.1184125
http://dx.doi.org/10.1080/00224545.2016.1184126
http://dx.doi.org/10.1080/00224545.2016.1184128
http://dx.doi.org/10.1080/01443410.2016.1176125
http://dx.doi.org/10.1080/10508619.2016.1174568
http://dx.doi.org/10.1080/10508619.2016.1183251
http://dx.doi.org/10.1080/10584609.2016.1166168
http://dx.doi.org/10.1080/1359432x.2016.1176021
http://dx.doi.org/10.1080/14780887.2016.1183065
http://dx.doi.org/10.1080/14789949.2016.1178504

They may be behind paywall