search

dharmafly / noodle

A node server and module which allows for cross-domain page scraping on web documents with JSONP or POST.
https://noodle.dharmafly.com/
745 stars 69 forks source link

Out of memory #97

Closed megastef closed 9 years ago

megastef commented 10 years ago

Hi, my scrapper is running out of memory (after ~20000) log messages. What could go wrong? How is the cache working? I get a lot of messages like.

My Programm extracts fields from webpages and stores it to a database, I doubt that on this part is anything wrong, did it many times with other tools that noodle.js.

[noodle log #4853] Wed Apr 16 2014 00:23:49 GMT+0200 (CEST) Memory: 1321.11mb (1385281536 bytes) Cache: Result has been cached

premasagar commented 10 years ago

Hi @megastef, Perhaps a change to the cache settings will do it. See http://noodlejs.com/reference/#behaviour_settings You could try changing the resultsCacheMaxSize and/or pageCacheMaxSize.