Due an issue with PanthomJS (I've tested on 1.7.0 in both MacOSX and Debian 6) issue 353, the page.open() event is being called multiple times on some url's. This is related to iframes being created within the page (you can find more details in the open issue).
pjscrape.js (master branch)
line 680 // run the scrape
line 681 page.open(url, function(status) {
Below you can see an output example of how the log looks like when scraping is invoked many times:
I've applied a workaround in the meantime in order to stop duplicated events. I use the visited array to know if that page was already visited. I added a condition before line 700 as you can see below:
pjscrape.js (master branch) line 700
if(visited[url]) {
log.msg('Page recalled: ' + url);
return;
}
// mark as visited
visited[url] = true;
Due an issue with PanthomJS (I've tested on 1.7.0 in both MacOSX and Debian 6) issue 353, the
page.open()
event is being called multiple times on some url's. This is related to iframes being created within the page (you can find more details in the open issue).pjscrape.js (master branch)
line 680 // run the scrape line 681 page.open(url, function(status) {
Below you can see an output example of how the log looks like when scraping is invoked many times:
I've applied a workaround in the meantime in order to stop duplicated events. I use the visited array to know if that page was already visited. I added a condition before line 700 as you can see below:
pjscrape.js (master branch) line 700
Hope this help to fix this bug. Diego