search

ContentMine / quickscrape

A scraping command line tool for the modern web
MIT License
259 stars 43 forks source link

Multiple url error #31

Closed blahah closed 9 years ago

blahah commented 9 years ago 
quickscrape   --urllist urls.txt   --scraperdir ~/code/journal-scrapers/scrapers/
info: quickscrape launched with...
info: - URLs from file: undefined
info: - Scraperdir: /Users/rds45/code/journal-scrapers/scrapers/
info: - Rate limit: 3 per minute
info: - Log level: info
info: urls to scrape: 6
info: processing URL: https://peerj.com/articles/704/
info: [scraper]. URL rendered. https://peerj.com/articles/704/.
info: waiting 20 seconds before next scrape

TypeError: Cannot call method 'concat' of undefined
    at Object.module.exports.compose (/Users/rds45/.nvm/v0.10.24/lib/node_modules/quickscrape/lib/eventparse.js:63:15)
    at null.<anonymous> (/Users/rds45/.nvm/v0.10.24/lib/node_modules/quickscrape/bin/quickscrape.js:153:18)
    at EventEmitter.emit (/Users/rds45/.nvm/v0.10.24/lib/node_modules/quickscrape/node_modules/thresher/node_modules/eventemitter2/lib/eventemitter2.js:339:22)
    at null.<anonymous> (/Users/rds45/.nvm/v0.10.24/lib/node_modules/quickscrape/node_modules/thresher/lib/thresher.js:69:14)
    at EventEmitter.emit (/Users/rds45/.nvm/v0.10.24/lib/node_modules/quickscrape/node_modules/thresher/node_modules/eventemitter2/lib/eventemitter2.js:339:22)
    at null.cb (/Users/rds45/.nvm/v0.10.24/lib/node_modules/quickscrape/node_modules/thresher/lib/scraper.js:309:15)
    at Ticker.tick (/Users/rds45/.nvm/v0.10.24/lib/node_modules/quickscrape/node_modules/thresher/lib/ticker.js:32:10)
    at null.<anonymous> (/Users/rds45/.nvm/v0.10.24/lib/node_modules/quickscrape/node_modules/thresher/lib/scraper.js:258:20)
    at EventEmitter.emit (events.js:98:17)
    at Request._callback (/Users/rds45/.nvm/v0.10.24/lib/node_modules/quickscrape/node_modules/thresher/lib/renderer/basic.js:16:16)
blahah commented 9 years ago

This was fixed in upstream thresher