add a new nodejs crawler

BruceDone / awesome-crawler

A collection of awesome web crawler,spider in different languages

MIT License

6.36k stars 688 forks source link

Closed zhuyingda closed 6 years ago

zhuyingda commented 6 years ago

add a new nodejs crawler, webster.

zhuyingda commented 6 years ago

Webster is a individual crawl framework, using puppeteer to scrape async content and using redis to implement task queue for concurency.