DynamicScraper: Support timeout for page loading

ruipgil / scraperjs

A complete and versatile web scraper.

MIT License

3.7k stars 188 forks source link

DynamicScraper: Support timeout for page loading #35

Closed IvanGoncharov closed 8 years ago

IvanGoncharov commented 9 years ago

During scrapping I encounter couple pages that hangup during loading. I'm doing scrapping on thousands of pages in parallel, and some of them stuck forever(at least 10+ min). Timeout will solve this problem, ideally it would have user supplied value, but reasonable hardcode is also good.

ruipgil commented 9 years ago

That is a really good idea. I'll look into it.