search

opencharles / charles

Java web crawling library
BSD 3-Clause "New" or "Revised" License
32 stars 9 forks source link

Implement retrial of crawling #60

Closed amihaiemil closed 8 years ago

amihaiemil commented 8 years ago

The crawl might fail due to runtime exceptions. E.g. crawling with phantomjs sometimes fails. A way of retrial, with the possibility of using other WebDrivers should be implemented.

amihaiemil commented 8 years ago

Still need to implement SwitchableCrawl, which takes 2 WebCrawl object, and if the first one fails, the second one is run. This way, the user can try to crawl a website in different ways.

E.g. phantomjs webdriver failed, so switch to a webcrawl which uses other driver.

amihaiemil commented 8 years ago

done