search

opensangja / abot

Automatically exported from code.google.com/p/abot
Apache License 2.0
0 stars 0 forks source link

Add crawl recovery #29

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
Add crawl recovery that reloads pages that were crawled, pages to crawl and 
other context. This allows the crawl to pick up where it left off. May also 
need to add a stop for this work properly

Original issue reported on code.google.com by sjdir...@gmail.com on 16 Nov 2012 at 5:15

GoogleCodeExporter commented 9 years ago 
Serialize the crawlcontext/scheduler/etc to file... While your at it add 
start/pause/stop/resume.

Original comment by sjdir...@gmail.com on 31 Dec 2012 at 3:58

GoogleCodeExporter commented 9 years ago

Original comment by sjdir...@gmail.com on 31 Dec 2012 at 6:09

GoogleCodeExporter commented 9 years ago

Original comment by sjdir...@gmail.com on 13 Jan 2013 at 9:33

GoogleCodeExporter commented 9 years ago 
Not enough good use cases right now to justify adding this feature. I wouldn't 
consider this a necessary feature for a crawling framework and that is exactly 
what Abot is. As of right now I will not implement this feature to be sure to 
keep the api as simple as possible.

Original comment by sjdir...@gmail.com on 3 Feb 2013 at 10:23