Current behaviour:
- projects are executed sequentially.
- once all executed the crawling stops.
Desired behaviour:
The crawler must be able to run non stop:
1. low memory consumption fluctuation.
2. run multiple projects in parallel.
3. read the configuration file on constant intervals to change dynamically
the settings: adding/removing of projects/other settings like bandwidth,
depth etc.
4. trigger re-crawling of pages/projects based on rss triggers.
Original issue reported on code.google.com by andrei.p...@gmail.com on 17 Jul 2008 at 7:55
Original issue reported on code.google.com by
andrei.p...@gmail.com
on 17 Jul 2008 at 7:55