laurentprudhon
commented
5 years ago Couldn't find a generic way to do this + #33 and #35 enhancements enable a better way to deal with this problem.
Recommended process :
- run a first extraction
- extraction will be stopped when unique percent drops below threshold
- look at the logs : identify a path pattern which was at the origin of the problem
- open the params file : add this path the exclusion list
- restart the crawl where it was interrupted
Today, we stop the crawl if the % of unique text blocks falls below 10%.
But sometimes, the problem is just that Abot is crawling a repetitive part of the website, and there are a lot of other interesting pages elsewhere on the site.
For example, while we are stuck enumerating all actions in this part of the website : http://bourse.latribune.fr/webfg/action/AIR-LIQUIDE-Euronext-Paris http://bourse.latribune.fr/webfg/action/TELEFONICA … the % of unique text block will fall below 10% But it doesn't mean there isn't a whole lot of interesting content in other parts of this site.
Implement a finer grained stopping scheme, where we simply stop crawling the current directory and all its subdirectories, reset the metric, and start again, until there is nothing more to crawl.