Closed JuliusHenke closed 2 years ago
aecio
commented
2 years ago I think you should probably not be using the NonRandomLinkSelector. This is poorly documented, but it still exists in the codebase only for legacy reasons. Can you try another selector such as the default TopkLinkSelector or the MaximizeWebsitesLinkSelector if you are crawling more than one website?
JuliusHenke
commented
2 years ago Thanks for the quick reply. I did try all of these type of link selectors:
As expected, it resulted in a different order of when a page was crawled, but did not change the total amount of crawled pages. I also tried not using Tor.
aecio
commented
2 years ago What if you also set the link classifier to something like the following?
link_storage.link_classifier.type: MaxDepthLinkClassifier
link_storage.link_classifier.max_depth: 2147483647The link classifier (documented here: https://ache.readthedocs.io/en/latest/crawling-strategies.html#link-classifiers) is the class actually assigns the scores and defines the crawling order. The MaxDepthLinkClassifier will crawl pages in the same order that they are discovered up to a maximum depth.
Another option could be the LinkClassifierRandom.
link_storage.link_classifier.type: LinkClassifierRandomSetting this actually did the trick and the crawler downloaded all expected pages:
link_storage.link_classifier.type: MaxDepthLinkClassifier
link_storage.link_classifier.max_depth: 2147483647Thanks a lot!
Hi, I am currently trying to crawl a forum, which uses paginated result lists. When crawling the forum with a whitelist regex filter, the crawler does not seem to crawl all relevant pages. I am unsure if this is due to links not being recognized or falsely not being scheduled. I do think my regex is correct. The crawler is configured to use Tor and started with Docker Compose.
ache.yml config
tor.seeds
link-filters.yml
Result of crawled pages, which are listed in "/default/data_monitor/crawledpages.csv"
The result misses all pages between 7 and 980. Do you have any idea, if I have misconfigured something? If this is a bug and you can point me into the right direction, I am also willing to fix this. What is the best setup to debug something like this with the ACHE source code?
Crawler log