Closed buismaarten closed 9 months ago
I think the current code is necessary: in normal circumstances we want to stop crawling when we're not on our own domain anymore, but if CrawlSubdomains
is used, it is allowed.
When the CrawlAllUrls
profile is used, is it needed to skip the URL? This crawler normally works when we're not on our own domain.
Yes, it is needed. Otherwise the crawl would start crawling other domains and never stop.
Maybe I don't fully understand it, but isn't the reason why we can specify a depth or a limit while crawling?
No, the idea of the Crawler is that you only crawl one domain, and not go outside that domain by default.
Is it an idea to make the if-statement configurable? Because the current crawler profiles cannot fix this problem at this moment.
Feel free to send a PR with a concrete proposal + tests on how to handle this better
When the
CrawlSubdomains
profile is not used, no additional code is used to match some hostnames in my opinion. Is there a specific reason to include this code?Fixes issue #450