ollietreend
commented
9 years ago Actually, my mistake – I misunderstood how the crawler handles external link. My understanding now is that external links do not have their children traversed – which is what I was asking for.
Please ignore. Nothing to see here. :blush:
It would be great to have the option to disable the crawling of external links.
I'd like to crawl an entire website, but am not interested in external links. By setting the 'depth' option to something suitably high in order to capture the entire website, I also end up doing a deep crawl of external websites.