Describe the bug
Crawlers follow_redirects option means links that redirect are included, which is good. However redirects that end up on external domains should not be included in the crawl.
Sample configuration
Expected behavior
Redirects should be restricted to the rules applied elsewhere (e.g shouldCrawl)
Describe the bug Crawlers
follow_redirects
option means links that redirect are included, which is good. However redirects that end up on external domains should not be included in the crawl.Sample configuration
Expected behavior Redirects should be restricted to the rules applied elsewhere (e.g
shouldCrawl
)