Describe the bug
Crawlers follow_redirects option means links that redirect are included, which is good. However redirects that end up on external domains should not be included in the crawl.
Sample configuration
Expected behavior
Redirects should be restricted to the rules applied elsewhere (e.g shouldCrawl)
stooit
commented
4 years ago
Describe the bug Crawlers
follow_redirectsoption means links that redirect are included, which is good. However redirects that end up on external domains should not be included in the crawl.Sample configuration
Expected behavior Redirects should be restricted to the rules applied elsewhere (e.g
shouldCrawl)