Closed rivermont closed 7 years ago
There should be an option (disable-able) to ignore links that are forbidden by a site's robots.txt. Another library or huge regex might need to be used to parse out the domain the page is on.
robots.txt
There should be an option (disable-able) to ignore links that are forbidden by a site's
robots.txt. Another library or huge regex might need to be used to parse out the domain the page is on.