Optionally ignore the Robots Exclusion Protocol

wummel / linkchecker

check links in web documents or full websites

http://wummel.github.io/linkchecker/

GNU General Public License v2.0

1.41k stars 233 forks source link

Optionally ignore the Robots Exclusion Protocol #556

Open sebastian-luna-valero opened 9 years ago

sebastian-luna-valero commented 9 years ago

Hi all,

More than an issue it should be read as a feature request.

I was wondering whether it is possible or not to optionally ignore the robots exclusion protocol so you can check for broken links without modifying the configuration on your website.

I assume it could be easily implemented as a new command-line or configuration file option where you can temporary disable the current behaviour.

Many thanks in advance for your support.

batterystaples commented 9 years ago

This appears to be a duplicate of #508.