I honor that linkcheck respects robots.txt, but it makes the tool mostly useless for me.
A link checker should be able to check all links on the specified URL.
How about continuing respecting robots.txt by default but adding a command-line option to turn off this behavior, if needed?
For example: --ignore-robots-txt
I honor that linkcheck respects robots.txt, but it makes the tool mostly useless for me. A link checker should be able to check all links on the specified URL.
How about continuing respecting robots.txt by default but adding a command-line option to turn off this behavior, if needed? For example: --ignore-robots-txt