Curation: document user agent string for linkchecker

clarin-eric / cereal

Central Repository for All documentation

GNU General Public License v3.0

0 stars 0 forks source link

Curation: document user agent string for linkchecker #3

Closed twagoo closed 6 months ago

twagoo commented 1 year ago

Item 18 explains that "[the] Link Checker respects robots.txt", but it would be good to also give some concrete information on how to adapt one's robots.txt file to allow/disallow link checking.

wowasa commented 6 months ago

done - it's in the faq