search

MSU-NatSci / DomainAccessibilityAudit

Web application to create domain and subdomain accessibility audits, with violation statistics.
GNU General Public License v3.0
48 stars 19 forks source link

Respect Robots.txt Files #19

Open mgifford opened 3 years ago

mgifford commented 3 years ago

Robots have rights too. Well, ok, not yet, but we can get into trouble by ignoring the robots.txt files that some sites use.

Would be great if by default the scanner respected the wishes of the site owner.