This should be configurable by admins via the admin interface (so that admins using managed hosting can set these rules). For now it could be handled like the custom CSS interface: a <textarea> that lets you edit and save your robots.txt.
Probably a good idea to link to a robots.txt "cheat sheet" as well, containing common rules that you might want to implement.
Motivation
Communities should be able to set their own rules regarding web crawlers of all sorts.
Pitch
This should be configurable by admins via the admin interface (so that admins using managed hosting can set these rules). For now it could be handled like the custom CSS interface: a
<textarea>that lets you edit and save your robots.txt.Probably a good idea to link to a robots.txt "cheat sheet" as well, containing common rules that you might want to implement.
Motivation
Communities should be able to set their own rules regarding web crawlers of all sorts.