Closed cmungall closed 10 years ago
Add a robots.txt with a stanza to allow indexing, with a delay. cc @kltm @harryhoch
https://en.wikipedia.org/wiki/Robots_exclusion_standard#Nonstandard_extensions
Can someone comment on the status of this?
oops, I made a dupe:
This is done, if people agree with the settings (follow the commit message in the other ticket)
Add a robots.txt with a stanza to allow indexing, with a delay. cc @kltm @harryhoch