search

mysociety / popit

DEPRECATED - Development on PopIt has stopped and it is no longer being maintained
https://goo.gl/Vvej4Q
Other
76 stars 33 forks source link

"do not spider" option doesn't seem to be working #766

Closed mhl closed 9 years ago

mhl commented 9 years ago

For example, http://yournextmp-staging.popit.staging.mysociety.org/about says "Do not spider: Yes" but http://yournextmp-staging.popit.staging.mysociety.org/robots.txt is not found (404).

chrismytton commented 9 years ago

Fixed by updating the apache config. Turns out this was working correctly on ajax, but I'd foolishly put the robots.txt in the vhost root, so the files went away when it was re-deployed on pluto.

Additionally apache wasn't passing requests through to the live site's robots.txt it was trying to serve up a file that didn't exist, which I've also fixed. So now the "do not index" setting now works correctly on the live site and all the staging sites will automatically serve up the do not index version.