search

akvo / akvo-provisioning

Server and development environment provisioning data and configurations
9 stars 2 forks source link

Akvodev.org & akvouat.org robots.txt #264

Closed Lynngre closed 8 years ago

Lynngre commented 8 years ago

As per https://github.com/akvo/akvo-web/issues/152 robots.txt was previously setup to stop search engines crawling our test servers, yet akvodev.org & akvouat.org is being crawled by google regardless.

I checked akvodev.org but cannot find a robots.txt file (or maybe my permissions don't allow me to see it) can someone please check this asap & if it is not set please block all search engines from crawling the test site.

stellanl commented 8 years ago

I am pretty sure the name must be robots.txt (plural).

Lynngre commented 8 years ago

Yes typo, robots.txt. Doing some Google webmaster testing but it's not reading the verification on the domain as yet.

Lynngre commented 8 years ago

screen shot 2016-02-22 at 21 03 40

So robots.txt for akvodev.org & akvouat.org is currently set to allow all to crawl. This should be set to disallowed as before BUT akvo.org's robots.txt should NOT change during automated deployment (so this change should only affect akvodev.org & akvouat.org).

orifito commented 8 years ago

duplicate of #158, closing it