search

humanmade / hm-base

Standard project layout for Human Made Projects.
81 stars 18 forks source link

Ensure robots.txt discourages all on dev/staging servers #62

Closed joemcgill closed 6 years ago

joemcgill commented 6 years ago

It's easy to forget to change the "discourage Google" setting in WP > Settings > Reading each time you add a site to one of the environments on the server, which can lead to our development sites getting crawled. To avoid this, we could add code to our project configs that forces this setting on our dev/staging environments.

mikeselander commented 6 years ago

I think that this might have been addressed within our stack; @joehoyle or @nathanielks do you know?

nathanielks commented 6 years ago

We don't have a robots.txt, but we do disable indexing via WordPress in non-production environments: https://github.com/humanmade/hm-platform/blob/master/load.php#L59-L67

mikeselander commented 6 years ago

Do you think the above would satisfy the need @joemcgill?

joehoyle commented 6 years ago

I believe this is one and the same. Disabling indexing causes the robots.txt to Disallow, which is the same as the back-end option in settings

joemcgill commented 6 years ago

Missed responding to this earlier. I agree that this should do the trick. We should add documentation for HM_DISABLE_INDEXING to the platform/deploys page on the engineering handbook.