Closed joemcgill closed 6 years ago
I think that this might have been addressed within our stack; @joehoyle or @nathanielks do you know?
We don't have a robots.txt, but we do disable indexing via WordPress in non-production environments: https://github.com/humanmade/hm-platform/blob/master/load.php#L59-L67
Do you think the above would satisfy the need @joemcgill?
I believe this is one and the same. Disabling indexing causes the robots.txt to Disallow, which is the same as the back-end option in settings
Missed responding to this earlier. I agree that this should do the trick. We should add documentation for HM_DISABLE_INDEXING
to the platform/deploys page on the engineering handbook.
It's easy to forget to change the "discourage Google" setting in WP > Settings > Reading each time you add a site to one of the environments on the server, which can lead to our development sites getting crawled. To avoid this, we could add code to our project configs that forces this setting on our dev/staging environments.