Use noindex meta tag or header, not robots.txt, to block untranslated docs pages

[tobiasmcnulty]

As far as I can tell, we're currently using robots.txt to block untranslated docs pages from being indexed by Google. Looking through Search Console, it appears that is not the correct approach and these pages are still being indexed:

That warning links to: https://support.google.com/webmasters/answer/7440203#indexed_though_blocked_by_robots_txt

Which suggests using the noindex meta tag or header instead: https://support.google.com/webmasters/answer/93710

[tobiasmcnulty]

I've also taken a snapshot of the index coverage report today (3/2/19) here, with some commentary and links to other, related issues: https://docs.google.com/spreadsheets/d/1l86YAEcw5CbvivuY-ZN81oy75Nh7T9g0eJX0sZ8Jww0/edit#gid=0

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[carltongibson]

@tobiasmcnulty Hi, similar to #878, is this still valid? (again, seems quite addressable if so 🤔)

[tobiasmcnulty]

Yup, this still looks like a valid issue:

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[carltongibson]

Not stale.

[CuriousLearner]

It's not stale.