We added */contributors.txt to the robots.txt with the intention to hide those pages from Google, but in fact it seems to prevent Google from seeing the X-Robots-Tag header, so Google keeps the pages in the index, although we would like to have them removed from the index.
Solution
Remove the */contributors.txt entry from the robots.txt again.
Summary
(MP-1199)
Problem
We added
*/contributors.txtto therobots.txtwith the intention to hide those pages from Google, but in fact it seems to prevent Google from seeing theX-Robots-Tagheader, so Google keeps the pages in the index, although we would like to have them removed from the index.Solution
Remove the
*/contributors.txtentry from therobots.txtagain.How did you test this change?
Trivial change.