Closed spelkey-ucd closed 1 year ago
A dynamic robots.txt that disallows crawlers for non-production site was added in https://github.com/UCDavisLibrary/main-wp-website/issues/69
However, this does not remove google search results that were already indexed on these sites. To do this, will have to temporarily allow robots, and add a meta tag: https://developers.google.com/search/docs/crawling-indexing/block-indexing
Deployed in v3.7.0
spelkey-ucd
commented
1 year ago spelkey-ucd
commented
1 year ago
A dynamic robots.txt that disallows crawlers for non-production site was added in https://github.com/UCDavisLibrary/main-wp-website/issues/69
However, this does not remove google search results that were already indexed on these sites. To do this, will have to temporarily allow robots, and add a meta tag: https://developers.google.com/search/docs/crawling-indexing/block-indexing