search

topcoder-platform / community-app

React webapp for serving Topcoder Community
125 stars 214 forks source link

Exclude dev site from search engine indexes #5270

Open ajefts opened 3 years ago

ajefts commented 3 years ago

Describe the bug Our dev site is getting indexed in search engines like google. We should make sure robots.txt (or the preferred way of doing it) is set to exclude our topcoder-dev.com sites from being indexed. Currently our results are getting watered down by dev links and it also takes users to pages that we don't want to send them to.

To Reproduce

Search google for topcoder-dev.com and you'll see many pages are indexed.

Expected behavior No dev links are returned by searches.

ajefts commented 3 years ago

@Oanh-and-only-Oanh Can you please add this one to the list for one of the next releases?