Block search crawlers on CI, RC

mitodl / micromasters

Portal for learners and course teams to access MITx Micromasters® programs

https://mm.mit.edu

BSD 3-Clause "New" or "Revised" License

28 stars 16 forks source link

Closed pdpinch closed 1 year ago

pdpinch commented 1 year ago

As a google search user, it's confusing to see search results from MicroMasters RC.

on xPRO, we added a library django-robots which made it easy to customize the robots.txt file on each instance of the app.

[x] Add django-robots
[x] Do we need to revert https://github.com/mitodl/micromasters/pull/2540 ?
[ ] When deployed to RC, block all robots
[ ] When deployed to production, confirm that the expected pages are still crawlable.

pdpinch commented 1 year ago

@arslanashraf7 what's left to be done here?

arslanashraf7 commented 1 year ago

This is done.