feat: add django-robots for dynamic robots.txt

[arslanashraf7]

Pre-Flight checklist

• [x] Screenshots and design review for any changes that affect layout or styling
  • [x] Desktop screenshots
  • [ ] Mobile width screenshots
• [x] Migrations
  • [x] Migration is backwards-compatible with current production code
• [x] Testing
  • [ ] Code is tested
  • [x] Changes have been manually tested
• [x] Settings
  • [x] New settings are documented and present in app.json
  • [x] New settings have reasonable development defaults, if applicable

What are the relevant tickets?

5267

What's this PR do?

• Removes the static robots.txt
• Adds django-robots
• Adds an initial migration to pre-populate the robots.txt file with values from the existing one
• Adds settings for BASE_URL, ROBOTS_CACHE etc
• Enables Django's sites

How should this be manually tested?

• First, notice the existing robots.txt file in the project
• Add a value in .env for MICROMASTERS_BASE_URL=http://mm.odl.local:8079
• Move to this branch then docker-compose build followed by docker-compose up, notice the robots.txt file doesn't exist anymore in the static directory
• Upon moving this PR the build should be successful
• Check that a default website is added in sites in Django Admin with a value that matches MICROMASTERS_BASE_URL in your .env
• Check the Rules model under Robots in Django Admin and notice a rule exists with * (for all types of robots)
• Check that Url and check that it has existing disallowed URLs in that
• Check localhost/robots.txt you should see these added rules in that
• Add a Url manually, add that to the Rule and notice that it's reflected in localhost/robots.txt

Where should the reviewer start?

(Optional)

Any background context you want to provide?

The PR and its contents are inspired by https://github.com/mitodl/mitxpro/pull/681

Screenshots (if appropriate)

What GIF best describes this PR or how it makes you feel?

(Optional)

[codecov-commenter]

Codecov Report

Merging #5280 (f5e5734) into master (be8a31e) will increase coverage by 0.00%. The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master    #5280   +/-   ##
=======================================
  Coverage   93.64%   93.65%           
=======================================
  Files         501      501           
  Lines       23097    23101    +4     
  Branches      967      967           
=======================================
+ Hits        21629    21635    +6     
+ Misses       1361     1359    -2     
  Partials      107      107           

Impacted Files	Coverage Δ
micromasters/urls.py	93.33% <ø> (ø)
micromasters/settings.py	90.04% <100.00%> (+0.19%)	:arrow_up:
micromasters/admin.py	100.00% <0.00%> (+100.00%)	:arrow_up:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.