[Request]: Add a robots.txt file once guides are fully migrated

[echappen]

A description of the work

Once guides are migrated and we are sure of our redirects from old guides, adding a robots.txt file to the site will help search engines find our site.

Search.gov has a good overview of what belongs in a robots.txt file. Our robots.txt file should include a link to the sitemap, as well as the recommended crawl delay from Search.gov. Crawl delays for all other search engines should probably be kept low / not too high, so that they can crawl the site in a timely manner but without overloading the site.

Point of contact on this issue

Eleni Chappen

Reproduction steps (if necessary)

No response

Skills Needed

• [ ] Any Human
• [ ] Design
• [ ] Content
• [X] Engineering
• [ ] Acquisition
• [ ] Product
• [ ] Other

Does this need to happen in the next 2 weeks?

• [ ] Yes
• [X] No

How much time do you anticipate this work taking?

1 day

Acceptance Criteria

• robots.txt file exists at the root of the site (guides.18f.gov/robots.txt)
• robots.txt file contains a link to the site's sitemap
• robots.txt file specifies appropriate crawl delay for search.gov and other search engines

  ### Tasks

[echappen]

This is blocked by https://github.com/18F/guides/issues/224

[bpdesigns]

@cannandevjust checking in to see if this ticket is in the right column. I see its unblocked by #224 above. Are there other blockers preventing this work?