Update robots.txt

[cqueern]

Updated robots.txt so well-behaved search engine crawlers will not index Crossfeed instances, as I assume they're not meant to appear in search engine results due to the potentially sensitive information they collect.

🗣 Description

💭 Motivation and context

🧪 Testing

✅ Pre-approval checklist

• [x] This PR has an informative and human-readable title.
• [x] Changes are limited to a single goal - eschew scope creep!
• [ ] All future TODOs are captured in issues, which are referenced in code comments.
• [ ] All relevant type-of-change labels have been added.
• [ ] I have read the CONTRIBUTING document.
• [x] These code changes follow cisagov code standards.
• [x] All relevant repo and/or project documentation has been updated to reflect the changes in this PR.
• [x] Tests have been added and/or modified to cover the changes in this PR.
• [ ] All new and existing tests pass.

✅ Pre-merge checklist

• [ ] Revert dependencies to default branches.
• [ ] Finalize version.

✅ Post-merge checklist

• [ ] Create a release.

[Matthew-Grayson]

@cqueern Thanks for taking an interest in Crossfeed! You bring up an interesting point about whether or not we want to be indexed. I believe that allowing indexing was a deliberate decision, but I will look into it further.

[cqueern]

My pleasure! It's an important project.

Sounds good. If the decision is to exclude all robots from the entire server, suggest that line 3 in the robots.txt file remain

Disallow: /

If the decision is to allow all robots complete access, suggest that line 3 in the robots.txt file be updated to

Disallow: