added code for creating robots.txt

[rskbansal]

Description

Previously, the code was written so as to return text data when localhost/robots.txt is visited by providing the -r or the --robots argument. Now, if the above mentioned arguments are passed, it creates a robots.txt file in the root directory of the server (either ./ or ./public).

Relevant issues

Fixes #825

Contributor checklist

• [x] Provide tests for the changes (unless documentation-only)
• [x] Documented any new features, CLI switches, etc. (if applicable)
  • [x] Server --help output
  • [x] README.md
  • [x] doc/http-server.1 (use the same format as other entries)
• [x] The pull request is being made against the master branch

Maintainer checklist

• [ ] Assign a version triage tag
• [ ] Approve tests if applicable

[rskbansal]

@thornjad

[BigBlueHat]

@rskbansal not sure we want an http-server writing files. What's the motivation for the change?

[rskbansal]

A robots.txt file shall be physically present on the server, that's what it is meant to be

[zbynek]

file shall be physically present

but why? For the user it's only important that the response for localhost/robots.txt contains the data, not that the file exists. The current implementation looks fine (except for the === true check mentioned in #825)

[BigBlueHat]

A robots.txt file shall be physically present on the server, that's what it is meant to be

If the file needs to be physically present for your use case, I'd recommend generating it prior to hosting and just not using the --robots option.

@zbynek thinks for connecting to #825. That does need a fix.