Updated robots.txt URL formatter

[dstjacques]

A URL with trailing slash could cause it to fail e.g. https://en.wikipedia.org/ URLs having the path as a substring of the domain could also fail because of the string replace occurring too many times e.g. http://example.com/example

Checklist

• [x] This Pull will not add the same thing as another currently-open request.
• [x] Your Pull was made against the rivermont:dev branch and not rivermont:master.
• [x] This Pull does not commit any keys, passwords, personal data, or other private information.
• [ ] I updated lines 20 and 21 in the README to reflect any changed I made.

[dstjacques]

Fixes #70

[rivermont]

Thanks @dstjacques for the help but this was also solved by #77 and I just didn't get around to this until now. Cheers!