Robots file reader repaired

[stevelle]

The robots file reader now remembers a robots.txt file for each distinct hostname found during navigation and reuses it across all threads in the current run.

Instead of handing around a function handle a thread- safe object is used to keep track of the this, making the code a little easier to read.

Fixes #62

Checklist

• [x] This Pull will not add the same thing as another currently-open request.
• [x] Your Pull was made against the rivermont:dev branch and not rivermont:master.
• [x] This Pull does not commit any keys, passwords, personal data, or other private information.
• [?] I updated lines 20 and 21 in the README to reflect any changed I made.

[stevelle]

lines 20 and 21 might not be the same as when the template was created.