URLHeadBear.py: Use robots.txt

PrajwalM2212 commented 5 years ago

Requests that are not allowed by robots.txt are reported.

Closes https://github.com/coala/coala-bears/issues/1782

For short term contributors: we understand that getting your commits well defined like we require is a hard task and takes some learning. If you look to help without wanting to contribute long term there's no need for you to learn this. Just drop us a message and we'll take care of brushing up your stuff for merge!

Checklist

[ ] I read the commit guidelines and I've followed them.
[ ] I ran coala over my code locally. (All commits have to pass individually. It is not sufficient to have "fixup commits" on your PR, our bot will still report the issues for the previous commit.) You will likely receive a lot of bot comments and build failures if coala does not pass on every single commit!

After you submit your pull request, DO NOT click the 'Update Branch' button. When asked for a rebase, consult coala.io/rebase instead.

Please consider helping us by reviewing other peoples pull requests as well:

pick up any PR at https://coala.io/review
review it (check https://coala.io/reviewing for more info)
if you are sure that it needs work, use corobo mark wip <URL> to get it out of the review queue.

The more you review, the more your score will grow at coala.io and we will review your PRs faster!

PrajwalM2212 commented 5 years ago

PrajwalM2212 commented 5 years ago

Tests have not been added.

I wanted to know if this approach is okay before modifying existing tests.

PrajwalM2212 commented 5 years ago

@frextrite The message has to be shown. The issue asks the situation to be reported. Also I will wait for maintainers to see if this approach is okay as this bear is used by lot of other bears. Thanks for the review 😊

coala / coala-bears

URLHeadBear.py: Use robots.txt #2891

Checklist