ekalinin / robots.js

Parser for robots.txt for node.js
MIT License
66 stars 21 forks source link

Fixed a bug with relative redirects and added unit tests #5

Closed bingalls-compete closed 11 years ago

bingalls-compete commented 11 years ago

There was an issue with redirects where if it was relative, it wouldn't have the full url to fetch and would throw an exception. I added a url.resolve call to cover this.

I also added a suite of unit tests to cover the parser.read functionality, and it should be pretty easy to extend in the future. It mocks up http and https so there shouldn't be any external calls.

ekalinin commented 11 years ago

Great patch! Thanks.