search

ekalinin / robots.js

Parser for robots.txt for node.js
MIT License
66 stars 21 forks source link

Fix redirection when there is no location header. #21

Closed tfoxy closed 9 years ago

tfoxy commented 9 years ago

If the connection receives a redirection status code, but there is no location header, ignore the redirection and read the data.