search

ekalinin / robots.js

Parser for robots.txt for node.js
MIT License
66 stars 21 forks source link

Allow custom HTTP options #14

Closed richorama closed 10 years ago

richorama commented 10 years ago

... to enable basic auth, or other custom headers on the HTTP request.

ekalinin commented 10 years ago

Great patch! Thanks!