search

ekalinin / robots.js

Parser for robots.txt for node.js
MIT License
66 stars 21 forks source link

3xx redirects, Crawl-Delay, and response callback #1

Closed mlodz closed 12 years ago

mlodz commented 12 years ago

Hi ekalinin,

We've added support for 301, 302 redirects. So a request for http://reddit.com/robots.txt would successfully grab http://www.reddit.com/robots.txt

We've also added support for the crawl-delay directive, which allows site owners to specify the maximum speed at which they'd like a crawler to crawl their pages.

In addition, we've also added an argument to the parser constructor. It is a callback function, which is called after the remote file is downloaded and parsed.

We've really found your code useful, and are glad to help improve it. Let us know if any of our changes aren't up to par and we'll do our best to adhere to your coding style.

Thanks! Steve

ekalinin commented 12 years ago

Hi @mlodz,

Thank you for awesome patch! It was applied. Glad to see that robots.js was useful for you.

Regards, Eugene.