search

taganaka / polipus

Polipus: distributed and scalable web-crawler framework
MIT License
92 stars 32 forks source link

Robots.txt option #12

Closed ABrisset closed 10 years ago

ABrisset commented 10 years ago

Hello,

Is there an option for robots.txt file, something like : obey_robots:true/false ?

Thanks.

taganaka commented 10 years ago

Not supported yet. On my todo list, though.

ABrisset commented 10 years ago

Glad to hear that ! Thank you for the good job on this gem.

taganaka commented 10 years ago

Support added: https://github.com/taganaka/polipus/pull/30

Available from version 0.3.0