search

nrabinowitz / pjscrape

A web-scraping framework written in Javascript, using PhantomJS and jQuery
http://nrabinowitz.github.io/pjscrape/
MIT License
996 stars 159 forks source link

Robots.txt #58

Open johngeorgewright opened 9 years ago

johngeorgewright commented 9 years ago

Any thoughts on obeying robots.txt file?