issues
search
nrabinowitz
/
pjscrape
A web-scraping framework written in Javascript, using PhantomJS and jQuery
http://nrabinowitz.github.io/pjscrape/
MIT License
996
stars
159
forks
source link
Robots.txt
#58
Open
johngeorgewright
opened
9 years ago
johngeorgewright
commented
9 years ago
Any thoughts on obeying robots.txt file?
Any thoughts on obeying robots.txt file?