Closed buren closed 8 years ago
Eh, I prefer to ignore robots.txt
and spider everything. However, I do like the idea of providing the option though. Instead of adding it to allowed_links
, I think it should be added to Spidr::Agent#visit?.
Something like e2c2883? Its not finished yet, I haven't added any tests yet. Just wanted to get your feedback first :)
Yeah, but get rid of the NoRobots null object. Just do a @robots && @robots.allow?(uri)
. Note that Robots#allowed? accepts URIs. Also, should robots be all or nothing, or configurable per host?
I removed the null object, and did @robots ? @robots.allowed?(uri) : true
to have it return true if options[:robots]
is falsy.
My guess is that in most cases you would either respect it for all hosts or just ignore robots.txt completely. What do you think?
Usage:
Closes #11