A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
Add optional Logging/debug output to Spidr::Agent. Agent#initialize should accept a logger option for passing in custom Logger compatible objects. It should also support a logging: true|false option, which initializes @logger to a new Logger instance.
Add optional Logging/debug output to
Spidr::Agent
.Agent#initialize
should accept alogger
option for passing in custom Logger compatible objects. It should also support alogging: true|false
option, which initializes@logger
to a newLogger
instance.