A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
When I user spidr, I find it really helpful and efficient. At present, I want to shorten the crawling time, is there any way or API to set the depth of spidr?
When I user spidr, I find it really helpful and efficient. At present, I want to shorten the crawling time, is there any way or API to set the depth of spidr?