A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
Support passing a URI object as the proxy settings. All #proxy= methods should accept a URI::HTTP object which is then converted to a Spidr::Proxy object.
Support passing a URI object as the proxy settings. All
#proxy=methods should accept aURI::HTTPobject which is then converted to aSpidr::Proxyobject.