A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
MIT License
800
stars
109
forks
source link
Anyway to limit the total number of pages crawled or shutdown the crawler after some criteria? #40
postmodern
commented
8 years ago
Use every_page and check queue.length.