How to control the depth of crawling?

postmodern / spidr

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

MIT License

800 stars 109 forks source link

How to control the depth of crawling? #82

Closed masterbo98 closed 2 years ago

masterbo98 commented 2 years ago

When I user spidr, I find it really helpful and efficient. At present, I want to shorten the crawling time, is there any way or API to set the depth of spidr?

postmodern commented 2 years ago

You can use the max_depth: keyword argument which can be passed to Agent.start_at, .Agent.host,Agent.site, andAgent#initialize`. https://rubydoc.info/gems/spidr/Spidr/Agent#initialize-instance_method