search

postmodern / spidr

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
MIT License
800 stars 109 forks source link

Lambda based delay #63

Closed dcadenas closed 3 years ago

dcadenas commented 7 years ago

This changes let the user specify a dynamic delay so that any pattern (exponential, random?) can be used. A number is still supported so it's backwards compatible

postmodern commented 6 years ago

Could we detect whether @delay.respond_to?(:call) and then treat it as a callable object? This way we avoid converting constant numeric delays into lambdas. Also wouldn't mind adding some agent.delay { ... } syntactic sugar.