search

postmodern / spidr

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
MIT License
800 stars 109 forks source link

Switch to using async-http #76

Open postmodern opened 2 years ago

postmodern commented 2 years ago

Switch from using Ruby's net/http to using async-http. This would allow for easy connection pooling and concurrent requests, without the overhead of threads and mutexes.

postmodern commented 2 years ago

This might be temporarily blocked by socketry/async-http#35. The test suite depends heavily on WebMock.