Make checks thread safe, parallelize checks

benbalter / site-inspector

Ruby Gem to sniff information about a domain's technology and capabilities.

https://site-inspector.herokuapp.com

MIT License

89 stars 29 forks source link

Make checks thread safe, parallelize checks #70

Closed benbalter closed 9 years ago

benbalter commented 9 years ago

Fixes #65.

This roughly cuts the time to crawl a site in half and required some changes to how we memoize our Hydra instance in order to make it thread safe. TL;DR: Typhoeus::Hydra.hyrdra is thread safe, while Typhoeus::Hydra.new is not.

/cc https://github.com/oscardelben/firebase-ruby/issues/15, https://github.com/typhoeus/ethon/issues/85, https://github.com/typhoeus/ethon/issues/96