search

thequbit / BarkingOwl

scalable web scraper framework for finding documents on websites.
GNU General Public License v3.0
19 stars 7 forks source link

Dispatch work via 0mq rather than threading #6

Closed thequbit closed 11 years ago

thequbit commented 11 years ago

It would be nice to dispatch work via 0mq rather than launch threads from a single python script. This would make the scraper significantly more scale-able.

thequbit commented 11 years ago

This was done with RabbitMQ. Closing.