search

thequbit / BarkingOwl

scalable web scraper framework for finding documents on websites.
GNU General Public License v3.0
19 stars 7 forks source link

scraper appears to loose connection randomly #36

Closed thequbit closed 9 years ago

thequbit commented 10 years ago

I think this is a non-wrapped error that is happening that start_consuming() is catching within pika. This happens when launching barkingowl-scraper.py. It may also happen when importing pypi package. Need to wrap all the things within try/except and find out what is going on.

thequbit commented 9 years ago

This is not seen if the scraper is not the first end point on the rabbitmq bus. If the dispatcher is launched first, there is no issue. This is probably the way I am sending messages to the bus through the BusAccess class.

thequbit commented 9 years ago

Fixed in 0.6.0. Was an issue with pika not being in it's own thread(s) for rx and tx.