Multiprocess dispatcher

[austinbyers]

to: @javuto cc: @ryandeivert cc: @airbnb/binaryalert-maintainers size: small

Background

112 generalized the dispatcher, but did so by alternating between the different queues and polling from each. The AWS documentation for SQS Long Polling recommends instead to have different threads for each queue being polled. This way, the dispatcher is not blocked if one of its queues is empty.

Changes

• Use one Process for each queue that the dispatcher is polling
• Fix a small bug in the downloader introduced by #112 and update the downloader README
• Instead of invoking the dispatcher every minute, invoke every 5 minutes and use the maximum long poll timeout (20 seconds)
• Remove custom metrics from the dispatcher. They are not in the dashboard and have never proved useful for troubleshooting. Built-in metrics for Lambda and SQS have proven sufficient.

Testing

• ./manage.py configure - enable downloader
• ./manage.py deploy
• ./manage.py cb_copy_all - enqueue thousands of fake events to SQS
• ./manage.py analyze_all
• ./manage.py configure - remove downloader
• ./manage.py deploy - ensure the dispatcher still works with only a single queue

The dispatcher logs show that it is interleaving events from both the analyzer and downloader queues

[coveralls]

Coverage decreased (-0.06%) to 93.009% when pulling 28e4cdc8491bbfd5cc21609c07f5784ec07faf59 on multiprocess-dispatcher into 733675d13bb6386c135e992a0e2cfbcb1f2421c6 on master.

[ryandeivert]

nice work!! 😄