Update rabbit config to reject messages when queue is too large

[mrchrisadams]

At present, we have Rabbit MQ set to keep accepting messages even when we have too many backed up.

Given we've had something like a 20x increase in traffic in the last 6 months, this turns out to have been a bad idea, and as a result, RabbitMQ keeps falling over.

I'm not that well versed in RabbitMQ, but it looks like we'll need to implement some kind of backpressure to reject jobs that are coming in when we can't service them.

https://www.rabbitmq.com/maxlength.html

This will also involve thinking through a helpful error message for consumers of the API, so they know to back off.

[mrchrisadams]

Okay, I've set a policy of 500k max for now

rabbitmqctl set_policy max_500k_messages "^enqueue\.app\.default$" \
  '{"max-length":500000}' \
  --apply-to queues

At the rate of 250 messages which is common, that means we'll only ever have the queue get 30 mins behind before rejecting jobs.