Switch to amqplib

[nathan-muir]

• [x] Major version bump
• [x] performance testing.
• [x] Review channel management
  • Publish tasks on a single channel (?)
  • Consume results on a single channel (?)
• [x] fail fast when connection buffer is full
• [ ] Support reconnects
• [ ] re-integrate support for onStarted
• [ ] Test failure modes
• [ ] Connection.close() doesn't wait for pending rpc calls to finish.
• [x] Support RPC backend
• [ ] Support Task v2 format.

[nathan-muir]

Performance

• Seems reasonable
• Much faster without confirms
• Much faster publishing on a single channel (until throughput exceeds buffers)
• Consuming from a single queue/channel would be better

Considerations

• An error on a channel causes the whole channel to enter an "error" state [1]
• Celery doesn't use "confirm"'s by default [2][3], to enable, set broker_transport_options={"confirm_publish":True}
• Celery v3.1.25 / v4 introduces a new task message protocol[4][5]

• Celery v4 introduces the RPC results backend [6] which involves using a single results queue per process (instead of creating a queue per invocation)

  [1] http://www.squaremobius.net/amqp.node/channel_api.html#dealing-with-failure [2] http://docs.celeryproject.org/en/latest/userguide/configuration.html#std:setting-broker_transport_options [3] https://tech.labs.oliverwyman.com/blog/2015/04/30/making-celery-play-nice-with-rabbitmq-and-bigwig/ [4] http://docs.celeryproject.org/en/latest/_modules/celery/app/amqp.html#AMQP.create_task_message [5] http://docs.celeryproject.org/en/latest/whatsnew-4.0.html#new-task-message-protocol [6] http://docs.celeryproject.org/en/latest/whatsnew-4.0.html#rpc-result-backend-matured

[nathan-muir]

Good reference for implementing AMQP