search

VentureGardenGroup / RFC

Request for Comments
0 stars 0 forks source link

Choosing a Message Queue #1

Open wale-o opened 8 years ago

wale-o commented 8 years ago

I am working on a project that needs a message queue. I have been researching a message queue to use and I have narrowed it down to rabbitMQ. Before a final decision is made, I will like to have opinions on this. This consideration will affect the following

  1. Strategy: this means everyone in need of a message queue will have to use rabbitMQ for their projects so we don't duplicate efforts and also encourage the team to develop expertise on this
  2. Infrastructure: We will eventually need to deploy the message queue. Sticking to one means everyone can quickly leverage the same instance instead of acquiring additional hardware cost to deploy another one

Other message queues I have considered are Apache ActiveMQ, Redis, Amazon SQS and Kafka. But I have decided against all these for one reason or the other

OElesin commented 8 years ago

Hi Wale,

Queuing messages is a great way to scale your application, I recommend you use Apache Kafka as it is currently part of the DS team stack.

  1. Persistence: From my experience, RabbitMQ does not persist messages as much as Apache Kafka does. By setting your retention configuration, Kafka can retain messages in a topic based on the duration you set. This could also be set in RabbitMQ configuration during queue creation.
  2. Scalability: Apache Kafka scales horizontally out of the box. By adding more brokers, you can easily expand your Kafka cluster to multiple nodes. This also deep dives into high availability. In case your master broker goes down, Kafka automatically assigns another node in the cluster as the master while ensuring messages in your topic are still intact.
  3. Community Support: Kafka was designed by LinkedIn and the community usage is growing rapidly as other top companies in the technology space are adopting Kafka.
  4. Client Libraries: Kafka has integration with the major programming languages.

However, one must consider the use case appropriately as well as system resources available. Message queuing is brilliant way to scale applications (especially when designing microservices)

johnzon commented 8 years ago

Hi Wale,

I narrowed down my review to RabbitMQ and Kafka.

  1. RabbitMQ has great client libraries for other programming languages including C#. Thus integrating this with our existing projects will be seamless. Kafka is Java based, its non-java client libraries are still not mature.
  2. With good documentation, developers can accomplish much. RabbitMQ no doubt has great and mature documentation.
  3. Kafka has its architecture tuned to a particular usage pattern: performance, strict orderliness of messages while RabbitMQ is more of general purpose.

The use case really matters as @OElesin has said, I agree with Deji that we should try the two and adopt the best. Meanwhile this conversation between Jonathan(of Kafka) and Alex(of RabbitMQ) provides great insight into the internal workings of the two. It's long but not boring.