Generalize from Kafka API to other message queues

feldera / dist-design

Design documents for distributed DBSP

6 stars 1 forks source link

Generalize from Kafka API to other message queues #4

Open blp opened 11 months ago

blp commented 11 months ago

The Kafka API is the dominant one among potential users that we've talked to. Other message queues might be of interest. It's worth thinking about how distribution could support other ones.

AMQP and its many implementations, e.g. RabbitMQ might be a second example to consider.

Suggested by @mihaibudiu.

blp commented 11 months ago

Possibly relevant AMQP implementations:

RabbitMQ
Apache Qpid
Apache ActiveMQ

Other possibly relevant message queues:

Apache RocketMQ (see that page for a RocketMQ vs ActiveMQ vs Kafka chart)

lalithsuresh commented 11 months ago

Amazon SQS is quite relevant.

One thing I'm not sure about with any of these is what their performance profile is like.