[FEATURE REQUEST] Support other message brokers

[toszo]

Describe the solution you'd like RabbitMQ as a message broker for tasks.

Describe alternatives you've considered MQTT broker which would enable IoT usage.

Additional context Working on a distributed system that part of which will run on the Edge devices.

[hibiken]

@toszo Thank you for opening this issue!

I've thought about adding support for other message brokers and RabbitMQ is one of them. And this is definitely in the project's roadmap. However, I'd like to work on this after we have solid features implemented with Redis as a broker. If you need support for RabbitMQ soon, I'd say it's better to look for other solutions. It'll probably take a few months before I start working on adding support for other message brokers :)

[zhaolion]

😄 Hello, @hibiken. I had use Kafka & Redis as message broker for async tasks system at the same time(an ugly combination, but it's useful), I hope you will consider adding support for Kafka/RabbitMQ :)

[zhaolion]

@toszo Thank you for opening this issue!

I've thought about adding support for other message brokers and RabbitMQ is one of them. And this is definitely in the project's roadmap. However, I'd like to work on this after we have solid features implemented with Redis as a broker. If you need support for RabbitMQ soon, I'd say it's better to look for other solutions. It'll probably take a few months before I start working on adding support for other message brokers :)

How can I get the solid features plan in your design, Maybe I can contribute some features 🤓

[hibiken]

@zhaolion thanks for commenting! I've been exploring Kafka recently (I don't have any experience with it, so I need to learn it first 😄 ). I think it'd be great if asynq can also support Kafka as a message broker.

In terms of features to implement before working on adding support for other message brokers, I'd like to at least resolve these issues:

• Periodic Task feature #183
• Support Redis Cluster #182

Once I have tackled those issues, I think we can start looking into adding support to Kafka, RabbitMQ, etc

[rugwirobaker]

Nats would be nice too. The only reason I hesitate with Redis is data persistence with Redis is a bit more involved. I would love to help with nats of course if there is an isolated Store interface.

[no-serve-people]

maybe rocketmq is better

[Me1onRind]

An idea, other message queues are supported as plugin. Use interface to decoupling from specific storage.

[danielkurniadi]

Hi @hibiken just want to check on the progress of the blocking features that you mentioned above:

• Periodic Task feature [FEATURE REQUEST] Recurring scheduled tasks using a CRON schedule #183
• Support Redis Cluster Support RedisCluster client #182

I saw as of today, November 2021. I see its already merged into the codebase for asynq. If we haven't had any progress on the external support for other broker backend (.e.g RabbitMQ, Pulsar, RocketMQ, Kafka, or etc.) I can help with it.

What I will do for you, suppose you think its a good time to start or perhaps has not started yet on these features:

1. I'll survey into other async task-based queue on what their port (interface) / adapter design looks like to abstract and implement different broker backend. I think Machinery is a good example to look into as they have supported Redis, RabbitMQ, and Google Pubsub(?).
2. Come up with technical design on how to add port (interface) / adapter design and discuss with the team whether its a good design, review thread-safety issues and possible optimisation.
3. Then discuss the roadmap and plan how to refactor or migrate current solution to include more broker backend based on agreed design in (2).

Let me know if I can help on this part. You can contact me

• Email: daniel.thekurniadi@gmail.com or
• LinkedIn: https://www.linkedin.com/in/danielkur/

[hibiken]

@iqDF Thank you for checking in!

I'm still hesitant to start working on this yet since I still feel we haven't built out all the features we want to implement with Redis. For example, these are some of the feature requests I'm interested in tackling before we can say the library is feature complete with Redis as a message broker:

• 339

• 244

When we add support for other message brokers, it'll be difficult to add new features.

That being said, we can start looking into the design of the broker interface. Currently we do have an interface type (take a look at internal/base.Broker interface) but it's tightly coupled to the current implementation with Redis as a broker.

[danielkurniadi]

This blocking features of Redis are still available?

[danielkurniadi]

Anyway yup I'll take a look into the design of broker interface.

[shuqingzai]

Looking forward to this release, we now use RabbitMQ for queues and asynq for periodic tasks

[vladfr]

Hi @hibiken!

I found asynq while looking up Go and Mongo and queues. I write a small app in vue, backed by a REST server in Go and Mongo, and I really don't want to go into hosting Redis just for a few simple queues.

Is there any renewed interest on this issue? I would love to help out and abstract a backend layer. I really like the simplicity of this project, its interface and ease of use, cli and UI.

I'm thinking of MongoDB because it's widely used, good for small projects, and has the features we'd need (tail cursor, capped cols and atomic updates via FindAndModify().

[gedw99]

https://github.com/choria-io/asyncjobs Does same with nats and so is distributed globally using nats clustering

be Ayse it’s nats which has its own router you can add tasks at runtime , so pretty cool option is that’s what you need