search

hibiken / asynq

Simple, reliable, and efficient distributed task queue in Go
MIT License
9.43k stars 681 forks source link

Can the consumer group function be supported? #410

Closed erosiners closed 2 years ago

hibiken commented 2 years ago

@erosiners Thank you for opening an issue!

Could you provide more details? It'd be easier to understand if you could provide an example use case.

erosiners commented 2 years ago

For example, after a user has successfully registered, multiple consumers may send emails or text messages to users at the same time and add points to users when a message is published. Emails and text messages are independent services, and they all need to subscribe to receive messages before processing them. The current single-consumer model cannot complete the appeal function. My English is not good, I hope you can understand

hibiken commented 2 years ago

@erosiners Thank you for providing more details!

Asynq does not support something like Kafka consumer group, so you need to use a different approach.

Using the example above, I can think of two approaches:

  1. Multiple messages approach: Enqueue one task for sending email and enqueue another task for sending text message
  2. Single message approach: Enqueue one task to send notifications (email + text message) and include idempotent key. Within the task handler, call EmailService and TextMessageService with the idempotent key (we need this idempotent key to make sure that we send one email and one text message, even if the task gets retried multiple times)

Let me know if you have any questions :)

hibiken commented 2 years ago

Closing this, feel free to reopen this issue if you have follow-up questions :)