weyoss
commented
10 months ago @wernermorgenstern Thank you for opening this issue.
RedisSMQ mainly focuses on producing and consuming messages while keeping things as simple as possible. So many features that you may find in BullMQ do not exist in RedisSMQ. Otherwise it won't be RedisSMQ.
Regarding consuming one message per Group ID or consumer group ID in general, such feature does not exist but I think that it may be considered to be included in the project's roadmap in future.
Nevertheless, you may use a workaround. For example an exclusive lock per consumer based on its Group ID. The lock should be freed by the consumer which acquired it when finishing its work. Adding a lock timeout may help to automatically free it in case of a consumer crash.
Keeping this issue open for now until further updates.
Hi, we are working on a redesign of our system. We pass messages from a Producer to consumer (Microservices).
So for example, a Producer will get a message from a MQTT Broker, and then send the message to a queue. Then a consumer service will consume the message, decrypt it, and do other things with it, and then put it on another queue, for other microservices to consume it.
And these microservices will need to be able to scale horizontally.
However, the MQTT Publisher will send a message every 10 seconds to the MQTT Broker. So the consumers will need to be able to pick up these in order per specific Device ID (which is the MQTT Publisher).
BullMQ Pro has got the batching and concurrency feature. We can set the group ID to be the device ID, and then the consumer will only pick up one message per Group ID (which helps in that it won't consume multiple messages from the same Group ID at the same time).
Does Redis-SMQ have those features? I have not seen those features in the documentation.
I do like Redis-SMQ, especially with the performance numbers (I have not compared them myself yet, within our system). I will do that in the next few days. Also, Redis-SMQ was pretty simple to add, and replace BullMQ Pro within our code.
The afore mentioned feature in regards to Batching and Concurrency is important.
Also, BullMQ supports sandboxed worker threads within NodeJS. Does Redis-SMQ support that? Because if we can use all CPU's within our AWS EKS environment, and allow NodeJS to handle "concurrency" using worker threads, that would help a lot with resource Usage.