search

adjust / rmq

Message queue system written in Go and backed by Redis
MIT License
1.57k stars 206 forks source link

Unable to resume consuming once stopped #56

Closed kernle32dll closed 10 months ago

kernle32dll commented 5 years ago

Hi,

I don't know if this is by design - but after stopping to consume a Queue via StopConsuming, it is impossible to resume consuming via StartConsuming. The problem is that the latter function checks for deliveryChan being not nil. However, this will always be the case after consumption has started.

It would probably more correct to check for the consumingStopped value. However, I have no idea to gracefully handle the channel renewal (since the prefetchLimit might have changed).

wellle commented 5 years ago

@Kernle32DLL: So far StopConsuming was not meant to be used. In #33 we just changed it to be used and also documented it in the Readme. In there we also mention that it's not safe to stop consuming and then start again, as this would make the implementation more complicated.

If there is a real use case, we think it should be possible to implement. But so far we don't understand why you would need that. Can you give us a real example? Thank you.

kernle32dll commented 5 years ago

@wellle sure thing:

I have a setup with multiple running instances. All instances consume a given queue. However, in a scheduled manner one of the instances assumes the rule of a producer, and fills the queue in the first place. This is effectively a distributed task scheduler, where the producer is not predefined, but elected.

Now for stopping consumption: When the elected instance starts producing, I would like for it to temporarily stop consuming. Its not "that" important, but just for ease of mind (the instance is busy producing, so I don't want to further stress it). After the producing is done, the instance should transparently resume consuming.

I could of course build the consumer anew, but this seems a bit overkill on my end.

Jesse0Michael commented 4 years ago

@wellle I'm also interested in being about to pause and resume queue consumption. For my use case, we have built a circuit breaker into our rmq flow. We would like to, if the downstream service is dead causing the circuit breaker to open, pause the queue consumption to not exacerbate the issue and also not put unnecessary stress on Redis.

Right now, if the circuit breaker is open, the rmq consumer immediately calls delivery.Push(). Which I'm not sure if that kind of rapid pulling and pushing off/on the ready queue could cause unexpected issues with the server.

If there is a recommended way of already accomplishing this behavior, please let me know. Or if this is a feature that you think could belong in this library, but you don't the time to implement it yourself. Please let me know how you think it could be designed into rmq, and I may give it a shot. (Hacktoberfest is coming up 🎃!)

wellle commented 10 months ago

Closing, see conversation in #96. :v: