search

taoensso / carmine

Redis client + message queue for Clojure
https://www.taoensso.com/carmine
Eclipse Public License 1.0
1.15k stars 131 forks source link

Queue worker throttle-ms doesn't respect given value #152

Closed uaalto closed 8 years ago

uaalto commented 8 years ago

Perhaps is a missunderstanding of how the queue works, but using 1 thread and 50 throttle-ms, 50 eoq-backoff-ms handles the messages at a rate closer to 1 second than 50 milliseconds. The handler is definitely not the bottleneck. I've tried also with 4 threads and different values of throttling and backoff, but it never gets nowhere closer to 50 msecs.

ptaoussanis commented 8 years ago

Hi Ulysses,

If you've got 50ms for both :throttle-ms and :eoq-backoff-ms then the maximum poll time should be 100ms (50ms to backoff + 50ms to initiate poll). If you've got an empty queue, that'd make maximum pickup time about 200-300ms.

So handling time will be <(300ms + hander time + queue backlog).

How many items in the queue and how long does the handler take to run? For debugging, might be helpful to mod your options to {:throttle-ms 1 :eoq-backoff-ms 1} - that'll give you an idea of your max possible processing performance for the selected number of threads. You'll want to adjust the thread count based on the queue size.

For reference: each item in the queue will (eventually) require at least 2 polls to process (1 to pickup and 1 to garbage collect). If you've got lots of message backoffs, the poll count goes up so you'll likely also want more threads.

uaalto commented 8 years ago

Thank you for the reply @ptaoussanis I've indeed tried with both values set at 1. But it still takes closer to a second to operate (way longer than the operation does, no matter how many threads). This happens of testing, could it be that testing profile slows this down on purpose? I couldn't see that in the code though.

ptaoussanis commented 8 years ago

How many items are in your queue?

This happens of testing, could it be that testing profile slows this down on purpose?

No, sorry. And if you've got {:throttle-ms 1 :eoq-backoff-ms 1} with an empty queue, time to pickup should be <5ms so time-to-handle should be dominated by your handler times.

You're welcome to send me a reproducible example if you'd like me to take a closer look.

uaalto commented 8 years ago

I have 20 items only. The time to process them is constant. Even if the handler takes this long (which doesn't), wouldn't multiple threads handle several simultaneously reducing the effect of blocking i/o operations?

ptaoussanis commented 8 years ago

Time to process n items should be O(n); if the time you're seeing is constant, that may be an sign that something's off.

Have you confirmed that your method of measuring times is accurate?

Note that this isn't about blocking i/o but about worker threads. If you have 20 items and 1 thread, then that thread is going to poll, backoff, and work sequentially.

Otherwise, sorry- have my hands a bit full. Happy to take a look at a reproducible example, otherwise I may suggest you do some more testing on your end to try identify possible causes.

:throttle-ms, :eoq-backoff-ms and :nthreads are the relevant mq options; beyond that would require some debugging.

uaalto commented 8 years ago

Thank you @ptaoussanis. I will try to create a reproducible test case!

uaalto commented 8 years ago

Hey @ptaoussanis. I did find the error and it was my code. I had a blocking operation that only shows up in testing. Thank you for your prompt reply!

ptaoussanis commented 8 years ago

No problem, happy you found a resolution :-)