tarantool / queue

Create task queues, add and take jobs, monitor failed tasks
Other
234 stars 52 forks source link

tube: fix slow take on busy utubes #229

Closed DerekBum closed 2 months ago

DerekBum commented 3 months ago

If some of the utubes for tasks at the start of the queue were busy most of the time, take would slow down for every other task. This problem is fixed by creating a new space space_ready. It contains first task with READY status from each utube.

This solution shows great results for the stated problem, but with the cost of slowing the put method. Thus, this workaround is disabled by default. To enable it, user should set the storage_mode = "ready_buffer" as an option while creating the tube (or queue.driver.utube.STORAGE_MODE_READY_BUFFER for utube and queue.driver.utubettl.STORAGE_MODE_READY_BUFFER for utubettl). As example:

local test_queue = queue.create_tube('test_queue', 'utube',
        {temporary = true, storage_mode = queue.driver.utube.STORAGE_MODE_READY_BUFFER})

Or

local test_queue = queue.create_tube('test_queue', 'utubettl',
        {temporary = true, storage_mode = queue.driver.utubettl.STORAGE_MODE_READY_BUFFER})

Also added two benchmarks to compare storage_mode = "default" and storage_mode = "ready_buffer" (storage_mode = "default" is default and disables the workaround).

  1. Benchmark for simple put and take methods. 30k utubes are created with single task each. Task creation time is calculated. After that 30k consumers are calling take + ack, each in the separate fiber. Time to ack all tasks is calculated. The results are as follows (for comparison I also used fifo tube):
put (30k) take+ack
default 180ms 1.6s
ready 270ms 1.7s

As we can see, new utube implementation has slower put method.

  1. Benchmark for the stated problem. 10 tubes are created. Each contains 1000 task. After that 10 consumers are created (each works on his tube only, one tube -- one consumer). Each consumer will take, then yield and then ack every task from their utube (1000 tasks each). After that we can also run this benchmark with 10k tasks on each utube, 100k tasks and 150k tasks. But all that with 10 utubes and 10 consumers. The results are as follows:
1k 10k 50k 150k
default 53s 1.5h 100h 1000h
ready 450ms 4.7s 26s 72s

We can see great time performance improvement.

Same change was also made for utubettl. Here is benchmarks results for utubettl (on the same benchmarks):

put (30k) take+ack
default 200ms 1.7s
ready 320ms 1.8s
1k 10k 50k 140k
default 80s 1.6h 100h 1000h
ready 520ms 5.4s 28s 83s

(note that for busy tasks were used 140k and not 150k tasks).

Closes #228

DerekBum commented 3 months ago

Right now only utube is fixed. utubettl will follow soon.

DerekBum commented 2 months ago

Added fix for utubettl in a different commit.

oleg-jukovec commented 2 months ago

@DifferentialOrange @better0fdead this is a non-trivial fix, we need additional reviews with a clear head here.

DerekBum commented 2 months ago

Updated the code according to comments. Added links to the new issue https://github.com/tarantool/queue/issues/230 about implementing new storage modes for vinyl engine.

DerekBum commented 2 months ago

Updated the code according to the comments.

DerekBum commented 2 months ago

Updated the code according to comments.