tarantool / queue

Create task queues, add and take jobs, monitor failed tasks
Other
236 stars 52 forks source link

Introduce "shared" sessions #139

Closed LeonidVas closed 3 years ago

LeonidVas commented 3 years ago

Before the patchset a connection to server was synonym of the queue session. Now the session has a unique UUID (returned by the "queue.identify()" method), and one session can have many connections. To connect to an existing session, call "queue.identify(uuid)" with the previously obtained UUID. Also, for the queue the ttr option has been added. ttr in seconds - the time after which, if there is no active connection in the session, it will be released with all its tasks.

Closes #85

@ChangeLog: The "shared" sessions was added to the queue. Previously, a connection to server was synonym of the queue session. Now the session has a unique UUID (returned by the "queue.identify()" method), and one session can have many connections. To connect to an existing session, call "queue.identify(uuid)" with the previously obtained UUID. Also, for the queue the ttr option has been added. ttr in seconds - the time after which, if there is no active connection in the session, it will be released with all its tasks.

LeonidVas commented 3 years ago

The benchmark was run before and after adding "shared" sessions. The command that was used to run the test: rm -rf *.snap *.xlog && taskset 3 tarantool t/benchmark/multi_consumer_work.lua For each variant of the parameters, ten runs of tests were carried out, and after the result was averaged. Result format: (time it takes to fill the queue / time it takes to confirm the tasks) in microseconds.

Master:

consumers
batches
1 100 10000
1 102 / 393 5479 / 10681 65914 / 322383
100 7272 / 9107 49043 / 224925 4962018 / 33961217
10000 41834 / 220479 3972183 / 25542455

With "shared" sessions:

consumers
batches
1 100 10000
1 123 / 446 2776 / 12724 64437 / 509038
100 7900 / 8411 55467 / 274383 4899408 / 35677578
10000 54831 / 266131 4259797 / 30703081

In most cases, we have a loss of performance ~ 5 - 20%.

LeonidVas commented 3 years ago

After refactoring.

Master:

consumers
batches
200 400
200 132887 / 659421 268149 / 1490263
400 247721 / 1538678 480667 / 3362063

With "shared" sessions:

consumers
batches
200 400
200 136281 / 620819 259882 / 1426101
400 249045 / 1361905 490443 / 2613279
LeonidVas commented 3 years ago

Thanks for your patch. Consider several comments below. I'll give more detailed review a bit later.

Hi! Thanks for the review.

Also I think we could push d8de8b9 out of order to simplify patchset and reduce diff.

No, it's needed fo https://github.com/tarantool/queue/pull/139/commits/a3d6f4ea51df9b49c21415373e0f6677458188ea