alex-petrenko / faster-fifo

Faster alternative to Python's multiprocessing.Queue (IPC FIFO queue)
MIT License
179 stars 29 forks source link

Benchmark Update against multiprocess.Queue in Python 3.11 #45

Closed steve8210 closed 1 year ago

steve8210 commented 1 year ago

Can we have benchmark updated against default multiprocessor.Queue in Python 3.11?

I've done some my own experiment with Python 3.9 & Python 3.11 -- multiprocess.Queue was definitely slower in 3.9 but seems like it can be faster or as fast as faster-fifo on Python 3.11 under certain situations.

alex-petrenko commented 1 year ago

@steve8210 Interesting! Well, this would be amazing actually, glad they recognized it was a problem. Less code is always better. Let me run a quick check. I've heard a lot of performance improvements are coming to Python 3.11, glad they re-implemented the Queue as well. In 3.8 I saw 30x difference or even more in some configurations.

alex-petrenko commented 1 year ago

I ran some tests on Python 3.11 and while mp.queue definitely became faster it does not seem like faster-fifo is obsolete:

# Update (2023.08.11)
# Ubuntu 18, Intel(R) Xeon(R) CPU E5-2650, Python 3.9.16
# [2023-08-11 17:37:58,796][40648] Configuration (1, 1, 200000), timing [ff: 2.28s, ff_many: 2.46s, mp.queue: 3.48s]
# [2023-08-11 17:37:58,796][40648] Configuration (1, 10, 200000), timing [ff: 2.71s, ff_many: 2.82s, mp.queue: 11.65s]
# [2023-08-11 17:37:58,796][40648] Configuration (10, 1, 100000), timing [ff: 13.69s, ff_many: 1.95s, mp.queue: 18.39s]
# [2023-08-11 17:37:58,796][40648] Configuration (3, 20, 100000), timing [ff: 2.97s, ff_many: 2.30s, mp.queue: 21.10s]
# [2023-08-11 17:37:58,796][40648] Configuration (20, 3, 50000), timing [ff: 19.75s, ff_many: 1.08s, mp.queue: 23.24s]
# [2023-08-11 17:37:58,796][40648] Configuration (20, 20, 50000), timing [ff: 2.81s, ff_many: 3.73s, mp.queue: 70.49s]
# Ran 1 test in 206.923s

# Ubuntu 18, Intel(R) Xeon(R) CPU E5-2650, Python 3.11.4
# [2023-08-11 17:46:36,056][42634] Configuration (1, 1, 200000), timing [ff: 1.67s, ff_many: 1.77s, mp.queue: 2.45s]
# [2023-08-11 17:46:36,056][42634] Configuration (1, 10, 200000), timing [ff: 2.27s, ff_many: 2.31s, mp.queue: 5.61s]
# [2023-08-11 17:46:36,056][42634] Configuration (10, 1, 100000), timing [ff: 13.15s, ff_many: 1.86s, mp.queue: 14.05s]
# [2023-08-11 17:46:36,056][42634] Configuration (3, 20, 100000), timing [ff: 2.97s, ff_many: 2.23s, mp.queue: 12.71s]
# [2023-08-11 17:46:36,056][42634] Configuration (20, 3, 50000), timing [ff: 19.22s, ff_many: 0.99s, mp.queue: 17.28s]
# [2023-08-11 17:46:36,056][42634] Configuration (20, 20, 50000), timing [ff: 2.60s, ff_many: 3.52s, mp.queue: 43.30s]
# Ran 1 test in 149.972s

This might have something to do with the fact that my workstation has a very old Ubuntu 18. Feel free to run the comparison yourself and share the results! You can just run the test ComparisonTestCase from this file: https://github.com/alex-petrenko/faster-fifo/blob/b68fde311e80d06a1817450dea3e9ea986cdb395/cpp_faster_fifo/tests/comparison_tests.py#L178

alex-petrenko commented 1 year ago

I can see some configurations where mp queue is a little bit faster now when get_many() is not used. But there are still configurations where difference is massive (e.g. 20 consumers 20 producers)

steve8210 commented 1 year ago

Here is my result. Agree with your assessment above. mp.Queue has become efficient for few simplest situations (1 producer and 1 or more readers?) but faster-fifo has significant advantage under most scaled up situations.

Tested under python3.11.4 docker running under AWS c5a.12xlarge instance.

Results:

[2023-08-14 18:58:31,893][00012] Configuration (1, 1, 200000), timing [ff: 4.50s, ff_many: 4.38s, mp.queue: 4.08s]
[2023-08-14 18:58:31,893][00012] Configuration (1, 10, 200000), timing [ff: 4.51s, ff_many: 4.61s, mp.queue: 4.59s]
[2023-08-14 18:58:31,893][00012] Configuration (10, 1, 100000), timing [ff: 28.61s, ff_many: 1.93s, mp.queue: 23.17s]
[2023-08-14 18:58:31,893][00012] Configuration (3, 20, 100000), timing [ff: 6.30s, ff_many: 5.36s, mp.queue: 10.57s]
[2023-08-14 18:58:31,893][00012] Configuration (20, 3, 50000), timing [ff: 15.31s, ff_many: 2.07s, mp.queue: 32.70s]
[2023-08-14 18:58:31,893][00012] Configuration (20, 20, 50000), timing [ff: 4.67s, ff_many: 4.60s, mp.queue: 36.58s]
alex-petrenko commented 1 year ago

Thank you for sharing this!

Also, glad to see that Python 3.11 performance improvements are real!