Closed steve8210 closed 1 year ago
@steve8210 Interesting! Well, this would be amazing actually, glad they recognized it was a problem. Less code is always better. Let me run a quick check. I've heard a lot of performance improvements are coming to Python 3.11, glad they re-implemented the Queue as well. In 3.8 I saw 30x difference or even more in some configurations.
I ran some tests on Python 3.11 and while mp.queue definitely became faster it does not seem like faster-fifo
is obsolete:
# Update (2023.08.11)
# Ubuntu 18, Intel(R) Xeon(R) CPU E5-2650, Python 3.9.16
# [2023-08-11 17:37:58,796][40648] Configuration (1, 1, 200000), timing [ff: 2.28s, ff_many: 2.46s, mp.queue: 3.48s]
# [2023-08-11 17:37:58,796][40648] Configuration (1, 10, 200000), timing [ff: 2.71s, ff_many: 2.82s, mp.queue: 11.65s]
# [2023-08-11 17:37:58,796][40648] Configuration (10, 1, 100000), timing [ff: 13.69s, ff_many: 1.95s, mp.queue: 18.39s]
# [2023-08-11 17:37:58,796][40648] Configuration (3, 20, 100000), timing [ff: 2.97s, ff_many: 2.30s, mp.queue: 21.10s]
# [2023-08-11 17:37:58,796][40648] Configuration (20, 3, 50000), timing [ff: 19.75s, ff_many: 1.08s, mp.queue: 23.24s]
# [2023-08-11 17:37:58,796][40648] Configuration (20, 20, 50000), timing [ff: 2.81s, ff_many: 3.73s, mp.queue: 70.49s]
# Ran 1 test in 206.923s
# Ubuntu 18, Intel(R) Xeon(R) CPU E5-2650, Python 3.11.4
# [2023-08-11 17:46:36,056][42634] Configuration (1, 1, 200000), timing [ff: 1.67s, ff_many: 1.77s, mp.queue: 2.45s]
# [2023-08-11 17:46:36,056][42634] Configuration (1, 10, 200000), timing [ff: 2.27s, ff_many: 2.31s, mp.queue: 5.61s]
# [2023-08-11 17:46:36,056][42634] Configuration (10, 1, 100000), timing [ff: 13.15s, ff_many: 1.86s, mp.queue: 14.05s]
# [2023-08-11 17:46:36,056][42634] Configuration (3, 20, 100000), timing [ff: 2.97s, ff_many: 2.23s, mp.queue: 12.71s]
# [2023-08-11 17:46:36,056][42634] Configuration (20, 3, 50000), timing [ff: 19.22s, ff_many: 0.99s, mp.queue: 17.28s]
# [2023-08-11 17:46:36,056][42634] Configuration (20, 20, 50000), timing [ff: 2.60s, ff_many: 3.52s, mp.queue: 43.30s]
# Ran 1 test in 149.972s
This might have something to do with the fact that my workstation has a very old Ubuntu 18. Feel free to run the comparison yourself and share the results!
You can just run the test ComparisonTestCase
from this file: https://github.com/alex-petrenko/faster-fifo/blob/b68fde311e80d06a1817450dea3e9ea986cdb395/cpp_faster_fifo/tests/comparison_tests.py#L178
I can see some configurations where mp queue is a little bit faster now when get_many() is not used. But there are still configurations where difference is massive (e.g. 20 consumers 20 producers)
Here is my result. Agree with your assessment above. mp.Queue has become efficient for few simplest situations (1 producer and 1 or more readers?) but faster-fifo has significant advantage under most scaled up situations.
Tested under python3.11.4 docker running under AWS c5a.12xlarge instance.
Results:
[2023-08-14 18:58:31,893][00012] Configuration (1, 1, 200000), timing [ff: 4.50s, ff_many: 4.38s, mp.queue: 4.08s]
[2023-08-14 18:58:31,893][00012] Configuration (1, 10, 200000), timing [ff: 4.51s, ff_many: 4.61s, mp.queue: 4.59s]
[2023-08-14 18:58:31,893][00012] Configuration (10, 1, 100000), timing [ff: 28.61s, ff_many: 1.93s, mp.queue: 23.17s]
[2023-08-14 18:58:31,893][00012] Configuration (3, 20, 100000), timing [ff: 6.30s, ff_many: 5.36s, mp.queue: 10.57s]
[2023-08-14 18:58:31,893][00012] Configuration (20, 3, 50000), timing [ff: 15.31s, ff_many: 2.07s, mp.queue: 32.70s]
[2023-08-14 18:58:31,893][00012] Configuration (20, 20, 50000), timing [ff: 4.67s, ff_many: 4.60s, mp.queue: 36.58s]
Thank you for sharing this!
Also, glad to see that Python 3.11 performance improvements are real!
Can we have benchmark updated against default multiprocessor.Queue in Python 3.11?
I've done some my own experiment with Python 3.9 & Python 3.11 -- multiprocess.Queue was definitely slower in 3.9 but seems like it can be faster or as fast as faster-fifo on Python 3.11 under certain situations.