mosquito / aio-pika

AMQP 0.9 client designed for asyncio and humans.
https://aio-pika.readthedocs.org/
Apache License 2.0
1.18k stars 186 forks source link

QueueIterator: speedup via __anext_impl w/wo timeout in __init__ #627

Open gglluukk opened 2 months ago

gglluukk commented 2 months ago

Hi!

During investigation of optimization RabbitMQ's reader I've faced up performance gap between aio-pika 6.6.1 and 6.7.0 (benchmarks for 10,000 reads/seconds):

aio-pika 6.8.2: 4.484
aio-pika 6.8.1: 4.374
aio-pika 6.8.0: 4.309
aio-pika 6.7.1: 4.279
aio-pika 6.7.0: 4.436 <--- gap
aio-pika 6.6.1: 4.013
aio-pika 6.6.0: 4.083
aio-pika 6.5.3: 4.098

More: https://github.com/gglluukk/rmq2psql

The performance gap arose due to the addition of timeout support in queue.iterator(), introduced in version 6.7.0:

# diff aio_pika-6.6.1/queue.py aio_pika-6.7.0/queue.py 
439c439,442
<             return await self._queue.get()
---
>             return await asyncio.wait_for(
>                 self._queue.get(),
>                 timeout=self._consume_kwargs.get('timeout')
>             )

This change caused a decrease in performance compared to version 6.6.1 and subsequent versions inherited this gap.

PR can hopefully resolve this.

Regards!

Darsstar commented 2 months ago

How does #615 do in this regard?

gglluukk commented 2 months ago

@Darsstar my custom task was to speed up the particular code (cancellation/connection check there is done by its own logic). during that i've researched the speed degradation and proposed patch. decision will it be accepted or not is out of my responsibility. btw i've benchmarked your patch and it turns to be ~10% slower:

before your patch:
rmq2psql.py --loop-type queue_iteration_with_timeout --max-bulks 100: #1: 4.905
rmq2psql.py --loop-type queue_iteration_with_timeout --max-bulks 100: #2: 4.503
rmq2psql.py --loop-type queue_iteration_with_timeout --max-bulks 100: #3: 4.809
rmq2psql.py --loop-type queue_iteration_with_timeout --max-bulks 100: #4: 4.870
rmq2psql.py --loop-type queue_iteration_with_timeout --max-bulks 100: #5: 4.741
Average: 4.766

after your patch:
rmq2psql.py --loop-type queue_iteration_with_timeout --max-bulks 100: #1: 5.481
rmq2psql.py --loop-type queue_iteration_with_timeout --max-bulks 100: #2: 5.123
rmq2psql.py --loop-type queue_iteration_with_timeout --max-bulks 100: #3: 5.397
rmq2psql.py --loop-type queue_iteration_with_timeout --max-bulks 100: #4: 5.437
rmq2psql.py --loop-type queue_iteration_with_timeout --max-bulks 100: #5: 5.128
Average: 5.313
coveralls commented 2 months ago

Coverage Status

coverage: 87.694% (-0.4%) from 88.125% when pulling 4a89ff06e7989eb1aeba307cdf5cde88e4a3402c on gglluukk:master into 848c0250045811c7598123b46a1d3dda2745c905 on mosquito:master.