gmr / rabbitpy

A pure python, thread-safe, minimalistic and pythonic RabbitMQ client library
http://rabbitpy.readthedocs.org
BSD 3-Clause "New" or "Revised" License
242 stars 58 forks source link

Faster message body retrieval #104

Closed JelleAalbers closed 7 years ago

JelleAalbers commented 7 years ago

Hi rabbitpy maintainers,

I'm using rabbitpy in a project (https://github.com/XENON1T/pax/pull/439) to pass around large (~20 MB) messages. I noticed receiving these messages was slow, and profiling revealed rabbitpy spent a lot of its time in this line: https://github.com/gmr/rabbitpy/blob/master/rabbitpy/channel.py#L486

_wait_for_content_frames is assembling the content of the body in a string/bytes, adding every frame to it with +=. Since strings/bytes are immutable, this requires a new string/bytes to be created every time, causing an N^2 growth of memory allocation work with the number of frames.

The recommended idiom for concatenating multiple strings/bytes is to use bytearrays (if you're in python 3) or to construct a list first and use ''.join. This pull request changes _wait_for_content_frames to use these methods (bytearray for py3, join on list of chunks in py2).

With this change, receiving large messages went 10x faster in my application (using python 3, I did not test in python 2). Let me know if you would consider this addition. Thanks for creating rabbitpy!

codecov-io commented 7 years ago

Current coverage is 72.10% (diff: 100%)

Merging #104 into master will decrease coverage by 0.21%

@@             master       #104   diff @@
==========================================
  Files            17         17          
  Lines          1875       1882     +7   
  Methods           0          0          
  Messages          0          0          
  Branches        279        281     +2   
==========================================
+ Hits           1356       1357     +1   
- Misses          398        406     +8   
+ Partials        121        119     -2   

Powered by Codecov. Last update 5b6d55d...eec83d2

gmr commented 7 years ago

Thanks, looks good!