r2pipe Python is slow when processing large results

[jmattheus]

Environment

radare2 5.7.9 29270 @ linux-x86-64 git.5.7.8-352-g53b5d81100 commit: 53b5d81100957dc536fd895cabee8d26f184fe86 build: 2022-10-17__16:55:35 Linux x86_64 r2pipe 1.7.3 Python

Description

When processing a command with a large output, there is a significant delay. An example of one such command is: pDj 0x380a00

Test

Run some large command:

import r2pipe
r = r2pipe.open('/bin/ls')
# I'm sure there's a better way to loop, but this works
result = r.cmd('(alltheoutput;f cnt=0xb000;loop:;x @ `?vi cnt`;f cnt=`?vi cnt-1`;?= cnt;?!();.loop:);.(alltheoutput)')

Suggested fix

open_sync is using string concatenation to pipe the output. Using a bytearray significantly improves performance.

https://github.com/radareorg/radare2-r2pipe/blob/df5c399159cf3dc20d2aa32520d8a89ce9b53e1c/python/r2pipe/open_sync.py#L125 Currently the output variable is set up as out = b"" Changing that line to out = bytearray() requires no other changes and has a positive impact when parsing large outputs. On my machine, the test script listed above took 55 seconds to run. Conversely, after switching to a bytearray, it only took 13 seconds to run.

[trufae]

Press the pencil button make the change and press the submit pr button. Its better to discuss code changes as pullrequests, not to mention the fix will be under your name.

thats a nice catch btw :) crazy that such a little change can have such big impact

thanks!

[trufae]

I just commited your fix referencing your name and the issue :) hope that's fine for you

also confirmed the fix was correct and published a new build (1.7.4)

thank you!

[jmattheus]

I opened the issue planning to reference it in a PR, but got pulled into something else. Thanks for pushing it in.

[trufae]

Don't worry, i just didnt wanted to wait or put you in trouble in case you couldnt contribute or something. Your fix was great and i was happy to read your patch and reasoning. Kudos