redballoonsecurity / ofrak

OFRAK: unpack, modify, and repack binaries.
https://ofrak.com
Other
1.82k stars 128 forks source link

Use aiofiles for filesystem IO #466

Open ANogin opened 1 month ago

ANogin commented 1 month ago

One sentence summary of this PR (This should go in the CHANGELOG!) Use aiofiles for filesystem IO

Link to Related Issue(s)

Please describe the changes in your request. Replace blocking IO for file read/write with async io using aiofiles

Anyone you think should look at this, specifically? Not sure

ANogin commented 1 month ago

No longer sure this would make things faster rather than slower, marking as draft for now.

I did a timing test of a gather of 2x (create_root_resource_from_file + unpack_recursively(do_not_unpack=[Elf]) + auto_run_recursively(all_identifiers=True)) on TinyCore iso 14 and 15 respectively. Here are the results

feature/aiofiles             114.38s wall   80.99s CPU   873.3ms async select   32.52s blocked
feature/aiofiles             113.00s wall   82.09s CPU   903.2ms async select   30.01s blocked
feature/aiofiles             120.63s wall   86.49s CPU  1.01s    async select   33.13s blocked
feature/aiofiles             120.12s wall   85.63s CPU   879.3ms async select   33.61s blocked
feature/aiofiles             118.97s wall   84.46s CPU   876.1ms async select   33.63s blocked

master f3e583c               108.79s wall   76.34s CPU   146.6ms async select   32.31s blocked
master f3e583c               111.36s wall   77.90s CPU   151.1ms async select   33.31s blocked
master f3e583c               109.05s wall   76.93s CPU   146.9ms async select   31.98s blocked
master f3e583c               113.03s wall   78.98s CPU   144.8ms async select   33.91s blocked
master f3e583c               112.77s wall   79.39s CPU   130.9ms async select   33.25s blocked

where I used the approach from https://gist.github.com/vxgmichel/620eb3a02d97d3da9dacdc508a5d5321 to break the non-CPU time into “select” vs “blocked IO”.

ANogin commented 1 month ago

With #472 merged into master and into this branch:

master                       27.35s wall   26.99s CPU   124.0ms async select   244.7ms blocked
master                       26.97s wall   26.55s CPU   125.3ms async select   293.7ms blocked
master                       26.90s wall   26.52s CPU   120.5ms async select   265.8ms blocked
master                       27.68s wall   27.29s CPU   123.3ms async select   267.2ms blocked
master                       26.91s wall   26.51s CPU   131.1ms async select   267.9ms blocked

feature/aiofiles             30.29s wall   31.06s CPU   951.7ms async select   0.0ms blocked
feature/aiofiles             29.69s wall   30.39s CPU   1.02s async select     0.0ms blocked
feature/aiofiles             29.87s wall   30.59s CPU   910.6ms async select   0.0ms blocked
feature/aiofiles             29.66s wall   30.44s CPU   948.8ms async select   0.0ms blocked
feature/aiofiles             29.65s wall   30.45s CPU   922.8ms async select   0.0ms blocked

(the "0ms blocked" is an underapproximation)