search

rommapp / romm

A beautiful, powerful, self-hosted rom manager
https://romm.app
GNU Affero General Public License v3.0
2.37k stars 96 forks source link

feat: Add streaming support for 7zip hashing #1282

Closed adamantike closed 1 week ago

adamantike commented 1 week ago

At the moment, 7zip files are generating memory issues and even OOM errors on user installations. This is because the current stable release of py7zr does not support decompression streaming, and RomM needs to decompress the each 7zip file in the library into memory to be able to calculate hashes.

This change introduces a py7zr fork I created to have a stable commit SHA to refer to in case upstream gets any forced pushes. It includes the contents of the pull request the py7zr creator is working on to support decompression streaming [1].

The way decompression streaming is implemented in py7zr is different than the other compression utilities. Instead of being able to provide a bytes iterator, we need to provide a Py7zIO implementation that will call a callback on each read and write operation.

[1] https://github.com/miurahr/py7zr/pull/620

github-actions[bot] commented 1 week ago

Test Results

88 tests  ±0   88 ✅ ±0   25s ⏱️ -1s  1 suites ±0    0 💤 ±0   1 files   ±0    0 ❌ ±0 

Results for commit 477d9b17. ± Comparison against base commit 277470b7.

:recycle: This comment has been updated with latest results.