zip extraction of many small files is very very slow

[rsn8887]

I frequently run into this problem: extracting a zip file with many small files on Vitashell is extremely slow (~5 kB/s). It is basically unusably slow. Extracting 10k files from a 70 MB .zip takes >5 hours!!! A single large file extracts quickly.

Is there a workaround or any other way to speed it up?

[fexofenadine]

Definitely happens with .7z files as well, for instance the [date]_Retroarch_data.7z files that build with the RetroArch nightlies. These take literally hours to unpack

[dude22072]

This is simply an issue with the format, not with VitaShell. Even on a powerful modern computer, one large file may extract at 40MB/s, but dozens of small files will be significantly slower, usually less that 1MB/s, even within the same zip file. ZIP files are compressed data, compressed by an algorithm, but with how little space it saves people often forget that. For every file, the decompression algorithm must be run, and it has to be done individually.

7z files are even worse, because 7z's algorithms can actually knock off a significant portion of the data. A portion that, when extracted, has to be remade.

If this were my project I'd close this issue.

[rsn8887]

You managed to write two paragraphs without explaining why this is slow.

My old Cyrix 166 MHz PC in 1996 could already inflate zip blazingly fast.

What exactly is slowing this down with many small files? Are you saying it is some slow implementation of the inflate algorithm, which could be fixed by maybe using some optimized assembler code or something?

Or are you saying it is a problem with slow I/O on the storage medium? In the latter case maybe a larger I/O buffer could be used so that many small files are decompressed to memory first and then written more efficiently.

I tried extracting some 80 MB zip file with 10,000 small files in Vitashell and it took longer than 15 hours and eventually I just quit the process. Clearly something is wrong because the Vita is not THAT slow (neither CPU nor I/O). Maybe the code used is just not optimized?

[TheOfficialFloW]

@rsn8887 Can you try changing the block size and see if it improves anything?: https://github.com/TheOfficialFloW/VitaShell/blob/master/archive.c#L59

[rsn8887]

I found out today that it not the unzipping of many small files, even just copying is slow. I unzipped my >10,000 small files on my PC and copied them to my SD card as separate files.

I then used Vitashell to simply copy the files from SD to Memory Card, and it was just as slow as the zip extraction.

[yiays]

I'm running into this issue with VitaShell 2.02, the RetroArch data and VPK both appear to basically not be extracting at all (0-1KB/s), they both start strong then drop to this speed. Seems to be the case regardless of whether I'm extracting and installing to the same memory card or extracting from one to the other.

[Danixu]

Same issue here but with an vpk which contains big files. I have tried a vpk with no compression and the speed is the same.

[Masamune3210]

Not really a way to help this, file operations involving lots of small files are inherently slow even on PC. It's just how things are unfortunately

[LilCiro]

I made a video about it, link https://www.youtube.com/watch?v=tyDL5xHf-3Q&ab_channel=LilCiro

[kimzeevaarders]

Hi, guys, I ran into this problem and managed to work around it by already extracting the .vpk file on my pc and then FTP the extracted content dir to my mounted USB (ux0:). In vita shell you can then select the extracted folder and select more --> install folder.

Installation is now a hell of a lot faster!