Extract whole archives in some cases

[andres-asm]

Something like this may be desirable for some file types: https://hastebin.com/idedelakaj.cs

Basically there should be a list of extensions that trigger a whole archive extraction instead of just the selected file. It would be slow but it would solve a bunch of cases with the current code

I tried to implement it but the extraction function is quite complicated.

--- Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/40462238-extract-whole-archives-in-some-cases?utm_campaign=plugin&utm_content=tracker%2F296058&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F296058&utm_medium=issues&utm_source=github).

[andres-asm]

https://github.com/libretro/dosbox-libretro/issues/49

[andres-asm]

https://github.com/libretro/RetroArch/issues/4094

[andres-asm]

https://github.com/libretro/RetroArch/issues/2614

[RobLoach]

File extensions that would need this:

bat com exe game cue gdi toc wad m3u scummvm

[klepp0906]

Just chiming in hoping this is on the menu to be addressed. Otherwise I have to literally convert an entire Psx library.

I posted this here thinking it was a core issue considering Sega cd bin/ cue seem to extract and load fine. Psx not so much.

https://github.com/libretro/beetle-psx-libretro/issues/223

[SpaceAgeHero]

I've posted a bounty on this. :-) Please kindly also take into account my referenced request https://github.com/libretro/RetroArch/issues/2614 which includes multi-bin dumps as well.

[Ferk]

I honestly think that it would be a bad (and hopefully temporary) solution to simply unpack all the files in the zip.

Not all platforms have that kind of readily available storage, and imho it defeats the point of zipping the games if each game I want to play is gonna be unzipped into disk. And even if they were deleted afterwards it's a lot of strain for the storage, specially if you think about SD cards with Lakka and so.

Best would be if an VFS (#3715?) could read the files directly from the compressed file without dumping them to disk. That's how I initially thought the zip support in Retroarch worked, I was a bit disappointed when I discovered it's just unzipping them to disk, and it's not even in a temporary folder but directly next to the game, which might or might not be a writable location and which might never be cleaned by the OS in case they weren't deleted.

[ghost]

@Ferk What about zips with e.g. multi-disc games? People keep asking for this but I think it's clear we don't want to keep all that in RAM... but extracting it all every time would be very slow as well.

[klepp0906]

I just keep my seperate discs in seperate archives. I realize not everyone does it this way however.

Unzipping all to a temp directory same as everything else is the best solution I reckon.

[RobLoach]

In order to save RAM, disk space, and performance, extracting the whole archive should be opt-in by the user. It currently extracts the first file in the archive. The code is over at: https://github.com/libretro/RetroArch/blob/master/tasks/task_content.c#L454

[Ferk]

@bparker06 I don't think you need to extract the full file into RAM if you only want to read part of it. Not with the typical deflate algorithm used in zip files, and I expect the lzma from 7z would also allow uncompressing files partially (checking at the lib7z code I can see there are read and seek functions). This way after decompressed the data would take the same RAM the original read call would.

I assume a VFS would be meant to replace all the fopen calls in the core and it'd be able to uncompress on the fly big files as they are read, without keeping the entire uncompressed thing in RAM nor in disk. It might be slower than caching them on disk (if you have a fast disk and a slow cpu, otherwise it would be similar or perhaps faster depending on the compression method and the threading), but it will still be faster than the physical optical devices from actual consoles, it'd be a cleaner way to do it that doesn't mess up with the system and it'd be better supported in more platforms since you won't need to write to disk at all.

[SpaceAgeHero]

Any clue why my bounty https://www.bountysource.com/issues/40462238-extract-whole-archives-in-some-cases is not referenced? Wish this would get more attention...

[klepp0906]

I haven’t forgotten I tell ya that much. This is a huge deal. Amazed it’s still an issue this many years in.

Ty for the bounty btw <3 As soon as closing on my refinance goes through (unfortunately a few more weeks) ill add to the bounty. this being solved is a NEED lol.

On Nov 2, 2017, at 3:06 AM, SpaceAgeHero notifications@github.com wrote:

Any clue why my bounty https://www.bountysource.com/issues/40462238-extract-whole-archives-in-some-cases is not referenced? Wish this would get more attention...

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[Tatsuya79]

It would be really nice if it could already unzip everything to disk.

Use case: cores that read games from multiple floppies and have a GUI to load them (NP2Kai, PX68K). If you load a zip that gets entirely decompressed into a cache folder, then you just need those cores config path to point to that same folder to have the exact files you need usable in their GUI. That's something really neat as those GUI are particularly awful for browsing a huge amount of files. That works for MAME Software List sets as they are using multiple disks in a single archive.

I wonder if that simply could be an option in retroarch settings... core? "Extract everything from loaded archive." and if that could be used as a core override as well.

(Is that where the change needs to be made?)

[Ferk]

Wouldn't the VFS pull (#5664) together with PhysFS integration (RobLoach#3) allow any core supporting the VFS to access all the content of the zip files without having to extract anything at all to disk?

[i30817]

In order to save RAM, disk space, and performance, extracting the whole archive should be opt-in by the user.

Extracting (to disk) should be already optional (it would be the first thing i'd disable). But isn't for some reason.

Honestly, if you want my opinion, (which i'm sure no one does), RA should deprecate extracting compressed formats.

Either the emulator supports it (like mame and zips) and they're passed directly to the core, or RA can seek them on the virtual filesystem fast (which is currently basically only chd, very small zip files and maybe if RA gets special support for some variants of indexed zip) and nothing else but bare files.

This would get users to change to .chd (i'm assuming that current RA doesn't just extract them too even if they're seekable), or use windows/linux folder/filesystem compression and allow RA to simplify what sounds like extremely complicated and inefficient IO code.

But, since this isn't going to happen...