Support for compressed ROMs?

tomvanbraeckel commented 4 months ago

On the Fri3D Camp 2024 Badge we have very limited internal flash storage space, so I'm looking into compressed of ROMS. It has an SD card reader but an SD card won't be provided along with the hardware.

So I was wondering, does retro-go have support for compressed ROMs? Or, since gw-emulator seems to have it, which parts do and which don't?

It seems the upstream original has support for compressed roms: https://github.com/OtherCrashOverride/go-play/blob/master/gnuboy-go/components/gnuboy/loader.c

But I can't find this in the current repo... did it get lost, or does it need a resync with upstream, or...?

Thank you!

tomvanbraeckel commented 4 months ago

This one seems recent and even has xz decompression: https://github.com/rofl0r/gnuboy

ducalex commented 4 months ago

Crashoverride's go-play never had zip support. Those lines are just remnants from the original Gnuboy but if you check the build flags it's disabled.

Retro-go did have zip support at one point, though. But I removed it because:

Bigger roms would fail to load (you need enough memory to fit the compressed stream and decompressed file. at least with miniz, quite possible that zlib allows us to chunk it)
I regularly had weird crashes or failures (again, likely miniz's fault because it affected PNGs as well before I switched to lode)
It breaks my bank-switching in GBC that allows loading 4/8MB roms (don't know if that's easily fixable without causing very obvious in-game stalls)
It breaks CR32 cover art (can be fixed I suppose)

I'm open to adding support back if you want to do it, assuming adding zlib+unzip doesn't add absurd bulk to the sizes.

I could only find the function prototype to give you a starting point, I guess files have been renamed since the code existed and vscode's timeline isn't helpful...

/**
 * Extracts one file from a zip archive to a new buffer. If `filter` is NULL, the first file found is selected. 
 * Otherwise the first file whose `stristr(name, filter) != NULL` is selected. The first byte of filter can be 
 * used to define anchoring if you want: =: exact match, $: ends with, ^: starts with. Example: "$.gbc".
 */
bool rg_storage_unzip_file(const char *zip_path, const char *filter, void **data_out, size_t *data_len);

ducalex commented 4 months ago

This one seems recent and even has xz decompression: https://github.com/rofl0r/gnuboy

Neat I didn't know others were working on gnuboy. Maybe they fixed some issues I've been having. And maybe I fixed some issues they still have, will have to have a look :)

Even though I've had issues with miniz, I'm ok if you want to just try and reuse their miniz/pkunzip code for a first prototype, we'll see how it works. (just put that in libretro-go so that all apps can access it, not in gnuboy)

ducalex commented 4 months ago

I should also mention that parts of miniz is built in rom on the esp32 (include rom/miniz.h) and at first glance their pkunzip only uses tinfl_decompress_mem_to_heap which is in available to us, so no increased program size yay!

ducalex commented 4 months ago

I was bored so I implemented an unzip function here: https://github.com/ducalex/retro-go/commit/2208256a1697c49051b8ce02f7d6299bc9c9207d

But before using it it would be worth checking if adding the complete miniz and use its zip API would be better.

tomvanbraeckel commented 4 months ago

Woah, here's hoping you'll be bored more often! ;-)

I'm not very familiar with of miniz, pkunzip, deflate, zlib and how you envision things so just to clarify:

1) how would compressed files be identified (initially)? Some ideas: A) based on their extension (say .gbc.zip or .gbc.gz) ? B) based on "magic tests", similar to what the "file" command does? C) something else?

2) which compressed file formats would be supported (initially)? Some ideas: A) https://wikipedia.org/wiki/ZIP_(file_format) ? I've seen some ROM packs that contain .zip files, but a zip file can contain much more files than just a single ROM so that brings the challenge of how to identify which file is the actual ROM. Abis) or, on a higher level, the full contents of the .zip file could be added to the list of ROMs in the launcher, and then choosing the actual ROM is up to the user. That would be nice but maybe slower (need to list the contents of the zip to show in the user interface) and more complex to implement. B) https://wikipedia.org/wiki/Gzip file format, which contains only a single file, making it a no-brainer to identify the actual ROM. Simple, maybe a good place to start. C) something else?

ducalex commented 4 months ago

I'm not very familiar with of miniz, pkunzip, deflate, zlib:

zlib has a neat API that replaces stdio functions and detects if the file is compressed or not. All you have to do is #define fopen/fread to gzopen/gzread, no change to the code needed. That would have been a no-brainer starting point, but zlib is pretty big and gzseek is very slow (because it has to re-read the stream) and it uses a lot of memory and I think supporting .zip is important.

So I think the best strategy will be to only support decompressing the entire file at once. This will be a problem for GBC but we'll cross that bridge when we get there.

which compressed file formats would be supported (initially)?

For formats I think (pk)zip is the best format because all users know how to make a zip. I did start writing a gzip parser (stopped short of extended headers) so we can also support .gz. I don't think I want to support more formats or compression algorithms for now.

As for strategy in multi-file zips, right now I was thinking take the first file that matches a filter (file extension) and if none matches, just take the first file in the archive and try it. Simple and effective. I like your idea of showing all the files in the launcher but I think it would be way too slow. Might be interesting to try it anyway, though, I'm often wrong about things :)

tomvanbraeckel commented 4 months ago

That experimental zip support is very cool! I'm tempted to merge it as-is because my deadline for the Fri3D Camp 2024 retro-go delivery is beginning next week.

ducalex commented 4 months ago

Let me know how it work out! With your amount of RAM it's likely going to work perfectly.

On the ESP32 there are still issues when trying to load a zipped ROM that is over 2MB or so, that's the only reason I haven't merged it yet. I want to reduce memory usage of rg_storage_unzip_file.

But if you tell me it works great for you then I guess I could silently merge it and just not advertise zip support until I fixed the memory issues :)

ducalex commented 4 months ago

ZIP support has been merged to the dev branch! All applications except MSX and DOOM support it. For now I have no plan to add zip support to them.

The main issue remaining is that large ROMs will fail to load (at least with 2/4MB PSRAM) so I'm not going to advertise ZIP support yet. But I should be able to fix that soon.

tomvanbraeckel commented 4 months ago

You sir, are amazing!!!

I was able to include around 10MB of homebrew ROMs plus the full 4MB Doom v1.9 on a VFS FAT partition of 6.43MB and it's using just 4.7MB of the space. So there's even 1.7MB of free space still available!

tomvanbraeckel commented 4 months ago

All applications except MSX and DOOM support it. For now I have no plan to add zip support to them.

Regarding DOOM, may I ask why there are no plans to add zip support for them? I see clearly how they would benefit (making WAD files half the size) and I don't see the downside, other than devices with low RAM not supporting it, but then they will just have to use non-zipped WADs, as they do today... or am I missing something?

But I should be able to fix that soon.

Curious to hear more about that fix! Transparent/streaming decompression maybe?

ducalex commented 4 months ago

Curious to hear more about that fix! Transparent/streaming decompression maybe?

Just inflating in chunks instead of loading the entire compressed stream to memory. I've already implemented it in a5cc490275c59399bde75e1b2b3414fe8b8ab477, so larger ROMs now work.

GBC is still a problem with 4/8MB ROMs because even uncompressed ROMs can't fit all at once in RAM, that's why I made a dynamic paging/banking system. Which of course won't work with ZIP. I can't think of a good way to fix that problem, so for now 4/8MB GBC ROMs are no-go with ZIP.

Regarding DOOM, may I ask why there are no plans to add zip support for them? I see clearly how they would benefit (making WAD files half the size) and I don't see the downside, other than devices with low RAM not supporting it, but then they will just have to use non-zipped WADs, as they do today... or am I missing something?

You're not missing anything, it just doesn't apply to any of our main targets and I can't test it myself. The real solution for DOOM would be transparent block-based compression. Because keeping the entire WAD in memory will not work for DOOM/DOOM2, even if you had 16MB of PSRAM.

I'll take your opinion into consideration and I will think about merging ZIP support to DOOM. You are right that it doesn't add much of a burden on me and has no impact on users!

ducalex commented 3 months ago

I've added zip support to doom (https://github.com/ducalex/retro-go/commit/2c0e10b94f74b0910b51287710ebc801bd6d4778 and https://github.com/ducalex/retro-go/commit/f659d4709eb5ba0c72690b82f1731e9b702cc648) but I'm not able to test it.

Please check if it works and if so you can close this issue!

tomvanbraeckel commented 3 months ago

Just tested the dev branch, works great, thank you so much!!!

ducalex / retro-go

Support for compressed ROMs? #132