Feature: blit + sd data bundles

[ali1234]

This is a proposal to extend the blit format to handle extra game data files which should be installed on the SD card.

Blit files can have extra data appended after #785. The questions then are how should the system handle locating the data and how should it work for users?

Ideally installing a bundle blit should not be any different to a flash-only blit. Installing with either the tools or copying to SD card and running it from the launcher should both work. The tools will need to be extended to refuse to install a bundle if no SD card is available in the target.

The game will need to locate the extra data when running from flash. I can see three ways of doing that:

1. Store the path to the bundle dynamically in the metadata during install.
2. Require the user to always launch the blit from the SD card and hide it from the internal flash menu.
3. Unpack the extra data into the SDK game data path (currently only used for saves).

Of these options, only 3 requires no extra metadata. It also allows compression and directory trees to be unpacked during installation. Option 2 has the benefit that bundles with unavailable data won't show up anywhere, but requires the user to hunt on the SD card for the bundle every time they want to run it.

I propose the following:

1. Install the blit to flash as normal.
2. If there is extra data, extract it to the SDK save path eg /<author>/<blitname>/gamedata/.
3. Add a function to the sdk to open a filename from the gamedata directory without needing to know its path.
4. Show an error if the gamedata is not available when attempting to launch the blit from the flash menu.

[Daft-Freak]

I'm assuming the SDL-side handling of this is basically "copy files to build/install dir". The function to open these could possibly just be another open flag like the "cache" one.

I guess the big part is what format to use for the "bundle" and a tool to make those. Possibly the install command would need to handle unpacking itself if the dest is flash as stream-unpacking does not sound fun... or it could just not allow that and force installs through the SD card as you need it anyway.

[Daft-Freak]

:thinking: Some of this could probably be reused to help with the lua/interpreted games packaging thing.

[ali1234]

Hmm... I didn't consider SDL. I suppose there the data path would just be alongside the executable in the output zip, as you say.

[Gadgetoid]

I love this as a concept and totally see where it would be useful - ignoring the implementation details your proposal as laid out seems pretty reasonable.

Over on the RP2040/Pico side I accomplished... something?... like this by packing up a manifest of files into a LittleFS filesystem that gets appended onto the firmware .uf2 (https://github.com/Gadgetoid/dir2uf2/). Yeah it's not the same but... files... in places.

I guess zlib and gzip are the obvious answers, but then we'd have to roll some kind of filesystem metadata ourselves since I think a bunch of individual compressed files with the FNAME set, cat'd together wouldn't compress as efficiently as a single data stream?

I wonder if there are any existing, simple filesystems that would be amenable to this kind of tomfoolery without being zero-padded to oblivion.

• What about EMScripten/Web Assembly?
• Is compression even really required?, I guess it could be optional
• RP2040 ports technically kinda sorta have (4MB iirc) storage, even if it isn't an SD card
• However we accomplish this it should probably also work pretty much exactly the same way for files appended to .blits that are small enough to just get wholesale flashed to XiP
• Something something updates

[Daft-Freak]

Oh, meant to look into this more, but uh guess what happened :laughing:

• Emscripten has it's own embedded file thing, probably reusable in some way for this.
• If we're also allowing just appending files that don't get extracted, then I'd want those to not be compressed (as this is what I'm doing with my messy scripts for daftboy/doom)
• Some of the RP2040 ports might get SD card support if I ever finish the code... though there isn't a launcher over there anyway (also on the list...)

[Gadgetoid]

Regarding the format, MicroPython have a simple, linear filesystem that can be memory mapped in the works here - https://github.com/micropython/micropython/pull/8381

They also have a mpremote deploy-mapfs <directory> which packs up a local directory into this format.

Short of potentially adding optional zlib compression on top of this, this might be more or less what we want?

[Daft-Freak]

Definitely a simple format. Looks like

MF

uint16 name len + dir flag
uint32 data len
name
data

... more files

null terminator

vs my

APPFILES
uint32 num files

uint16 name len
uint32 data len
[... for each file]

name
data
[... for each file]

... except mine doesn't represent dirs.

The only problem with compression is that it would be the firmware doing the decompression... so not much space. (Unless I decide to go chopping giant chunks of the HAL off again)