SDL3 Filesystem Subsystem wishlist

[icculus]

Working on something adjacent to this, so I'm writing it here while I'm submerged in the muck:

The RWops and filesystem components could be merged into a new filesystem API, designed less around wrapping fopen and more on abstracting "containers" - that is, most platforms now have APIs centered on the concept of "title" containers and "storage" containers; title containers are the read-only filesystem for the application content, and storage containers are for save/config data created by the application at runtime. Storage containers are notable because you're actually meant to have multiple containers, specifically for multiple user accounts. Right now we depend on the PC doing this via OS accounts, but this may not be applicable for platforms like Steam Deck which use a single OS account with multiple Steam accounts, and it definitely isn't applicable on consoles where each controller will have some kind of account associated with it (be it a full account or a "guest" account).

A fully redesigned filesystem API that exposes such OS APIs is important for a number of reasons:

• It would fix a ton of issues in one go, i.e. #1374 #1214 #3121
• It would allow for significantly improved console support (right now it's either "have a single-player game on a platform that magically makes accounts vanish" or roll your own filesystem API from scratch, for every application)
• It would also allow for us to better support container APIs now found on PC. The notable examples are Xbox Live storage and SteamRemoteStorage, which apps currently implement totally separate from normal PC storage, assuming it's implemented at all (most of the time, something like Steam autocloud is used instead). If Steam ever exposes multi-account support (which I've asked for a number of times, and Deck may add pressure here), traditional filesystem APIs would not be able to handle this well (if at all). If Valve's software patent actually becomes a product beyond Fossilized shaders/media, this is where the title container idea would likely come in.

At least on my end, I'm familiar with this design because of XNA, where we have a storage namespace to reimplement in addition to a "TitleContainer":

https://github.com/FNA-XNA/FNA/tree/master/src/Storage https://github.com/FNA-XNA/FNA/blob/master/src/TitleContainer.cs

All content reading is meant to be done from TitleContainer (a requirement for Xbox 360), and saves are done by selecting a storage device (typically based on the player index and an associated user account), opening the container, then closing the container when finished (which then flushes/commits the changes). It's not perfect but it would probably be my starting point if I were writing this myself.

Originally posted by @flibitijibibo in https://github.com/libsdl-org/SDL/issues/3519#issuecomment-1192900354

[sezero]

Not a wishlist item, but an old issue: Especially now that SDL_RWFromFP is gone, please make sure https://github.com/libsdl-org/SDL/issues/4026 is now a non-issue.

[icculus]

please make sure https://github.com/libsdl-org/SDL/issues/4026 is now a non-issue.

I've reopened it so we don't forget.

[sezero]

please make sure #4026 is now a non-issue.

I've reopened it so we don't forget.

Wel, closed it again. It really looks like a non-issue today.

[sezero]

Possible wishlist: Maybe a callback based RW with user-supplied read, seek, tell, etc function pointers? If that'd be interesting for many, of course.

[slouken]

The intent is for you to just be able to instantiate your own SDL_rwops object, fill in the pointers, and pass it to any API that uses SDL_rwops. That's why the structure definition is public.

[sezero]

OK

[nfries88]

any chance we can get query functions for user media directories (ie ~/music/) too while we're at it?

[flibitijibibo]

Working with GDK storage this week and it turns out what they have for save storage is still pretty much what XNA's looked like, so will post it here since I will likely be making something very similar to this:

https://learn.microsoft.com/en-us/gaming/gdk/_content/gc/reference/system/xgamesave/xgamesave_members

[icculus]

Started up a separate bug for concrete API discussion in #8129.

[icculus]

any chance we can get query functions for user media directories (ie ~/music/) too while we're at it?

This happened in #7654.

[leo60228]

I don't know how much interest there would be in practice, but Apple has their own cloud save API: https://developer.apple.com/documentation/gamekit/saving_the_player_s_game_data_to_an_icloud_account

Notably, this is AFAIK the only reasonable way to store save data on tvOS (since persistent storage on the local filesystem isn't available there).

[flibitijibibo]

The beginnings of SDL_Storage are now in main - this would be my checklist for calling this done:

• [ ] Android storage
• [ ] iCloud user storage
• [ ] XGameSave user storage (Xbox can use Generic for title)

We pretty much already have Switch working, just need to migrate our current system to this API. We should probably check PlayStation as well but that can probably come later.

[leo60228]

It seems like the GameKit cloud save API is inexplicably unavailable on tvOS, despite the rest of GameKit and CloudKit both being available. I must have missed that when I posted about it before. I'm not sure what would be reasonable to do for tvOS because of this.

This is despite Apple claiming that Apple Arcade games on tvOS have their saves synced via Game Center. I'm very confused on what they're expecting developers to do here....?

[icculus]

Just consolidating back down to one issue.

@flibitijibibo said this:

Yep, should be good to go - before freezing we should definitely make sure PlayStation works; Xbox and Switch should be fine as-is.

And @TylerGlaiel mentioned this:

Random request if filesystem stuff is getting added into SDL, tracking when a file changes (for hot-reloading purposes) is something that requires platform-specific code (or weird hacks) to achieve normally, so some stuff in SDL to help with that would be pretty cool actually

SDL_WatchFileForChanges(const char *path);

and then some SDL_EVENT_FILECHANGED to get notified when the file is written

I think everything else from #8129 was resolved in some way or another.

[slouken]

@flibitijibibo, is there anything that might change the ABI for this at this point?

[flibitijibibo]

Nothing other than a PlayStation implementation, aside from that all my remaining items are additions like async support so this should be safe to lock down.

[slouken]

Would the additions require changes to SDL_StorageInterface?

[flibitijibibo]

Oh, right, we have the public vtable - yes, that would be a breakage. I'm currently trapped in GPU land so I won't be able to figure out async support; did we ever make async I/O for the low level filesystem?

[slouken]

We didn't. @icculus, do you want to share the details?

[icculus]

So I intended to make this grand and glorious design that just made all SDL_IOStreams (SDL_RWops) async, faking it with a thread behind the scenes when there wasn't a platform API that handled it, and that was a mess for several reasons.

Now I'm thinking of a very small interface that is completely unrelated to SDL_IOStream that only handles async i/o. The Windows version will use "overlapped" i/o internally with it, etc.

The problem at the moment, beyond this code not yet existing, is the obvious need for SDL_Storage to offer it, since there will certainly be a cloud API somewhere that benefits from it, even if basic file i/o doesn't improve. But this needs an addition to the vtable.

I'm inclined to add a SDL_FunctionPointer reserved; field to the vtable so we can add it in a future version without breaking ABI, since we really want to lock down sooner than later and everyone is crunching on other things right now.

[slouken]

Do we want several reserved function pointers? How do we guarantee the table is zeroed in the API usage?

[icculus]

Do we want several reserved function pointers?

So the idea is when we add/change stuff (RARELY), we just make a version 2 of the struct and a new entry point that knows about the new thing, and change the old entry point to provide reasonable defaults for the stuff that wasn't in version 1, before passing it on to version 2.

I'd just rather our first change post-3.2.0 not be to make a version 2 of the table. :)

How do we guarantee the table is zeroed in the API usage?

Dereference the reserved pointer if it isn't NULL. :)

This is all a terrible idea, let's just try to get this into 3.2.0 instead, even if the async interface just always returns -1 immediately for now.

[icculus]

Async stuff (which needs changes to the Storage interface) will be tracked in #1374. I'm moving this issue out of the ABI milestone because that's the only thing that needs ABI changes, but we probably still want to add file watching later.