SDL3 filesystem API

[icculus]

People have asked for this for years, and it's probably worth adding it.

The problem

Windows. Windows is the problem.

• Windows wants you to mess with UTF-16 strings for Unicode filepaths, everything else wants UTF-8.
• Windows wants you to use Win32 APIs, everything else uses POSIX.
• Windows wants you to use FILETIME (100-nanosecond increments since 1601), everything else uses POSIX (time_t, Unix epoch).
• Windows wants you to use '\\', everything else uses '/'
• Windows has drive letters, everything else has mount points.
• Windows sorta has symlinks in modern times, many other things always do. But some things don't at all!

The solution

We could do this as complicated as we want or just make a simple wrapper over mkdir, opendir, etc.

Relatively simple:

typedef enum SDL_StatPathType
{
    SDL_STATPATHTYPE_REGULAR, /**< a normal file */
    SDL_STATPATHTYPE_DIRECTORY, /**< a directory */
    SDL_STATPATHTYPE_SYMLINK, /**< a symlink */
    SDL_STATPATHTYPE_OTHER /**< something completely different like a device */
} SDL_StatPathType;

typedef struct SDL_Stat
{
    Sint64 filesize;
    Sint64 modtime;
    Sint64 createtime;
    Sint64 accesstime;
    SDL_StatPathType filetype;
} SDL_Stat;

typedef struct SDL_DirectoryHandle SDL_DirectoryHandle;  // opaque

extern DECLSPEC const char * SDLCALL SDL_GetFilePathSeparator(void);  // "\\" or "/" 
extern DECLSPEC void SDLCALL SDL_FileTimeToWindows(Sint64 ftime, Uint32 *low, Uint32 *high);
extern DECLSPEC Sint64 SDLCALL SDL_FileTimeToUnix(Sint64 ftime);
extern DECLSPEC int SDLCALL SDL_StatPath(const char *path, SDL_Stat *statdata);
extern DECLSPEC int SDLCALL SDL_CreateDirectory(const char *path);
extern DECLSPEC int SDLCALL SDL_RemoveDirectory(const char *path);
extern DECLSPEC int SDLCALL SDL_RenameFile(const char *oldpath, const char *newpath);
extern DECLSPEC int SDLCALL SDL_DeleteFile(const char *path);
extern DECLSPEC SDL_DirectoryHandle *SDL_OpenDirectory(const char *path);
extern DECLSPEC char *SDL_ReadDirectory(SDL_DirectoryHandle *dir);
extern DECLSPEC void SDL_CloseDirectory(SDL_DirectoryHandle *dir);

Even this has a problem: there are legit errors one will want to check for without parsing SDL_GetError(), like mkdir() failed because the directory already existed? Or unlink() failed because the file didn't exist? That's totally recoverable by just carrying on. So we'll want to return something other than -1 or 0 for these, but that can be a whole can of worms.

We don't want to get to the point where this is exposes the entire system API (like...file permissions, etc), but in terms of complexity, there might be some value in simplifying a few things (RenameFile making efforts to copy the file transparently if crossing a filesystem boundary, DeleteFile being smart enough to remove a directory, a function that will delete a whole tree instead of individual nodes, etc).

Do we want to mess with symlinks (making them and reading them)? I'm thinking no.

I don't necessarily think we should do an abstract interface like RWops ("look, I exposed my email inbox as a directory listing!"), but certainly PhysicsFS has made use of this idea to expose .zip files and other archives like filesystems, so I won't pretend the idea isn't useful, but just hiding the platform specifics and C runtime here might be a net positive and we call it a day without going wild.

Looking for feedback before I start on this in earnest.

[flibitijibibo]

The main thing that would be really great to have is an OpenContainer and CloseContainer - for most systems this will probably just be a stub, but with so many platforms moving to a custom filesystem for cloud saving and such (consoles especially), it'd be great to have this so that it's possible to expose save containers via SDL without having to make a whole new API for each and every platform we support (which is pretty much how we do it for SDL2, even GDK is doing this via a simplified version of their official API: #7621)

Whatever we end up making we should definitely make sure it's possible to support Nintendo/Xbox/PlayStation at minimum before locking this in; Steamworks would be a nice bonus.

[icculus]

The way this worked for the Switch in SDL2 was super-hacky: if you called SDL_GetPrefPath, it would mount the save filesystem (and if you didn't call it, SDL assumed you were handling it directly), and we added a switch-specific API to say "we have written everything this save game needs, please do the atomic commit magic now." But that last part was mostly a convenience function, so you didn't have to call into C++ and link directly against the Nintendo SDK, since SDL was already doing both of those things for you.

Is this the sort of thing you are talking handling better when you say "containers"?

[icculus]

Actually, I just reread the comment at the top of #6644, if that'll save you typing. :)

[icculus]

Steamworks would be a nice bonus.

Also I am legitimately surprised we haven't had a moment where we had to dlopen Steamworks yet. Just wait until we add an achievement API, lol.

Surely most games are not aware of any sort of Steam user account separate from the user's login, even though this separation has been true as long as there has been a Steamworks SDK at all. How does this work now? Does Steam do anything to try and move the right user's data into place, or is assumed that most games on Steam are currently buggy in this regard and most people simply don't notice...?

[ell1e]

I think a good idea would be to use surrogate escaping since WINAPI file paths aren't UTF16, but UTF16-plus-invalid-surrogate-pairs. This uses surrogate codepoints in UTF-8 to encode the invalid pairs found in the windows string (which is strictly speaking invalid UTF-8, so basically an UTF-8-and-trash superset), so that it can losslessly translate back. This is afaik also something CPython does, for example. Last time I checked PhysFS didn't seem to do this which is why I stopped using it, IMHO any API chosen should maintain the ability that any file that shows up in a directory listing originating from WinAPI can somehow end up in something UTF-8-alike and be passed back to actually do operations on all these files meaningfully without some files suddenly being inaccessible due to their naming.

Edit: beyond that, I think keeping it simple is a good idea. Like no symlinks, and why not put containers into a separate addon library instead? I would have suggested the same for the new complex sound system honestly, I think it's a good idea to not stuff too many things into the core. SDL2 is used increasingly beyond games, and save game containers are very game-specific, as is in some ways a complex sound mixer.

[ell1e]

Small addendum: I would recommend being careful with SDL_RemoveDirectory if it's meant to support non-empty directories. This could become complicated really quick both on the implementation side, for example you would then want to ensure that you don't descend into symlinks (which requires fd handles for directories to not open yourself up to race conditions) and on the expectations side, since opinions might vary a lot on whether after one inside item failed to delete if the entire operation should stop or instead try to carry on deleting as many other contents as possible (at which point you often would want to get back a list of errors for each failed item instead of just the first or last error). I guess the most dumb way to implement it would just stop at the first failed item and not provide any options for other behavior, but then how do you reliably know on the SDL2-using-app side if it was "just" failing to delete some sub dir that unexpectedly got written to again in a race and therefore failed to delete after being emptied (in which case many apps using it would want to retry) or a permissions or I/O error (in which case many apps using it would want to abort)? It's a bit of a can of worms.

Edit: and I also think that maybe moving such more complex filesystem things into an addon library should at least be considered. For example, in my bigger applications I hand-roll all these things since SDL2 couldn't possibly provide all the functionality I need, but then why then bring all the limited, not-quite-there variants into SDL2 to just sit around unused when shipping apps like mine? (Since there's often a point to bringing SDL2 along with the app, but then having more and more in it that isn't universally used hampers that a bit.) But I can see how that one might be wanted enough while also small enough to have it in the core, but just as a thought.

Sorry for my rambling, I hope some of this is useful.

[flibitijibibo]

Also I am legitimately surprised we haven't had a moment where we had to dlopen Steamworks yet. Just wait until we add an achievement API, lol.

Once I looked at both containers and action sets I kinda went "welp, the floodgates have opened I guess"

Surely most games are not aware of any sort of Steam user account separate from the user's login, even though this separation has been true as long as there has been a Steamworks SDK at all. How does this work now? Does Steam do anything to try and move the right user's data into place, or is assumed that most games on Steam are currently buggy in this regard and most people simply don't notice...?

Right now it's still single-account, but with Deck doing well I'm hoping they'll eventually throw together a system that allows multiple accounts to be signed in, even if only for Remote Play... but for now it would probably be single-account only.

Other systems would probably just need to take a void*, or a union SDL_SysAccount pointer that can tell a container what user it should try to open storage media for. This might be useful for input as well, since I believe at least one platform we have ties joysticks to OS users logged in.

[icculus]

I would recommend being careful with SDL_RemoveDirectory if it's meant to support non-empty directories.

Yeah, I have these same concerns. If we were to do this, it would be a separate function ("SDL_DeleteFileTree" or something) and it would exist under the assumption everyone rolling their own is going to get it wrong in these exact ways, so we might as well debug it once and make it available to everyone.

But whether we should do this specific thing at all is an open question. I keep leaning one way or the other on it. Right now on this specific thing I'm leaning towards "no."

SDL does offer convenience functions for some things, fwiw, and I'm not against it doing so. The idea that SDL is just a small cross-platform abstraction over platform-specific functionality hasn't been true in many years--if it was ever true--and while we aren't trying to build it out into being a AAA game engine, these sort of functions aren't out of place in the library. (But, again, I'm not sure this function will happen, I'm just speaking generally.)

[icculus]

Just moving some notes in here from other bugs, so they aren't lost:

• @1bsyl notes in #1214 that maybe it's silly to have a directory iterator in modern times, and just return a single atomic(ish) list of filenames instead.
• There are requests to make directory access work like RWops (a public struct of function pointers for which SDL provides a system-specific implementation but apps can get buck-wild with). It's basically the core of #1214.
• Ethan's full comment about Containers is here.

[ericoporto]

Maybe not important or significant here, but wanted to mention a use-case. Often I need to do some effort to check if the path is in a game directory, like where it is or in the appdata and similar first for the app - SDL already has a method for retrieving these directories.

When I am not in these directories, I often don't want any file operation to happen, to avoid messing with the user computer.

So like, if there's some space in the API, if there is function that can tell me if a path is inside a specific directory, it would be very useful to me.

Ethan's full comment about Containers

Ah, similar to this, in case of Emscripten I actually run a callback after file operations and then I aggregate these for a time period in an event to sync the files in memfs to the idbfs or server. Android has an interface that expects to be connected to a stream.

Ah, and currently I use my own streams and file streams to abstract all of these and just pass my rwops.

[ell1e]

if there is function that can tell me if a path is inside a specific directory, it would be very useful to me.

For what it's worth, I think this is quite complicated with mount points and hard links to resolve correctly, so it could also be quite a headache to implement properly. And non-properly it might be a bad idea to include.

As for directory iterators, I thought the reason they exist was really giant directories with tens of thousands of files. Those can still exist in modern times, so IMHO that approach make sense when using a more performance focused language like C. (That e.g. Python defaults to a list makes more sense to me.) But I guess realistically, 99% of SDL2 apps wouldn't need to handle directories of that size well.

[icculus]

As for directory iterators, I thought the reason they exist was really giant directories with tens of thousands of files

They probably exist because tens of thousands of files would have crushed a process's address space in 1970. :)

But yeah, in the sense that you could have theoretically infinite files in a finite machine, the iterator makes sense, but as Sylvain pointed out, a single 32-bit 4K image is 129600 256-byte filenames, so maybe it's easier to not have one in modern times. I don't know yet, so feedback is definitely good here.

[ell1e]

This seems kind of unimportant so feel free to ignore me, but for what it's worth, do you then allocate 129600 times to create that list? An iterator sure is easier to provide in a way that doesn't scale badly. I guess you could use some single allocated large pool area and then point to locations one after another to pack the names in, but that would also potentially waste a lot of memory. Basically, I don't think handling that array efficiently is necessarily easy even when in a pedantic sense your computer can handle an array of that size with no issue once it's fully allocated and filled in. I hope that thought is helpful. Nevertheless, I guess a list wouldn't be the worst way to handle it for most users in most situations.

[icculus]

@flibitijibibo When you get a chance, can you shove some function declarations in here, like:

int SDL_OpenContainer(WhateverShouldActuallyBeHere *blah);

Nothing complicated or documented, just enough that we're all talking about the same thing, like I did in the first comment.

It might be that OpenContainer or whatever is how you get the equivalent of a RWops for directories from SDL and that's all it has to be and we kill a couple birds with one stone...?

[flibitijibibo]

Will try and get this done within a day or so (Marble It Up launched today so expect some delays...)

It may also be worth making a distinction between Title containers and Storage containers; the former would be the read-only filesystem for your application's content, and the latter would be a read/write filesystem for user data.

[ericoporto]

Please take a look at my comment here as the closed issue was about android asset files. (see the forums ...)

[icculus]

This seems kind of unimportant so feel free to ignore me, but for what it's worth, do you then allocate 129600 times to create that list?

A hypothetical implementation would start with a small buffer and call realloc to double its size every time it fills, storing all strings in one buffer.

This gets slightly more complicated to manage the actual list, but it still would be--for the pathological case--tens of allocations, not thousands.

[flibitijibibo]

/*
 * Common list iterator for folders/files
 */

typedef struct SDL_PathList SDL_PathList;

/* Call this until the return value is NULL */
const char* SDL_PathListGetNext(SDL_PathList *list);

/*
 * App storage
 */

typedef struct SDL_AppStorage SDL_AppStorage;

/* Start by calling this... */
SDL_AppStorage* SDL_BeginOpenAppStorage(void);

/* ... check this bool until it's true */
SDL_bool SDL_AppStorageReady(SDL_AppStorage *storage);

/* Once you're completely done, you can free the pointer here */
int SDL_AppStorageClose(SDL_AppStorage *storage);

/* The filesystem, finally */
SDL_RWops* SDL_AppStorageOpenFile(SDL_AppStorage *storage, const char *path); /* Always read-only */
SDL_bool SDL_AppStorageDirectoryExists(SDL_AppStorage *storage, const char *path);
SDL_bool SDL_AppStorageFileExists(SDL_AppStorage *storage, const char *path);

/* App storage enumeration */
SDL_PathList* SDL_AppStorageGetDirectories(SDL_AppStorage *storage, const char *path, const char *searchPattern);
SDL_PathList* SDL_AppStorageGetFiles(SDL_AppStorage *storage, const char *path, const char *searchPattern);

/*
 * Storage Containers
 */

typedef union
{
#ifdef SDL_STORAGE_STEAM
    struct
    {
        CSteamID steamID;
    } steam;
#endif /* SDL_STORAGE_STEAM */
    Uint8 padding[sizeof(void*) * 8];
} SDL_SysAccountInfo;

typedef struct SDL_UserStorage SDL_UserStorage;

/* Start by calling this... */
SDL_UserStorage* SDL_BeginOpenUserStorage(const char *org, const char *app, SDL_SysAccountInfo *account);

/* ... check this bool until it's true */
SDL_bool SDL_UserStorageReady(SDL_UserStorage *storage);

/* Once you're completely done, you can free the pointer here */
int SDL_UserStorageClose(SDL_UserStorage *storage);

/* For all the functions following this one, call this to push all filesystem changes made since the last flush */
int SDL_UserStorageFlush(SDL_UserStorage *storage);

/* The filesystem, finally */
int SDL_UserStorageCreateDirectory(SDL_UserStorage *storage, const char *path);
SDL_RWops* SDL_UserStorageOpenFile(SDL_UserStorage *storage, const char *path, const char *mode); /* TODO: Access, Share? */
SDL_bool SDL_UserStorageDirectoryExists(SDL_UserStorage *storage, const char *path);
SDL_bool SDL_UserStorageFileExists(SDL_UserStorage *storage, const char *path);
int SDL_UserStorageDeleteDirectory(SDL_UserStorage *storage, const char *path);
int SDL_UserStorageDeleteFile(SDL_UserStorage *storage, const char *path);

/* User storage enumeration */
SDL_PathList* SDL_UserStorageGetDirectories(SDL_UserStorage *storage, const char *path, const char *searchPattern);
SDL_PathList* SDL_UserStorageGetFiles(SDL_UserStorage *storage, const char *path, const char *searchPattern);

[MarcelHB]

Please also keep in mind that there is case-insensitivity for file paths on Windows.

That's somewhat easy to handle for ASCII range chars, but turns out to be tricky when you try to find out how to get the directory named Männlich from API while there is an asset folder MÄNNLICH on disk (you may not have a chance to rename these files/dirs at will) on POSIX without any FS hacks. On Windows, this is not a problem at all.

[ell1e]

In theory a way to address that might be a path or file name comparison function offered by SDL2, which on Windows could possibly use CharLowerW, but in practice I'm not sure the filesystem is actually guaranteed to be case insensitive on Windows, and e.g. on the Steam Deck I think it's case insensitive despite it being on Linux. So I guess in practice implementing anything to help with that might be impractical, due to how difficult it is to find out if a filesystem actually is case sensitive in the first place.

[MarcelHB]

Yeah, Steam Deck makes use of casefolding ext4 for that reason.

Maybe one should leave it to the user to opt-in for casefolding? We solved it by trying an exact match first, and then a C.UTF-8 tolower comparison on Linux as fallback over iteration since we know exactly that this may occur to us.

[icculus]

Case folding is not as hard as people think it is (except for that one Turkish 'i' character, amirite?!), but also I don't think SDL should handle this for filesystem accesses.

[icculus]

@flibitijibibo First thought looking at this: this makes me want to put the RWops-style struct-full-of-function-pointers magic in here after all, with app storage implementing the writable parts of this as simple stub functions that return -1.

But also, it makes me want this to be a separate thing, unrelated to this work, or built on top of it; I think it's going to scare people off that just want to get a directory listing. Alternately, it might be that we replace the GetBasePath/GetPrefPath calls with this, instead, idk yet.

[sin-ack]

This might be unimportant in the grand scheme of things, but do these APIs support relative paths? Then I would recommend implementing some kind of *at alternative to these APIs at least, or these APIs could have *at variants instead. It can easily get disorienting to use relative paths when you're working with many different folders, and it also helps to be able to have an anchor that you can base your relative paths on.

For example:

// This can be opaque, with platform-specific implementations
typedef struct SDL_Directory SDL_Directory;
// Special SDL_Directory instance that means "use my current directory please", equivalent to POSIX' AT_FDCWD
extern SDL_Directory *SDL_CURRENT_DIRECTORY;

// "Opens" the directory, this could mean a dirfd on POSIX and just yeeting the path inside a struct on Windows
extern DECLSPEC SDL_Directory * SDLCALL SDL_OpenDirectory(const char *path);

// Relative to the current directory. Delegates to SDL_CreateDirectoryAt internally
extern DECLSPEC int SDLCALL SDL_CreateDirectory(const char *path);
// Relative to the directory that's passed in, equivalent to mkdirat(2)
extern DECLSPEC int SDLCALL SDL_CreateDirectoryAt(const char *path, SDL_Directory *directory);

[icculus]

I assume we wouldn't forbid relative and ".." paths, but you're at the mercy of the OS if you use them. For things where SDL would return a path, it'll return an absolute one.

[icculus]

Here is where I am right now on this.

• All paths are UTF-8, like the rest of SDL.
• We present a RWops-like interface, that apps and libraries can implement, and SDL provides a simple cross-platform wrapper over the native filesystem.
• Since the wrapper implementation can be given a subdirectory to treat as the "root" of the filesystem, one can trivially point it at specific things (a user-specific directory, the "title" data for a game, whatever) without us having to write implementation code for, say, where to find Steamworks data or whatever.
• We can add implementations for "weird" filesystems like Nintendo Switch (SDL_CreateNintendoSwitchROMFS()) that only exist in SDL-switch, SDL-playstation, etc.
• Things like case insensitivity, read-only access, etc, can be built as wrapper interfaces over other implementations. This doesn't have to be in SDL itself.
• I went with a callback for directory enumeration, so there's no question of maintaining a directory handle vs allocating all results. The app gets them one at a time before the enumeration method returns, it can stop at any time, and nothing strictly needs any allocating. We can offer a helper function that allocates a list through the callback so the app doesn't have to roll this.
• I have not decided if we should dictate that all paths use '/' for a separator, and the Windows-specific code will have the burden of converting behind the scenes. This is what PhysicsFS does, but maybe that's not in the best interest here.
• Also unlike PhysicsFS, this does not forbid the '..' path, even if it rolls past the root. I wasn't trying to prevent espionage here, and ideally there is no common code, so the implementation is in control from start to finish.
• Nothing is set in stone, including symbol names; all of this is open to feedback!

Obviously this takes some hints from PhysicsFS without getting bogged down in making some sort of bonkers Virtual File System layer, or worrying about .zip files and junk. Those things can be built outside of SDL with this.

This only exposes a bare minimum of what one might need for filesystems (mkdir, unlink/rmdir, stat, enumerate, open a file). Obviously there are a lot more verbs in something like POSIX, but we're just looking for a reasonable game developer set here. We might want a rename operation for atomic replacement of files, but I don't know yet.

/* Abstract filesystem interface */

typedef enum SDL_StatPathType
{
    SDL_STATPATHTYPE_FILE, /**< a normal file */
    SDL_STATPATHTYPE_DIRECTORY, /**< a directory */
    SDL_STATPATHTYPE_SYMLINK, /**< a symlink */
    SDL_STATPATHTYPE_OTHER /**< something completely different like a device node */
} SDL_StatPathType;

typedef struct SDL_Stat
{
    Sint64 filesize;    /* size in bytes */
    Sint64 modtime;     /* SDL filesystem timestamp */
    Sint64 createtime;  /* SDL filesystem timestamp */
    Sint64 accesstime;  /* SDL filesystem timestamp */
    SDL_StatPathType filetype;  /* is this a file? a dir? something else? */
} SDL_Stat;

/* Callback for filesystem enumeration. Return 1 to keep enumerating, 0 to stop enumerating (no error), -1 to stop enumerating and report an error. "origdir" is the directory being enumerated, "fname" is the enumerated entry. */
typedef int (SDLCALL *SDL_EnumerateCallback)(void *data, SDL_FSops *fs, const char *origdir, const char *fname);

/**
 * This is the directory operation interface, much like SDL_RWops is the filestream
 * operation interface.  SDL provides a platform-independent way to
 * access these that operate on the native filesystem, but the app and other libraries
 * are welcome to provide their own implementations of any data that needs a simple
 * tree-shaped hierarchy that looks a little like a filesystem.
 */
typedef struct SDL_FSops
{
    /**
     * Open a file in the filesystem.
     *
     * You do not necessarily have to use this, if you know you're working from
     * a real filesystem and just want to enumerate paths and such.
     *
     * The caller should not close the filesystem while the returned RWops is still open.
     *
     * \param fs The SDL_FSops object to use.
     * \param path the path in the SDL_FSops to open
     * \param mode a fopen-style "mode" string of how to access the path.  !!! FIXME: maybe just offer read/write/append enums?
     * \return a RWops for the path specified.
     */
    SDL_RWops *(SDLCALL open)(SDL_FSops *fs, const char *path, const char *mode);

    /* list all files in a directory. Each file is passed through a callback until we're done or the callback returns <= 0. */
    int (*enumerate)(SDL_FSops *fs, const char *dirname, SDL_EnumerateCallback cb, void *callbackdata);

    /* delete/unlink/rmdir a path. Will not remove non-empty directories! */
    int (*remove)(SDL_FSops *fs, const char *path);

    /* Create a directory. */
    int (*mkdir)(SDL_FSops *fs, const char *path);

    /* Obtain basic path metadata. */
    int (*stat)(SDL_FSops *fs, const char *path, SDL_Stat *stat);

    /* Close the filesystem and free its resources. */
    void (*closefs)(SDL_FSops *fs);

    /* Used by the SDL_FSops for private data. Presumably free'd by closedir() method. Don't touch! */
    void *opaque;

    /* Everything has properties! :) */
    SDL_PropertiesID props;
} SDL_FSops;

/* These are just optional function wrappers around abstract filesystem interface, like RWops does. */
extern DECLSPEC SDL_RWops * SDLCALL SDL_FSopen(SDL_FSops *fs, const char *path, const char *mode);
extern DECLSPEC int SDLCALL SDL_FSenumerate(SDL_FSops *fs, const char *dirname, SDL_EnumerateCallback cb, void *callbackdata);
extern DECLSPEC int SDLCALL SDL_FSremove(SDL_FSops *fs, const char *path);
extern DECLSPEC int SDLCALL SDL_FSmkdir(SDL_FSops *fs, const char *path);
extern DECLSPEC int SDLCALL SDL_FSstat(SDL_FSops *fs, const char *path, SDL_Stat *stat);
extern DECLSPEC void SDLCALL SDL_FSclosefs(SDL_FSops *fs);

/* some helper functions ... */

/* returns "\\" or "/" etc. Read-only static data, don't free or modify! */
extern DECLSPEC const char * SDLCALL SDL_GetFilePathSeparator(void);

/* converts an SDL file timestamp into a win32 FILETIME (100-nanosecond intervals since January 1, 1601). Fills in the two 32-bit values of the win32 struct. */
extern DECLSPEC void SDLCALL SDL_FileTimeToWindows(Sint64 ftime, Uint32 *low, Uint32 *high);

/* converts an SDL file timestamp into a Unix time_t (seconds since the Unix epoch). */
extern DECLSPEC Sint64 SDLCALL SDL_FileTimeToUnix(Sint64 ftime);

/* Things that make an SDL_FSops... */

/*
 * Create an SDL_FSops that represents the native filesystem, treating `basedir` as
 * the root of the tree. A NULL basedir is valid and gives you the actual root of the
 * native filesystem.
 *
 * Enumerating root on Windows will give you available drive letters like they were directories,
 * almost everything else assumes the root is a Unix "/" thing.
 *
 * Destroy this when done with SDL_FSclosefs().
 */
extern DECLSPEC SDL_FSops * SDLCALL SDL_CreateFilesystem(const char *basedir);

[icculus]

Also, if I didn't love the SDL_RWops name so much, I'd say SDL3 should make that more verbose, but since we haven't changed it, I matched it with "SDL_FSops" here. I'm not married to it, though.

[ericoporto]

Uhm, it was previously mentioned just returning the full list of what's in the directory and now there's a callback for iteration, wouldn't returning the list be easier to implement for each platform and easier to use, even if at the expense of maybe some repeated code ?

Ah, also probably irrelevant but some filesystems like in the case of Android bundles are compressed and have rather slow access, not sure if this is important but just mentioning because Android bundles look like an interesting case for reference.

[icculus]

Uhm, it was previously mentioned just returning the full list of what's in the directory and now there's a callback for iteration

It was mentioned, and some people hated it.

If we go this way, we will likely add a helper function...

/* call SDL_free on the result! */
char **SDL_EnumerateAll(SDL_FSops *fs, const char *path);

...which behind the scenes will call the interface function with its own callback that allocates a complete list.

[ell1e]

Does the new UTF8 path handling use the commonly named "WTF8" surrogate escaping for the non-unicode paths of windows? E.g. Python's standard library does that, but last I checked PhysFS didn't seem to do so. Not doing that makes certain paths inaccessible.

[madebr]

• Would the possibility to move/rename files be useful? Or useless for a gui/game?
• Would support for memory-mapped I/O be useful? Avoiding the memcpy in SDL_RWops is always nice. Or is this another can of worms?

I just browsed through PhysicsFS, and it looks like a lot of its functionality can be built on top of icculus' SDL_FSops proposal.

[icculus]

SDL_STATPATHTYPE_SYMLINK, /**< a symlink */

Symlinks should always be followed and never acknowledged as symlinks at this level, imho.

[icculus]

Not doing that makes certain paths inaccessible.

Is this a serious problem? Like, someone in Norway types a totally reasonable filename with an umlaut but in a way that generates invalid UTF-16 that gets written to an NTFS filesystem?

If this is a "oh yeah, that happens all the time outside of English-speaking America" thing, then we should definitely deal with it, but I'm not sure it's worth it beyond that.

[icculus]

I just browsed through PhysicsFS, and it looks like a lot of its functionality can be built on top of icculus' SDL_FSops proposal.

100%...PhysicsFS uses a similar interface, so apps that want to register their own external archive formats can hook in (but it has different needs and rules that wouldn't make sense in SDL, too).

[icculus]

Would the possibility to move/rename files be useful? Or useless for a gui/game?

So the most useful thing in a game for rename is having a savegame you're overwriting: you write a new savegame to a temp file and then rename it over the original filename, so you never have a point where you've destroyed the original but not yet fully generated its replacement, if the power goes out or the game crashes midway through the process.

LOTS of games get this wrong and in practice it's not a big deal, but in terms of technical correctness, we should have it, but maybe it's not worth the trouble.

Would support for memory-mapped I/O be useful?

Feels like a can of worms, and probably not useful generally. My suspicion is that you'd get roughly the same benefits just malloc()'ing a block and loading the whole file into it on most platforms, and async i/o is likely to be the better performance win or at least more useful paradigm change.

[ericoporto]

Would the possibility to move/rename files be useful? Or useless for a gui/game?

For the case of a game the use case I could think to maybe use this would be for save files.

Does the new UTF8 path handling use the commonly named "WTF8" surrogate escaping for the non-unicode paths of windows

Not sure if it's the case but there is WideCharToMultiByte to convert if it's the case

Also one thing for Android bundle is that there's no preceding slash for things in the bundle, you write dirname/file instead of /dirname/file. I saw there's no intention in dealing with "unixing" Windows paths, so if I understood this means all the burden of dealing with each platform specifics path differences is in the application right? They would deal with SDL_GetBasePath and SDL_GetPrefPath separately and this is disconnected with the things here.

[slouken]

• I have not decided if we should dictate that all paths use '/' for a separator, and the Windows-specific code will have the burden of converting behind the scenes. This is what PhysicsFS does, but maybe that's not in the best interest here.

Windows supports '/' as a path separator.

[ell1e]

Is this a serious problem? Like, someone in Norway types a totally reasonable filename with an umlaut but in a way that generates invalid UTF-16 that gets written to an NTFS filesystem?

It means when you see a file that the window file explorer can rename and access just fine, sometimes SDL3 (or PhysFS) would just give strange unexplained errors that to the user make no sense.

I know NTFS also has these weird \\?\ paths and other features, but the difference here is that as far as I know is that no normal windows app including also not windows file explorer, will properly support these either so there it's not an issue.

It's a consistency problem. Maybe less relevant for games but in my opinion a huge problem for a regular application where such weird breakages aren't nearly as tolerated by users. And SDL seems to be moving out of the just for games space, so I don't think this should just be glossed over.

Disclaimer: I won't be a user of this filesystem API, so I have no real dog in this fight. But if I were to pick a new filesystem library for my application, I would skip the ones that don't handle this correctly. It's why I eventually moved away from PhysFS as well, not that PhysFS should necessarily have catered to me. But then again, I also am wary about where SDL audio is moving and the general feature increase in general, so maybe I should be ignored. I'm definitely not the core audience for SDL with how I tend to use it. I use it more as infrastructure lib where leanness and getting corner cases right matters more than just as a use-and-forget highlevel gaming lib which is where it seems to be moving.

[icculus]

It means when you see a file that the window file explorer can rename and access just fine, sometimes SDL3 (or PhysFS) would just give strange unexplained errors that to the user make no sense.

Okay, but do this show up in the wild in any serious manner? Where someone intentionally created a file with ill-formed UTF-16?

But then again, I also am wary about where SDL audio is moving and the general feature increase in general, so maybe I should be ignored.

We're all just talking here.

[ell1e]

Where someone intentionally created a file with ill-formed UTF-16?

I would expect that this happens mostly when other software malfunctions when doing unicode conversion. I think we've all seen such things happen, so I don't see why it would be too weird to think of.

For a game it might be fine to just not work with such files, but e.g. if someone wrote a file manager based on SDL this would be a problem. Whether you care about such use cases I can't tell you, but at least in my eyes SDL used to be aiming to be a more lowlevel general infrastructure library since early SDL2 where this would in my opinion matter. Whether that is also the future trajectory I don't know, I often feel a little like most other users excluding me don't seem to care too much.

[icculus]

Yeah, I'm definitely not intending this to be feature-complete for what a file manager would need.

The current proposal is intentionally sparse, and may expand, but it's definitely not going to expand to significant things that a file manager app would want/need, like permissions, extended attributes, etc.

We have to cut somewhere, keeping in consideration both the size of the new API and the time available to build it, and getting it as right as possible will never make everyone happy, but despite that, decisions have to be made and code has to be written and shipped.

[ell1e]

For what it's worth, in my opinion sparse in terms of API isn't quite the same as incomplete regarding handling corner cases for the API that is added in. (Edit: nevertheless, you still make a valid point of course.)

[madebr]

Would it make sense for enumerate to accept globs? Such that you it only returns certain files. e.g. *.bmp, level001_*.obj, ...

[ell1e]

On Linux a file name can contain * outside of globs. It might be difficult to deal with that without caveats.

[icculus]

It's trivial for the caller to filter results, including using some sort of glob() implementation themselves, so I'd rather not at this level unless people feel strongly about it.

[jube]

SDL_STATPATHTYPE_SYMLINK, /**< a symlink */

Symlinks should always be followed and never acknowledged as symlinks at this level, imho.

It depends. When you remove a directory, you may not want to follow the symlinks inside the directory. That's the default behavior of rm on Linux.

[slouken]

Symlinks should always be followed and never acknowledged as symlinks at this level, imho.

It depends. When you remove a directory, you may not want to follow the symlinks inside the directory. That's the default behavior of rm on Linux.

Good point.

[icculus]

True, deleting it will succeed even if the directory it points to is not empty, because the symlink is being removed instead of the directory...that gets dicey.

I have an initial cut almost done; I'll be publishing a pull request tonight...we can figure out what to do with this detail before that is done.

[icculus]

Okay, PR is live, feel free to chitty-chat about it over there. :)

[ericoporto]

Is the idea to support multiple SDL_FSops for the same platform or does can it only have one?

I noticed the useful AAssetStream things from AAssetManager from Android aren't there in it's SDL_FSops so it can't find files that are in an asset bundle. How does this should work, should I create my SDL_FSops for Android assets or should my SDL_FSops have some code that does both, say it tries to find inside the bundle and if it fails it tries to interpret the path as the filesystem?