search

libsdl-org / SDL

Simple Directmedia Layer
https://libsdl.org
zlib License
10.2k stars 1.86k forks source link

SDL_storage: Streaming file I/O? #11547

Open darkerbit opened 2 days ago

darkerbit commented 2 days ago

(If this is planned for the async I/O FIXME, then feel free to just close this.)

I have some concerns about SDL_storage's "load the whole thing at once" model, mainly to do with streaming music and video.

While you can reasonably load an entire encoded music file into memory and decompress it from there these days, there would still be benefit in not having to keep audio data from minutes in the future in memory at all times. For video, loading the whole thing into memory is a complete non-starter, as video files can be upwards from hundreds of megabytes to gigabytes, depending on resolution and length.

Are there any plans for a traditional "open, read, close" model for SDL_storage, which would solve this? Or would this be too difficult to implement on certain platforms?

darkerbit commented 2 days ago

(In addition, this would likely only be a requirement for Title Storage, it makes sense to preserve atomicness (is that a word?) for the other storage types.)

slouken commented 2 days ago

@flibitijibibo, this is a reasonable point about videos, etc. What was the plan here?

flibitijibibo commented 2 days ago

For exceedingly large files this is where async would come in, but surprisingly for short videos it's not terrible to load a compressed video all at once since disk speed can be so low (consoles...) that it actually takes longer to read the stream than to decode it, so keeping the stream in-memory ends up being necessary to avoid hitching.

That said, if we had to prioritize any async function it would definitely be title reads over storage reads/writes, which would address this problem.

flibitijibibo commented 2 days ago

For seeking the plan was to avoid it as much as possible, but for title storage I guess it wouldn't hurt to have a persistent file handle that gets an exclusive ReadFile/OVERLAPPED structure - it would definitely need to be documented as a persistent handle to avoid it getting used except where files basically exceed host memory available.

(We sort of do this in FAudio, albeit synchronously: https://github.com/FNA-XNA/FAudio/blob/master/src/FACT_internal.c#L1978 )