floooh
commented
4 years ago Yep I understand exactly what you have in mind and I've been considering and experimenting with allowing to put expensive "decoding / preparation work" into the IO thread (mainly in earlier invocations of my async loading code though).
The TL;DR is that I haven't found a satisfying solution which solves the following two requirements, so I left it out completely for now:
- there's no satisfying solution for WASM / emscripten unless "true threading" can be used there (currently everything happens on the main thread, and "traditional" worker threads with message-passing suck for this type of work when big chunks of binary data must be moved back and forth)
- sometimes it would be better if the "decoding work" had access to systems that don't live in the IO thread (for instance when asynchronously creating resources for sokol_gfx.h, which is a problem that also needs solving).
...so basically I delegated that problem for now to the user.
What should work already however is that you can run as many sokol_fetch.h "instances" as you want in your own threads (one sokol-fetch per thread), that way you could have your own "data loader-and-decoder" thread off your main thread, which initializes and "pumps" its own sokol-fetch and where decoding/preparation happens on your own thread.
sokol-fetch can be configured to be very slim, so you could have a number of those "meta-IO" threads each with its own sokol-fetch instance and IO buffer(s).
This isn't tested much though, but if you want to try that approach I'd be interested in any stumbling blocks you encounter.
agorgl
Your sokol_fetch header is an awesome way to load resources in parallel while keeping the interactivity of the application (animated loading screens), but it currently assumes that the only critical time an application spends is in the fetching of the resources.
Most of the times, decoding / processing the fetched resource (like decoding jpeg data, preparing the mipmaps) takes a significant percent of the time. It would be awesome if we had a 'ready' state similar to dispatched / fetched states that will be called from the IO thread in order to process/decode/parse the resource in parallel.
The idea is that after the 'ready' state we would use the fetched (or another 'processed') state to handle the resource in the main thread, in order to upload the decoded/processed buffer on the GPU or any other operation that needs these data ready available in the main thread, minimizing the time spent on the main thread.
I'm not sure if this idea fits into the sokol_fetch library or it should be another more generic sokol_tasks library. I couldn't find a similar quality C library like yours that could satisfy my needs, and I would love to see that happening!