Refactor rendering to use SDL3 GPU

[NoelFB]

This is more of just a long term intention of replacing the current custom rendering API implementations with SDL3's GPU api, once it is ready.

This will likely require breaking changes to the C# Shader API as I believe the plan is for SDL3 to have a custom shader language.

[NoelFB]

I'm likely going to try implementing https://github.com/thatcosmonaut/SDL/tree/gpu in a branch. It's a SDL3 GPU implantation proposal that I think has a good chance of going through, and I've been following its development for quite a while. I think I'll likely keep OpenGL support as well to keep potential webgl and older platforms functional, but long term if it works out I might move to it entirely.

At which point, the entire current C layer's usefulness is really up for debate. There's still a fair amount of stuff (image loading, font loading) that really requires a native library but SDL3 (with this GPU implementation) really covers everything else required...

[MrBrixican]

Sounds great. I've been following development as well and I'm interested to see how complex using the new API is in practice compared to using particularly verbose backends directly (looking at you Vulkan). From what I can see it's not horrible, but still certainly a big shift in thinking from OpenGL. Also curious as to if there's any performance gains to be had.

[NoelFB]

still certainly a big shift in thinking from OpenGL

Yeah, Foster in general is set up to abstract a lot of that away because it's intended for simple 2D games, but the tradeoff is it doesn't support a lot of the more complex options that a modern 3D rendering API has. I could potentially see a C# library that is more 1-1 with everything SDL provides being pretty cool... I'm not sure if that makes sense for Foster or not, though.

[NoelFB]

Keeping OpenGL is a bit tricky due to the differences in how Uniforms work. Newer APIs tend to just use "Uniform blocks" where you just pass blobs of data for vertex/fragment uniforms and it's up to you to make sure the data matches what the shader expects. Where as OpenGL (or WebGL) combines the shader programs and then has a big list of named Uniforms you're expected to assign individually. I do like the new system a lot more but I'm not sure how to handle this gracefully without having to awkwardly have the user build a table of all their uniforms if they use OpenGL (which I believe is what sokol gfx does)

[RandyGaul]

I implemented a wrapper abstraction over sokol_gfx called a CF_Material. The material design holds a lookup table of string-key'd uniforms and composites them, at binding time (to a shader for a draw call), into a block of memory. This block of uniform data (in typical CPU memory still) gets handed to sokol_gfx which then funnels into the graphics backend in whatever way it needs to.

Now on the user side the material just acts as catch-bucket of cf_material_set_uniform_fs or cf_material_set_uniform_vs, where you set floats/textures onto the material by name. The name matches with the name in the shader for the uniform, as you'd expect. The matching happens at bind-time (like mentioned above), and any uniforms in material that don't exist in the shader are simply ignored. Any uniforms in the shader that aren't in the material get logged for warning and cleared to zero (the CPU uniform block is cleared to zero initially, so these zeroes get sent to the uniforms in the GPU even when missing in the material).

I think this has been an extremely easy to use design and wanted to share the design here. I figured Noel might like the design, or at least it could jog some ideas on how you want Foster to shape up in the future regarding shaders and uniforms.

Here's a brief sketch of my current material API for design reference:

CF_INLINE Material make_material() { return cf_make_material(); }
CF_INLINE void destroy_material(Material material) { cf_destroy_material(material); }
CF_INLINE void material_set_render_state(Material material, RenderState render_state) { cf_material_set_render_state(material, render_state); }
CF_INLINE void material_set_texture_vs(Material material, const char* name, Texture texture) { cf_material_set_texture_vs(material, name, texture); }
CF_INLINE void material_set_texture_fs(Material material, const char* name, Texture texture) { cf_material_set_texture_fs(material, name, texture); }
CF_INLINE void material_clear_textures(Material material) { cf_material_clear_textures(material); }
CF_INLINE void material_set_uniform_vs(Material material, const char* block_name, const char* name, void* data, UniformType type, int array_length) { cf_material_set_uniform_vs(material, block_name, name, data, type, array_length); }
CF_INLINE void material_set_uniform_fs(Material material, const char* block_name, const char* name, void* data, UniformType type, int array_length) { cf_material_set_uniform_fs(material, block_name, name, data, type, array_length); }
CF_INLINE void material_clear_uniforms(Material material) { cf_material_clear_uniforms(material); }

Happy to discuss ideas more here or elsewhere if that's helpful!

[NoelFB]

Thanks for sharing some thoughts here, this is definitely a cool direction. With SDL3's GPU, it doesn't actually expose any of the uniform names in the API as far as I know (and instead just uses blocks, and expects your data to match the block). I could maybe have some kind of tool to output the shader uniforms beforehand, or alternatively require the user to provide those. (ex. when you create a shader you also need to provide the uniforms that exist on it).

Alternatively I could just make it more 1-1 with SDL_GPU and just require the user to submit blocks of data, but that's not quite as nice as using uniform names in my opinion...

[RandyGaul]

I’m also trying to implement SDL_Gpu. I wanted to be able to pass in glsl and have it cross compile to all backends, so I’ve tried statically linking in some other spirv libraries. This is pretty annoying, but they are all open source and free. This is also temporary until icculus provides a better SDL_shader_tools. I have the online shader compilation stuff (glsl -> SPIRV) behind an optional cmake feature flag, since it does bloat exe size, takes more build time, and requires python3 for cmake to configure properly).

So far I’ve got runtime shader compilation from glsl to SPIR-V, which can be saved to disk as blobs and shipped. Then use a tool made by the SDL_Gpu team to cross-compile the bytecode to whatever backend SDL_Gpu is using — just like in the SDL_Gpu examples.

Since I’ve got the glsl -> SPIR-V blob working it was easy to also using SPIRV-Reflect (header only library) to pull out reflection info from the shaders. I’m pulling out those four count variables that SDL_GpuCompileFromSPIRV needs, but also vertex input layout, names, types, and more importantly uniform block names and names of all the members, types, offsets in the uniform block. The reflection is filling in these little structs internally:

struct CF_UniformBlockMember
{
    const char* name;
    CF_UniformType type;
    int array_element_count;
    int size; // In bytes. If an array, it's the size in bytes of the whole array.
    int offset;
};

On the CPU side it’s not too much trouble to then map user uniforms to a little block of memory with matching layout to the shader. Then one call to SDL_GpuPushVertexUniformData can be issued.

This way users can just write some glsl, send in uniforms by name keys, and be done with it. Since the shaders are compiled online it’s easy to add in hotloading support as well.

I imagine you’d probably want something like for Foster as well, to simplify uniform workflow for users and make authoring the shader have less manual steps.

Edit: Example reflect code https://github.com/libsdl-org/SDL/pull/9312#issuecomment-2282973157

[Kikio229]

There's still a fair amount of stuff (image loading, font loading) that really requires a native library

I know SDL_image and SDL_ttf exist, those might be of use.

[NoelFB]

I got the very early basics of this working in a branch in Foster... Currently shaders/materials are very hacked in (hard-coded, no reflection, only spir-v, etc) but it seems promising! It wasn't very hard to get the basics working.