Best way of enabling shader debugging for D3D11?

[hsjunnesson]

Currently, sokol_gfx will call d3dcompile() with the flags D3DCOMPILE_PACK_MATRIX_COLUMN_MAJOR | D3DCOMPILE_OPTIMIZATION_LEVEL3 which makes it very difficult to debug step through shaders in RenderDoc or other debuggers. I resorted to this, but I feel there might be a better way of doing this?

    UINT flags1 = D3DCOMPILE_PACK_MATRIX_COLUMN_MAJOR;

    #if defined(SOKOL_DEBUG)
        flags1 |= D3DCOMPILE_DEBUG | D3DCOMPILE_SKIP_OPTIMIZATION;
    #else
        flags1 |= D3DCOMPILE_OPTIMIZATION_LEVEL3;
    #endif

The shader is compiled via sokol-shdc.

[floooh]

Ah hmm right, maybe we need a runtime option for that in sg_shader_desc. For Metal it's enough to run sokol-shdc without the --binary option (e.g. MSL shaders are passed into sokol-gfx as sources, not as binary blobs), this is enough to enable source level debugging in the Xcode Metal debugger. But I rarely tried source-level debugging for D3D11.

PS: or maybe a global flag passed to sg_setup() makes more sense...

[hsjunnesson]

I'll let you cook. But like your example with Metal, I feel that this is pretty dependent on the specific backend. I havn't found a tool that lets me source debug shaders with OpenGL, so that's maybe a moot point for that backend. Vulkan will probably have seventeen different flavors of optimization flags and debuggability extensions, so maybe one global flag isn't enough there?

[floooh]

We can incrementally solve those backend-specific issues as they show up, and there's also no need to wrap this all under a single backend-agnostic flag, it's ok to have backend-specific flags in the desc structs as long as ignoring them doesn't break anything.

[Ignitron]

I had the same issue, shaders can't be decompiled with d3d11 backend. Temporary fixed it by hardcoding the required flags...