SPIR-V cannot map to HLSL registers robustly

[seanbaxter]

I'm having an existential crisis involving HLSL register packing. spirv-cross and all other producers of HLSL or DXIL are in the same situation, so I'm asking your strategy for this.

DXBC requires mapping in/out interface variables to registers, and packing them. The actual semantic names and indices of interface variables may be ignored by the driver, and are only used to issue errors when there is a register mis-match between adjacent shader stages.

GLSL and SPIR-V declare their interface variables globally and without order. When an interface variable is ODR used it gets drawn into the module. Additionally, a stage needn't declare as inputs all interface variables set as outputs by the preceding stage.

This presents a seemingly unworkable challenge to translating GLSL/SPIR-V to HLSL/DXIL. How do you map interface variables from different shader stages so that variables with the same semantic name and index (or really just location decoration) also are assigned the same registers across stages? To complicate matters, some of the SV_ variables, which have BuiltIn decorations instead Location decorations in SPIR-V, are actually assigned registers in DXBC.

My C++ -> SPIR-V -> DXIL attempt here failed, because location0 output from vert is assigned a different register than location0 input in frag: https://gist.github.com/seanbaxter/ed1f42b8d57ea192a1d53904235b648a

D3D12 ERROR: ID3D12Device::CreateGraphicsPipelineState: Vertex Shader - Pixel Shader linkage error: Signatures between stages are incompatible. Semantic 'location' is defined for mismatched hardware registers between the output stage and input stage. [ STATE_CREATION ERROR #660: CREATEGRAPHICSPIPELINESTATE_SHADER_LINKAGE_REGISTERINDEX]

When it doesn't segfault, the debug layer gives this message.

wgpu users have also run into this problem: https://github.com/gfx-rs/wgpu/issues/1171

I think the only viable solution is to patch the registers at runtime, directly before CreateGraphicsPipelineState. Since all shader stages are submitted at once, we can use the semantics to assign consistent register numbers.

The extreme downside to this is Dxil validation. If you were to poke in new register indices, that would invalidate the module's hash, and you'd have run validation again!!

I'm just trying to get my thoughts down here. There aren't many distinct shader efforts and I think we need to put our heads together and probably get MS to fix this mistake, because it's I think it's going to be a real menace.

[HansKristian-Work]

This is ... not an easy topic. The main problem with interface matching and HLSL is that you need information that does not necessarily exist. HLSL location/register packing is essentially a black box and the only rule is that same input generates same output basically ... SPIRV-Cross attempts to sort things based on location / builtins, but when inputs are missing it gets really messy.

[konstantinegorov]

Aren't the mask_stage_output_by_location/mask_stage_output_by_builtin functions what we need to solve this?