Closed Nielsbishere closed 4 days ago
Keenuts
commented
1 month ago Hi,
Thanks for the issue!
So if I understand correctly:
Nielsbishere
commented
1 month ago @Keenuts That's correct, ideally the two binaries would match (even the ordering), so that the shader hash remains the same unless something changes in DXC or the source changes. This has the benefits that it's easier for QA to see what shaders changed, to know these might need additional testing and for diff systems (such as incremental updates) to know that nothing has changed, even if the binaries were fully recompiled on any machine.
Keenuts
commented
1 month ago Right.
To diff 2 SPIR-V modules, I'd suggest not doing a diff of the assembly/binary, but using spirv-diff (part of SPIRV-Tools). This will be more robust as it will outline logical changes in the SPIR-V shader, not just irrelevant ID changes.
As for the actual issue, seems like of the 2 shaders you passed, seems like there is 1 real difference we may want to look into: the MainMiss function parameters are swapped between the windows and linux version of the shader. I'll see if I can reproduce this.
Nielsbishere
commented
1 month ago @Keenuts Problem is that naive tools do still diff manually, unless that behavior is manually changed to use spirv-diff. I can imagine that the 2nd one is where the problem lies then. Since with a single entrypoint, this is no problem.
Keenuts
commented
1 month ago Hello!
I tried to reproduce this issue:
dxc -T lib_6_6 repro.hlsl -spirv -fspv-target-env=vulkan1.3
And even with the binary output I noticed the output is actually the same across my windows build and linux build at HEAD.
Other main difference is the IDs I get are the classic DXC ids (named, so for example the entrypoints parameters are "%5 %globals %payload_0" and not just numerical IDs.
How did you get those SPIR-V disassemblies?
Nielsbishere
commented
1 month ago The arguments in my case:
dxc -I "D:/programming/repos/rt_core/res/shaders" "D:/programming/repos/rt_core/res/shaders/test.hlsl" -D__OXC3 -Zpc -O3 -HV 202x -Wconversion -Wdouble-promotion -spirv -fvk-use-dx-layout -fspv-target-env=vulkan1.2 -T lib_6_5 -D__OXC3_MAJOR=0 -D__OXC3_MINOR=2 -D__OXC3_PATCH=0 -D__OXC3_VERSION=8192 -Fo D:\programming\repos\rt_core\res\shaders\test.spv
I'll investigate what I get with the standalone tool as well. Because I run it via dxcompiler, but in theory that should be the same as CLI I guess. The spirv was also stripped by running it through spirv-tools:
spvtools::Optimizer optimizer{ SPV_ENV_VULKAN_1_2 };
optimizer.RegisterPassesFromFlags({ "-O", "--legalize-hlsl" });
optimizer.RegisterPass(spvtools::CreateStripDebugInfoPass()).RegisterPass(spvtools::CreateStripReflectInfoPass());
optimizer.Run((const U32*)resultPtr, binLen >> 2, &tmp);ala spirv-opt -O --legalize-hlsl --strip-reflect --strip-debug D:\programming\repos\rt_core\res\shaders\test.spv -o D:\programming\repos\rt_core\res\shaders\test0.spv
Disassembly was generated through:
spvtools::SpirvTools tool{ SPV_ENV_VULKAN_1_2 };
spv_binary_to_text_options_t opts = (spv_binary_to_text_options_t) (
SPV_BINARY_TO_TEXT_OPTION_FRIENDLY_NAMES |
SPV_BINARY_TO_TEXT_OPTION_NESTED_INDENT |
SPV_BINARY_TO_TEXT_OPTION_REORDER_BLOCKS |
SPV_BINARY_TO_TEXT_OPTION_COMMENT |
SPV_BINARY_TO_TEXT_OPTION_SHOW_BYTE_OFFSET |
SPV_BINARY_TO_TEXT_OPTION_INDENT
);
std::string str;
if(!tool.Disassemble((const U32*)resultPtr, binLen >> 2, &str, opts))
retError(clean, Error_invalidOperation(0, "Compiler_createDisassembly() SPIRV couldn't be disassembled"))Or spirv-dis D:\programming\repos\rt_core\res\shaders\test0.spv --offsets --comment though that doesn't seem to know vulkan 1.2 target.
It could ofc be possible that when stripping, it becomes inconsistent due to spirv-tools. So I'll do further investigation here.
Nielsbishere
commented
1 month ago With the above repro steps, I was able to reproduce it, even with just DXC from the latest release (and without the spirv-opt step in between). Using latest vulkan sdk on ubuntu 22.04 and windows 10:
; SPIR-V
; Version: 1.5
; Generator: Google spiregg; 0
; Bound: 41
; Schema: 0
OpCapability RayTracingKHR ; 0x00000014
OpExtension "SPV_KHR_ray_tracing" ; 0x0000001c
OpMemoryModel Logical GLSL450 ; 0x00000034
OpEntryPoint MissKHR %mainMiss "mainMiss" %globals %payload ; 0x00000040
OpEntryPoint ClosestHitKHR %mainClosestHit "mainClosestHit" %5 %globals %payload_0 ; 0x00000060
; Debug Information
OpSource HLSL 650 ; 0x00000088
OpName %type_globals "type.globals" ; 0x00000094, id %7
OpMemberName %type_globals 0 "_frameId" ; 0x000000ac
OpMemberName %type_globals 1 "_time" ; 0x000000c4
OpMemberName %type_globals 2 "_deltaTime" ; 0x000000d8
OpMemberName %type_globals 3 "_swapchainCount" ; 0x000000f0
OpMemberName %type_globals 4 "_swapchains" ; 0x0000010c
OpMemberName %type_globals 5 "_appData" ; 0x00000124
OpName %globals "globals" ; 0x0000013c, id %2
OpName %ColorPayload "ColorPayload" ; 0x0000014c, id %8
OpMemberName %ColorPayload 0 "color" ; 0x00000164
OpMemberName %ColorPayload 1 "hitT" ; 0x00000178
OpName %payload "payload" ; 0x0000018c, id %3
OpName %payload_0 "payload" ; 0x0000019c, id %6
OpName %mainMiss "mainMiss" ; 0x000001ac, id %1
OpName %mainClosestHit "mainClosestHit" ; 0x000001c0, id %4
; Annotations
OpDecorate %5 BuiltIn RayTmaxKHR ; 0x000001d8
OpDecorate %globals DescriptorSet 2 ; 0x000001e8
OpDecorate %globals Binding 0 ; 0x000001f8
OpDecorate %_arr_v4uint_uint_8 ArrayStride 16 ; 0x00000208
OpDecorate %_arr_v4uint_uint_23 ArrayStride 16 ; 0x00000218
OpMemberDecorate %type_globals 0 Offset 0 ; 0x00000228
OpMemberDecorate %type_globals 1 Offset 4 ; 0x0000023c
OpMemberDecorate %type_globals 2 Offset 8 ; 0x00000250
OpMemberDecorate %type_globals 3 Offset 12 ; 0x00000264
OpMemberDecorate %type_globals 4 Offset 16 ; 0x00000278
OpMemberDecorate %type_globals 5 Offset 144 ; 0x0000028c
OpDecorate %type_globals Block ; 0x000002a0
; Types, variables and constants
%uint = OpTypeInt 32 0 ; 0x000002ac
%int = OpTypeInt 32 1 ; 0x000002bc
%float = OpTypeFloat 32 ; 0x000002cc
%float_n1 = OpConstant %float -1 ; 0x000002d8
%v4uint = OpTypeVector %uint 4 ; 0x000002e8
%int_5 = OpConstant %int 5 ; 0x000002f8
%uint_8 = OpConstant %uint 8 ; 0x00000308
%_arr_v4uint_uint_8 = OpTypeArray %v4uint %uint_8 ; 0x00000318, ArrayStride 16
%uint_23 = OpConstant %uint 23 ; 0x00000328
%_arr_v4uint_uint_23 = OpTypeArray %v4uint %uint_23 ; 0x00000338, ArrayStride 16
%type_globals = OpTypeStruct %uint %float %float %uint %_arr_v4uint_uint_8 %_arr_v4uint_uint_23 ; 0x00000348, Block
%_ptr_Uniform_type_globals = OpTypePointer Uniform %type_globals ; 0x00000368
%v3float = OpTypeVector %float 3 ; 0x00000378
%ColorPayload = OpTypeStruct %v3float %float ; 0x00000388
%_ptr_IncomingRayPayloadKHR_ColorPayload = OpTypePointer IncomingRayPayloadKHR %ColorPayload ; 0x00000398
%_ptr_Input_float = OpTypePointer Input %float ; 0x000003a8
%void = OpTypeVoid ; 0x000003b8
%24 = OpTypeFunction %void ; 0x000003c0
%v3uint = OpTypeVector %uint 3 ; 0x000003cc
%_ptr_Uniform_v4uint = OpTypePointer Uniform %v4uint ; 0x000003dc
%globals = OpVariable %_ptr_Uniform_type_globals Uniform ; 0x000003ec, DescriptorSet 2, Binding 0
%payload = OpVariable %_ptr_IncomingRayPayloadKHR_ColorPayload IncomingRayPayloadKHR ; 0x000003fc
%payload_0 = OpVariable %_ptr_IncomingRayPayloadKHR_ColorPayload IncomingRayPayloadKHR ; 0x0000040c
%5 = OpVariable %_ptr_Input_float Input ; 0x0000041c, BuiltIn RayTmaxKHR
%uint_3 = OpConstant %uint 3 ; 0x0000042c
; Function mainMiss
%mainMiss = OpFunction %void None %24 ; 0x0000043c
%28 = OpLabel ; 0x00000450
%29 = OpAccessChain %_ptr_Uniform_v4uint %globals %int_5 %uint_3 ; 0x00000458
%30 = OpLoad %v4uint %29 ; 0x00000470
%31 = OpVectorShuffle %v3uint %30 %30 0 1 2 ; 0x00000480
%32 = OpBitcast %v3float %31 ; 0x000004a0
%33 = OpCompositeConstruct %ColorPayload %32 %float_n1 ; 0x000004b0
OpStore %payload %33 ; 0x000004c4
OpReturn ; 0x000004d0
OpFunctionEnd ; 0x000004d4
; Function mainClosestHit
%mainClosestHit = OpFunction %void None %24 ; 0x000004d8
%34 = OpLabel ; 0x000004ec
%35 = OpAccessChain %_ptr_Uniform_v4uint %globals %int_5 %uint_3 ; 0x000004f4
%36 = OpLoad %v4uint %35 ; 0x0000050c
%37 = OpVectorShuffle %v3uint %36 %36 0 1 2 ; 0x0000051c
%38 = OpBitcast %v3float %37 ; 0x0000053c
%39 = OpLoad %float %5 ; 0x0000054c
%40 = OpCompositeConstruct %ColorPayload %38 %39 ; 0x0000055c
OpStore %payload_0 %40 ; 0x00000570
OpReturn ; 0x0000057c
OpFunctionEnd ; 0x00000580vs
; SPIR-V
; Version: 1.5
; Generator: Google spiregg; 0
; Bound: 41
; Schema: 0
OpCapability RayTracingKHR ; 0x00000014
OpExtension "SPV_KHR_ray_tracing" ; 0x0000001c
OpMemoryModel Logical GLSL450 ; 0x00000034
OpEntryPoint MissKHR %mainMiss "mainMiss" %payload %globals ; 0x00000040
OpEntryPoint ClosestHitKHR %mainClosestHit "mainClosestHit" %5 %globals %payload_0 ; 0x00000060
; Debug Information
OpSource HLSL 650 ; 0x00000088
OpName %type_globals "type.globals" ; 0x00000094, id %7
OpMemberName %type_globals 0 "_frameId" ; 0x000000ac
OpMemberName %type_globals 1 "_time" ; 0x000000c4
OpMemberName %type_globals 2 "_deltaTime" ; 0x000000d8
OpMemberName %type_globals 3 "_swapchainCount" ; 0x000000f0
OpMemberName %type_globals 4 "_swapchains" ; 0x0000010c
OpMemberName %type_globals 5 "_appData" ; 0x00000124
OpName %globals "globals" ; 0x0000013c, id %3
OpName %ColorPayload "ColorPayload" ; 0x0000014c, id %8
OpMemberName %ColorPayload 0 "color" ; 0x00000164
OpMemberName %ColorPayload 1 "hitT" ; 0x00000178
OpName %payload "payload" ; 0x0000018c, id %2
OpName %payload_0 "payload" ; 0x0000019c, id %6
OpName %mainMiss "mainMiss" ; 0x000001ac, id %1
OpName %mainClosestHit "mainClosestHit" ; 0x000001c0, id %4
; Annotations
OpDecorate %5 BuiltIn RayTmaxKHR ; 0x000001d8
OpDecorate %globals DescriptorSet 2 ; 0x000001e8
OpDecorate %globals Binding 0 ; 0x000001f8
OpDecorate %_arr_v4uint_uint_8 ArrayStride 16 ; 0x00000208
OpDecorate %_arr_v4uint_uint_23 ArrayStride 16 ; 0x00000218
OpMemberDecorate %type_globals 0 Offset 0 ; 0x00000228
OpMemberDecorate %type_globals 1 Offset 4 ; 0x0000023c
OpMemberDecorate %type_globals 2 Offset 8 ; 0x00000250
OpMemberDecorate %type_globals 3 Offset 12 ; 0x00000264
OpMemberDecorate %type_globals 4 Offset 16 ; 0x00000278
OpMemberDecorate %type_globals 5 Offset 144 ; 0x0000028c
OpDecorate %type_globals Block ; 0x000002a0
; Types, variables and constants
%uint = OpTypeInt 32 0 ; 0x000002ac
%int = OpTypeInt 32 1 ; 0x000002bc
%float = OpTypeFloat 32 ; 0x000002cc
%float_n1 = OpConstant %float -1 ; 0x000002d8
%v4uint = OpTypeVector %uint 4 ; 0x000002e8
%int_5 = OpConstant %int 5 ; 0x000002f8
%uint_8 = OpConstant %uint 8 ; 0x00000308
%_arr_v4uint_uint_8 = OpTypeArray %v4uint %uint_8 ; 0x00000318, ArrayStride 16
%uint_23 = OpConstant %uint 23 ; 0x00000328
%_arr_v4uint_uint_23 = OpTypeArray %v4uint %uint_23 ; 0x00000338, ArrayStride 16
%type_globals = OpTypeStruct %uint %float %float %uint %_arr_v4uint_uint_8 %_arr_v4uint_uint_23 ; 0x00000348, Block
%_ptr_Uniform_type_globals = OpTypePointer Uniform %type_globals ; 0x00000368
%v3float = OpTypeVector %float 3 ; 0x00000378
%ColorPayload = OpTypeStruct %v3float %float ; 0x00000388
%_ptr_IncomingRayPayloadKHR_ColorPayload = OpTypePointer IncomingRayPayloadKHR %ColorPayload ; 0x00000398
%_ptr_Input_float = OpTypePointer Input %float ; 0x000003a8
%void = OpTypeVoid ; 0x000003b8
%24 = OpTypeFunction %void ; 0x000003c0
%v3uint = OpTypeVector %uint 3 ; 0x000003cc
%_ptr_Uniform_v4uint = OpTypePointer Uniform %v4uint ; 0x000003dc
%globals = OpVariable %_ptr_Uniform_type_globals Uniform ; 0x000003ec, DescriptorSet 2, Binding 0
%payload = OpVariable %_ptr_IncomingRayPayloadKHR_ColorPayload IncomingRayPayloadKHR ; 0x000003fc
%payload_0 = OpVariable %_ptr_IncomingRayPayloadKHR_ColorPayload IncomingRayPayloadKHR ; 0x0000040c
%5 = OpVariable %_ptr_Input_float Input ; 0x0000041c, BuiltIn RayTmaxKHR
%uint_3 = OpConstant %uint 3 ; 0x0000042c
; Function mainMiss
%mainMiss = OpFunction %void None %24 ; 0x0000043c
%28 = OpLabel ; 0x00000450
%29 = OpAccessChain %_ptr_Uniform_v4uint %globals %int_5 %uint_3 ; 0x00000458
%30 = OpLoad %v4uint %29 ; 0x00000470
%31 = OpVectorShuffle %v3uint %30 %30 0 1 2 ; 0x00000480
%32 = OpBitcast %v3float %31 ; 0x000004a0
%33 = OpCompositeConstruct %ColorPayload %32 %float_n1 ; 0x000004b0
OpStore %payload %33 ; 0x000004c4
OpReturn ; 0x000004d0
OpFunctionEnd ; 0x000004d4
; Function mainClosestHit
%mainClosestHit = OpFunction %void None %24 ; 0x000004d8
%34 = OpLabel ; 0x000004ec
%35 = OpAccessChain %_ptr_Uniform_v4uint %globals %int_5 %uint_3 ; 0x000004f4
%36 = OpLoad %v4uint %35 ; 0x0000050c
%37 = OpVectorShuffle %v3uint %36 %36 0 1 2 ; 0x0000051c
%38 = OpBitcast %v3float %37 ; 0x0000053c
%39 = OpLoad %float %5 ; 0x0000054c
%40 = OpCompositeConstruct %ColorPayload %38 %39 ; 0x0000055c
OpStore %payload_0 %40 ; 0x00000570
OpReturn ; 0x0000057c
OpFunctionEnd ; 0x00000580
s-perron
commented
1 month ago After reading the comments, my guess is that the issue is that we traverse a data structure whose order is not deterministic. Memory layout could affect the result. Since we have a good clue as to what is different: the order of the operands to an OpEntryPoint instructions.
I started tracing how that is generated, and I noticed that on Linux, the order of the operands is changed at some point. The the OpEntryPoint instruction is generate for the Miss shader, the operands are originally "%global %payload", and then they are flipped. I followed to where it was flipped, and found that is happens in the RemoveUnusedInterfaceVariablesPass in spirv-opt.
When I traced that code I found an unordered set that is traversed, and that is used to regenerate the operands in the OpEntryPoint instruction. That is most likely the problem. @Keenuts can you get that fixed?
s-perron
commented
1 month ago And even with the binary output I noticed the output is actually the same across my windows build and linux build at HEAD. Other main difference is the IDs I get are the classic DXC ids (named, so for example the entrypoints parameters are "%5 %globals %payload_0" and not just numerical IDs.
You can get the ids from spir-dis using the --raw-id option.
Keenuts
commented
6 days ago Thanks for looking into this! I sent the PR in spirv-tools and will integrate here once merged.
Keenuts
commented
5 days ago Send a PR to pull in the fixed SPIRV-Tools. Let me know if this fixes your issue once merged!
Nielsbishere
commented
5 days ago Awesome. Currently working on my shader reflection system. Will let you know once it's running again.
Nielsbishere
commented
2 days ago
Confirmed; binaries with debug info stripped match 1:1 for me on linux & windows 🥂 haven't been able to test mac yet, but I think it's the same.
Description When compiling across linux, OSX and windows, it seems the spirv register ids don't match, though the functionality of the shaders seems identical. This reordering of ids makes the binaries not byte for byte identical, which could be annoying if shaders are recompiled and seen as dirty, even though they shouldn't be (when switching between platforms).
Steps to Reproduce Compiling the following raytracing shader:
On Windows and Linux produces the following spirv disassemblies:
vs
The two seem very similar, but by running a diff, it's clear they're not the same:
Actual Behavior SPIRV generated should be byte for byte identical between compiles on different machines, since reflection and debug data is stripped. DXIL is byte for byte equal and SPIRV for non RT shaders (graphics and compute) seems to be fine. Important note: When I tried to repro this with 1 miss or 1 closesthit shader (rather than combining the two), it matches byte for byte. So it might be related to multiple entrypoint types sharing the same library.
Environment