YoYoGames / GameMaker-Bugs

Public tracking for GameMaker bugs
22 stars 8 forks source link

Shader Parser hangs with these shaders #6342

Closed mMcFab closed 1 month ago

mMcFab commented 3 months ago

Description

Included are 2 shaders. A GLSL ES shader, and a direct HLSL11 port of the shader (yeah, the shaders aren't super tidy or fully optimised yet, but that's not the point) The GLSL ES shader successfully transpiles to HLSL (the HLSL version included is just a simplified version of the automatic conversion done by GameMaker) When compiling the game with either one of these shaders, the build process hangs on the D3D11ShaderParser step

Sometimes when I add stuff to the shader, it hangs for less time, sometimes when I reorganize it to make it simpler the hangs last seemingly forever. I believe the shaders I've included are in the "infinite hang" state. I have tried restructuring, alternative methods, but as soon as I reach something that would produce the result I'm after, the hang occurs

After trying to figure out if it was me or the compiler, I determined that the problem stems from the automatic shader compile target option - it hangs during the iteration over the shader models, and I'm not sure which one or why, I only know that specifying the ps_4_0 shader target stops the hang completely

If I manually invoke D3D11ShaderParser.exe but replace "-profilep ps_auto" with "-profilep ps_4_0" the hang doesn't happen and the shader builds in like 2 seconds (of course, I cannot use the manually built shader so this doesn't really help outside of debugging)

My hope for a solution is something of the following:

Although it may be possible to skip the hanging ps_4_0_level_x_x steps entirely since the games require minimum DX11 and I don't think they're necessary? (at least, after looking at this: https://learn.microsoft.com/en-us/windows/win32/direct3dhlsl/specifying-compiler-targets) But I'm not 100% sure, I could be confusing hardware vs software support

Steps To Reproduce

1) Run the "game" 2) See that it hangs for an extremely long time

Which version of GameMaker are you reporting this issue for?

IDE v2024.4.1.152 Runtime v2024.4.1.201

Which operating system(s) are you seeing the problem on?

Windows 10.0.19045.0

Which platform(s) are you seeing the problem on?

Windows

07f257a2-6100-4171-96e1-13e09fb7ebe9

mMcFab commented 3 months ago

I also just want to confirm I get the same hang in Beta IDE 2024.600.0.576 Runtime 2024.600.0.602

yymrennie commented 1 month ago

This has been fixed in 2024.10 - it still tries compiling with shader versions from the minimum supported on DX11 up, but now doesn't try building shaders without optimisations (this was an attempt to maximise compatibility but which seems to have caused an internal error in the D3D shader compilation library function with complex shaders).

mgeddesGM commented 1 week ago

verified as of Beta IDE v2024.1100.0.626 Runtime v2024.1100.0.652