godotengine / godot

Godot Engine – Multi-platform 2D and 3D game engine
https://godotengine.org
MIT License
89.51k stars 20.26k forks source link

Incorrect compilation of shaders in Forward Mobile causes it + D3D12 crash Godot #93249

Closed MBCX closed 2 months ago

MBCX commented 3 months ago

Tested versions

System information

Godot v4.3.beta (e5b8ba7f2) - Windows 10.0.22631 - Vulkan (Mobile) - dedicated Radeon RX 580 Series (Advanced Micro Devices, Inc.; 31.0.21912.14) - Intel(R) Core(TM) i5-9600K CPU @ 3.70GHz (6 Threads)

Issue description

I was working on a game of mine, and I picked the Forward Mobile backend. In this project, I have incorporated 2 complex shaders, A Moebius art-style shader, and a beautiful distance-based pointillism (with depth).

When I change the driver to use D3D12 instead of Vulkan, initially nothing happens. However, when Godot tries to either compile the aforementioned shaders or (in my testing) even double clicking on the shader file to open it, makes the editor crash. When I try to open the project (with the setting set) it crashes silently, and the project cannot be open unless the rendering_device/driver.windows property is set back to vulkan.

It also gives me a backtrace that doesn't look useful, but here it is anyway.

.\windows_editor_x86_64_console.exe --rendering-driver d3d12 --rendering-method mobile --export-release "Windows x86_64" "E:\Repos\fan-a-game\build\windows\x86_64"
================================================================
CrashHandlerException: Program crashed with signal 11
Engine version: Godot Engine v4.3.beta.custom_build (e5b8ba7f2d747ec461dd782357b5f1b26e49b68a)
Dumping the backtrace. Please include this when reporting the bug to the project developer.
[1] error(-1): no debug info in PE/COFF executable
[2] error(-1): no debug info in PE/COFF executable
[3] error(-1): no debug info in PE/COFF executable
[4] error(-1): no debug info in PE/COFF executable
[5] error(-1): no debug info in PE/COFF executable
[6] error(-1): no debug info in PE/COFF executable
[7] error(-1): no debug info in PE/COFF executable
[8] error(-1): no debug info in PE/COFF executable
[9] error(-1): no debug info in PE/COFF executable
[10] error(-1): no debug info in PE/COFF executable
[11] error(-1): no debug info in PE/COFF executable
[12] error(-1): no debug info in PE/COFF executable
[13] error(-1): no debug info in PE/COFF executable
-- END OF BACKTRACE --
================================================================

I will have to test this further because not all complex shaders make Godot crash. This one for example has a lot of stuff going yet it doesn't crash in my testing, even after converting it to 3D.

Steps to reproduce

  1. Clone Moebius art-style shader project
  2. Import it.
  3. Change the rendering method from Forward+ to Mobile, then set the driver in Project -> Project Settings -> Rendering Device, change Driver.windows to D3D12.
  4. Open moebius.gdshader
  5. There will be a error of hint_normal_roughness_texture is only available when using the Forward+ backend. remove that feature.
  6. Wait for crash.

Minimal reproduction project (MRP)

Moebius art-style shader project

akien-mga commented 3 months ago

CC @RandomShaper @DarioSamo

matheusmdx commented 3 months ago

I followed the steps to reproduce the bug but in my case the crash happens after set the rendering device to d3d12 and restart the editor so i'm not sure if my crash is exactly the same cause, also if i start using --rendering-driver d3d12 imediatly crash the engine even if i don't try open any project. Here the errors + backtrace:

PS C:\Users\Matheus\Downloads\Godot Source\bin> .\godot.windows.editor.dev.x86_64.exe --rendering-driver d3d12 --debug
PS C:\Users\Matheus\Downloads\Godot Source\bin> Godot Engine v4.3.beta.custom_build.170af05c8 (2024-06-17 11:30:20 UTC) - https://godotengine.org
D3D12 12_0 - Forward+ - Using Device #0: AMD - AMD Radeon RX 580 2048SP
ERROR: Can't create buffer of size: 262144, error 0x80070057.
   at: (drivers\d3d12\rendering_device_driver_d3d12.cpp:917)
ERROR: Condition "!block.driver_id" is true. Returning: ERR_CANT_CREATE
   at: RenderingDevice::_insert_staging_block (servers\rendering\rendering_device.cpp:242)
ERROR: Condition "err != OK" is true. Continuing.
   at: RenderingDevice::initialize (servers\rendering\rendering_device.cpp:5510)
ERROR: Can't create buffer of size: 262144, error 0x80070057.
   at: (drivers\d3d12\rendering_device_driver_d3d12.cpp:917)
ERROR: Condition "!block.driver_id" is true. Returning: ERR_CANT_CREATE
   at: RenderingDevice::_insert_staging_block (servers\rendering\rendering_device.cpp:242)
ERROR: Condition "err != OK" is true. Continuing.
   at: RenderingDevice::initialize (servers\rendering\rendering_device.cpp:5510)
WARNING: PSO caching is not implemented yet in the Direct3D 12 driver.
     at: RenderingDeviceDriverD3D12::pipeline_cache_create (drivers\d3d12\rendering_device_driver_d3d12.cpp:4941)
ERROR: FATAL: Index p_index = 0 is out of bounds (size() = 0).
   at: CowData<struct RenderingDevice::StagingBufferBlock>::get (C:\Users\Matheus\Downloads\Godot Source\core/templates/cowdata.h:205)

================================================================
CrashHandlerException: Program crashed
Engine version: Godot Engine v4.3.beta.custom_build (170af05c8022ad3e0cefb287ca0cb3790c8747f6)
Dumping the backtrace. Please include this when reporting the bug to the project developer.
[0] CowData<RenderingDevice::StagingBufferBlock>::get (C:\Users\Matheus\Downloads\Godot Source\core\templates\cowdata.h:205)
[1] Vector<RenderingDevice::StagingBufferBlock>::operator[] (C:\Users\Matheus\Downloads\Godot Source\core\templates\vector.h:97)
[2] RenderingDevice::_staging_buffer_allocate (C:\Users\Matheus\Downloads\Godot Source\servers\rendering\rendering_device.cpp:261)
[3] RenderingDevice::_texture_update (C:\Users\Matheus\Downloads\Godot Source\servers\rendering\rendering_device.cpp:1228)
[4] RenderingDevice::texture_create (C:\Users\Matheus\Downloads\Godot Source\servers\rendering\rendering_device.cpp:826)
[5] RendererRD::TextureStorage::TextureStorage (C:\Users\Matheus\Downloads\Godot Source\servers\rendering\renderer_rd\storage_rd\texture_storage.cpp:114)
[6] RendererCompositorRD::RendererCompositorRD (C:\Users\Matheus\Downloads\Godot Source\servers\rendering\renderer_rd\renderer_compositor_rd.cpp:305)
[7] RendererCompositorRD::_create_current (C:\Users\Matheus\Downloads\Godot Source\servers\rendering\renderer_rd\renderer_compositor_rd.h:138)
[8] RendererCompositor::create (C:\Users\Matheus\Downloads\Godot Source\servers\rendering\renderer_compositor.cpp:43)
[9] RenderingServerDefault::_init (C:\Users\Matheus\Downloads\Godot Source\servers\rendering\rendering_server_default.cpp:220)
[10] RenderingServerDefault::init (C:\Users\Matheus\Downloads\Godot Source\servers\rendering\rendering_server_default.cpp:261)
[11] Main::setup2 (C:\Users\Matheus\Downloads\Godot Source\main\main.cpp:2833)
[12] Main::setup (C:\Users\Matheus\Downloads\Godot Source\main\main.cpp:2465)
[13] widechar_main (C:\Users\Matheus\Downloads\Godot Source\platform\windows\godot_windows.cpp:165)
[14] _main (C:\Users\Matheus\Downloads\Godot Source\platform\windows\godot_windows.cpp:206)
[15] main (C:\Users\Matheus\Downloads\Godot Source\platform\windows\godot_windows.cpp:220)
[16] WinMain (C:\Users\Matheus\Downloads\Godot Source\platform\windows\godot_windows.cpp:234)
[17] __scrt_common_main_seh (D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288)
[18] <couldn't map PC to fn name>
-- END OF BACKTRACE --
================================================================


If my case is a different bug please inform me and i'll open a separated issue

MBCX commented 3 months ago

@matheusmdx Yeah, I forgot to mention that as well. The editor also crashes after setting the rendering device to d3d12 and restarting the editor (though in your case, I see you are using the Forward+ method. Using Forward+ doesn't crash on my end).

MBCX commented 3 months ago

Also, it (could) be a separate issue. I can open any project, it's only when it tries to load it and the shaders that the crash occurs.

Zekinler commented 3 months ago

Same as @matheusmdx: Changing display/display_server/driver.windows to D3D12 causes the editor to crash after restarting.

Noiros commented 3 months ago

Same thing here, the crash append when we edit the project

Editing project: D:/Object-Wars
Godot Engine v4.3.beta2.official.b75f0485b - https://godotengine.org
D3D12 12_0 - Forward+ - Using Device #0: NVIDIA - NVIDIA GeForce GTX 1060
ERROR: Can't create buffer of size: 262144, error 0x80070057.
   at: (drivers/d3d12/rendering_device_driver_d3d12.cpp:917)
ERROR: Condition "!block.driver_id" is true. Returning: ERR_CANT_CREATE
   at: _insert_staging_block (servers/rendering/rendering_device.cpp:242)
ERROR: Condition "err != OK" is true. Continuing.
   at: initialize (servers/rendering/rendering_device.cpp:5510)
ERROR: Can't create buffer of size: 262144, error 0x80070057.
   at: (drivers/d3d12/rendering_device_driver_d3d12.cpp:917)
ERROR: Condition "!block.driver_id" is true. Returning: ERR_CANT_CREATE
   at: _insert_staging_block (servers/rendering/rendering_device.cpp:242)
ERROR: Condition "err != OK" is true. Continuing.
   at: initialize (servers/rendering/rendering_device.cpp:5510)
WARNING: PSO caching is not implemented yet in the Direct3D 12 driver.
     at: pipeline_cache_create (drivers/d3d12/rendering_device_driver_d3d12.cpp:4941)
ERROR: FATAL: Index p_index = 0 is out of bounds (size() = 0).
   at: get (./core/templates/cowdata.h:205)

================================================================
CrashHandlerException: Program crashed with signal 4
Engine version: Godot Engine v4.3.beta2.official (b75f0485ba15951b87f1d9a2d8dd0fcd55e178e4)
Dumping the backtrace. Please include this when reporting the bug to the project developer.
[1] error(-1): no debug info in PE/COFF executable
[2] error(-1): no debug info in PE/COFF executable
[3] error(-1): no debug info in PE/COFF executable
[4] error(-1): no debug info in PE/COFF executable
[5] error(-1): no debug info in PE/COFF executable
[6] error(-1): no debug info in PE/COFF executable
[7] error(-1): no debug info in PE/COFF executable
[8] error(-1): no debug info in PE/COFF executable
[9] error(-1): no debug info in PE/COFF executable
[10] error(-1): no debug info in PE/COFF executable
[11] error(-1): no debug info in PE/COFF executable
[12] error(-1): no debug info in PE/COFF executable
[13] error(-1): no debug info in PE/COFF executable
[14] error(-1): no debug info in PE/COFF executable
[15] error(-1): no debug info in PE/COFF executable
[16] error(-1): no debug info in PE/COFF executable
[17] error(-1): no debug info in PE/COFF executable
[18] error(-1): no debug info in PE/COFF executable
[19] error(-1): no debug info in PE/COFF executable
[20] error(-1): no debug info in PE/COFF executable
-- END OF BACKTRACE --
================================================================
RandomShaper commented 3 months ago

@matheusmdx's and @Noiros' crashes are due to D3D12 not being able to initialize. The way the situation is reported is cryptic indeed. We need to improve that. You may get further info with --verbose.

Regarding the MRP, I can open it successfully with the editor running on D3D12. I can also edit the shader and see it working. The only thing I need to change is the vertex function body to POSITION = vec4(VERTEX.xy, 1.0, 1.0); (due to the clip space change in 4.3).

Noiros commented 3 months ago

This is what Godot returns to me with --verbose


TextServer: Added interface "Dummy"

StringName: 137 unclaimed string names at exit.
ERROR: File 'D:\Object-Wars\addons\godot-jolt\windows/godot-jolt_windows-x64_editor.pdb' does not exist.
   at: (platform/windows/windows_utils.cpp:158)
WARNING: Failed to rename the PDB file. The original PDB file for 'D:\Object-Wars\addons\godot-jolt\windows/~godot-jolt_windows-x64_editor.dll' will be loaded.
     at: open_dynamic_library (platform/windows/os_windows.cpp:410)
Godot Engine v4.3.beta2.official.b75f0485b - https://godotengine.org
TextServer: Added interface "Dummy"
TextServer: Added interface "ICU / HarfBuzz / Graphite (Built-in)"
Devices:
  #0: NVIDIA NVIDIA GeForce GTX 1060 - Supported, Discrete
  #1: Intel Intel(R) HD Graphics 630 - Supported, Discrete
  #2: Microsoft Microsoft Basic Render Driver - Supported, CPU
- Shader:
  model: 6.5
- D3D12 Variable Rate Shading not supported
- D3D12 multiview supported:
  max view count: 4
- Relaxed casting not supported
- D3D12 16-bit ops supported: no
D3D12 12_0 - Forward+ - Using Device #0: NVIDIA - NVIDIA GeForce GTX 1060
ERROR: Can't create buffer of size: 262144, error 0x80070057.
   at: (drivers/d3d12/rendering_device_driver_d3d12.cpp:917)
ERROR: Condition "!block.driver_id" is true. Returning: ERR_CANT_CREATE
   at: _insert_staging_block (servers/rendering/rendering_device.cpp:242)
ERROR: Condition "err != OK" is true. Continuing.
   at: initialize (servers/rendering/rendering_device.cpp:5510)
ERROR: Can't create buffer of size: 262144, error 0x80070057.
   at: (drivers/d3d12/rendering_device_driver_d3d12.cpp:917)
ERROR: Condition "!block.driver_id" is true. Returning: ERR_CANT_CREATE
   at: _insert_staging_block (servers/rendering/rendering_device.cpp:242)
ERROR: Condition "err != OK" is true. Continuing.
   at: initialize (servers/rendering/rendering_device.cpp:5510)
WARNING: PSO caching is not implemented yet in the Direct3D 12 driver.
     at: pipeline_cache_create (drivers/d3d12/rendering_device_driver_d3d12.cpp:4941)
Using "winink" pen tablet driver...
ERROR: FATAL: Index p_index = 0 is out of bounds (size() = 0).
   at: get (./core/templates/cowdata.h:205)

================================================================
CrashHandlerException: Program crashed with signal 4
Engine version: Godot Engine v4.3.beta2.official (b75f0485ba15951b87f1d9a2d8dd0fcd55e178e4)
Dumping the backtrace. Please include this when reporting the bug to the project developer.
[1] error(-1): no debug info in PE/COFF executable
[2] error(-1): no debug info in PE/COFF executable
[3] error(-1): no debug info in PE/COFF executable
[4] error(-1): no debug info in PE/COFF executable
[5] error(-1): no debug info in PE/COFF executable
[6] error(-1): no debug info in PE/COFF executable
[7] error(-1): no debug info in PE/COFF executable
[8] error(-1): no debug info in PE/COFF executable
[9] error(-1): no debug info in PE/COFF executable
[10] error(-1): no debug info in PE/COFF executable
[11] error(-1): no debug info in PE/COFF executable
[12] error(-1): no debug info in PE/COFF executable
[13] error(-1): no debug info in PE/COFF executable
[14] error(-1): no debug info in PE/COFF executable
[15] error(-1): no debug info in PE/COFF executable
[16] error(-1): no debug info in PE/COFF executable
[17] error(-1): no debug info in PE/COFF executable
[18] error(-1): no debug info in PE/COFF executable
[19] error(-1): no debug info in PE/COFF executable
[20] error(-1): no debug info in PE/COFF executable
-- END OF BACKTRACE --
================================================================```
MBCX commented 3 months ago

@RandomShaper As said previously, with Forward+ method and D3D12 the MRP works just fine (and also applying the position fix). The crash occurs when you switch the method to Mobile while the driver is D3D12. It'll crash as soon as the engine tries to load the scene with the shader applied (and sometimes, the crash is before the scene loads).

MBCX commented 3 months ago

And here is the log loading the MRP before the crash using --verbose:

./windows_editor_x86_64_console.exe --path "E:\Repos\Github\moebius_shader" --rendering-method "mobile" --verbose --rendering-driver "d3d12" --editor >> log.txt
Godot Engine v4.3.beta.custom_build.4d229ce00 (2024-06-19 12:50:19 UTC) - https://godotengine.org
TextServer: Added interface "Dummy"
TextServer: Added interface "ICU / HarfBuzz / Graphite (Built-in)"
TextServer: Added interface "Fallback (Built-in)"
Devices:
  #0: AMD Radeon RX 580 Series - Supported, Discrete
  #1: Microsoft Microsoft Basic Render Driver - Supported, CPU
- Shader:
  model: 6.6
- D3D12 Variable Rate Shading not supported
- D3D12 multiview supported:
  max view count: 4
- Relaxed casting not supported
- D3D12 Mesh Shader not supported
- D3D12 16-bit ops supported: no
D3D12 12_0 - Forward Mobile - Using Device #0: AMD - Radeon RX 580 Series
Using "winink" pen tablet driver...
Shader 'CanvasSdfShaderRD' (group 0) SHA256: 7a2e5523d31ce900e4bf173d5c6a77c726fb496ff6fa15c1858372f4cf0f79a5
Shader 'SkeletonShaderRD' (group 0) SHA256: 290411bd75e0bc0a4d9a6d72c826e3e0226093368f11e1f546d7b8b771c7d925
Shader 'SortShaderRD' (group 0) SHA256: ee7ad584cae0225345c8d285b96d6083a1ce54d30962256a12d5b70f939bac02
Shader 'ParticlesShaderRD' (group 0) SHA256: e33a6ad1402728e87c3e35752a5a903a8d8e3d0eaa930399922f05caba3cff32
Shader 'ParticlesCopyShaderRD' (group 0) SHA256: e821103a9f28223025d26ebac50eee67f0d677dac58f878588a9924a5e831751
Shader 'CanvasShaderRD' (group 0) SHA256: 21816333e18e90a622ca67400a30073cd1a422943b3aae661ed5e530cc3ed7e7
Shader 'CanvasOcclusionShaderRD' (group 0) SHA256: 9829a8610307863ce4a1613677ef0d725a5ebbc8a4cc3fd281042df58a355bda
Shader 'SceneForwardMobileShaderRD' (group 0) SHA256: 51dd1b422d6e85754988ca916f6040f1afe0a3d064c811a69c7ac4cb3d7d9186
Shader 'SkyShaderRD' (group 0) SHA256: 2d3ac217f2eedeb2ed42fda6c7a44fc87f8a5327af667c758b7e6ebe6f8efd0e
Shader 'BokehDofRasterShaderRD' (group 0) SHA256: ca5b05ff88388e00e006bb248efc82a676417afebaad9bd37887bf8239937273
Shader 'BlurRasterShaderRD' (group 0) SHA256: 9a455a922abb85aebc79eb736f5b293994b45d5e87b96389e12c03855191dd54
Shader 'CopyShaderRD' (group 0) SHA256: 9d330c9bef965150725bb9674e5b32cb748e5c82cd35c7c2d30a23e1249c395e
Shader 'CopyToFbShaderRD' (group 0) SHA256: 6f5c0718d9d5f02895be6fd956ed5f6e61026de385fbd9f10a3e87a11923afbf
Shader 'CubeToDpShaderRD' (group 0) SHA256: 2dd5a2ab67d4fcaff5c3d4e96fcc242c5e3d65beb694329d1278110fe268449c
Shader 'CubemapDownsamplerRasterShaderRD' (group 0) SHA256: 9396cc3fb6e4fc2f33478c86a61cf74e833e8ff8c6c46375c39aedffc74bf1ae
Shader 'CubemapFilterRasterShaderRD' (group 0) SHA256: beb28f5db457cf4876695b97f3f36b990eaaae65ad495b2c1feee6c59f0deac8
Shader 'CubemapRoughnessRasterShaderRD' (group 0) SHA256: c32d71025bd91a52186a47c189ab35f845fbf1fea46220fe9893b07972f19ad2
Shader 'SpecularMergeShaderRD' (group 0) SHA256: 43654185cf3e97ad3bf1cf234b75224a2ed3af5d63dc84628cdcebec2722e758
Shader 'ShadowFrustumShaderRD' (group 0) SHA256: 587bffaca6514aa4588a8e9c3ff9949a843d7501364bc4e3ea9e0ea823f68f0c
Shader 'MotionVectorsShaderRD' (group 0) SHA256: 425b000acd6c6a0919175cdf2159b0bc77742a2ad003fa31841bd3384199a4d1
Shader 'LuminanceReduceRasterShaderRD' (group 0) SHA256: d8503a63f9cdcf9ad6e5f0a159fd25b320671ea1af990ea1225bf013935cfc74
Shader 'TonemapShaderRD' (group 0) SHA256: 610278b8e790acdd51230295ff7d71d919faeabcbe21a953a5afd228ef40781a
Shader 'BlitShaderRD' (group 0) SHA256: f82f0a262b6784d38bc572e7c300471eb2cce7f5a476a67e4d63eae160e68fb5
WASAPI: Activated output_device using IAudioClient3 interface
WASAPI: wFormatTag = 65534
WASAPI: nChannels = 2
WASAPI: nSamplesPerSec = 48000
WASAPI: nAvgBytesPerSec = 384000
WASAPI: nBlockAlign = 8
WASAPI: wBitsPerSample = 32
WASAPI: cbSize = 22
WASAPI: mix_rate = 48000
WASAPI: fundamental_period_frames = 480
WASAPI: min_period_frames = 480
WASAPI: max_period_frames = 480
WASAPI: selected a period frame size of 480
WASAPI: detected 2 channels
WASAPI: audio buffer frames: 480 calculated latency: 10ms

Using swap chain flags: 0, sync interval: 1, present flags: 0
TextServer: Primary interface set to: "ICU / HarfBuzz / Graphite (Built-in)".
CORE API HASH: 2829119556
EDITOR API HASH: 3571171096
EditorSettings: Load OK!
EditorTheme: Generating new theme for the config '66886095'.
EditorTheme: Generating new icons.
EditorTheme: Generating new fonts.
EditorTheme: Generating new styles.
Loaded system CA certificates
EditorSettings: Save OK!
Using swap chain flags: 0, sync interval: 1, present flags: 0
akien-mga commented 3 months ago

Moved the issue about failing to initialized D3D12 to #93670.

MBCX commented 3 months ago

I found where it crashes trying to load MRP with Forward Mobile

image

And the call stack image

I guess I'll asume that the shaders (and features) compiled by Forward Mobile sometimes don't play nicely with the D3D12 driver.

MBCX commented 3 months ago

I will update the title of this issue to be more precise after my findings.

clayjohn commented 2 months ago

After compiling with #93707 I can't reproduce this crash.

In master, I also couldn't reproduce this crash as I was getting the same crash at startup as reported in https://github.com/godotengine/godot/issues/93249#issuecomment-2173465395

@MBCX Can you try testing with #93707 and see if it fixes the crash for you? It fixes an issue with reading textures back from the GPU. The editor uses this to display resource previews. So that could explain why you didn't get the crash until you selected the shader.

For more context. The project manager doesn't read back any textures, so that explains why it didn't crash in the project manager for you (but did crash in editor). Also, many of the texture readbacks that the editor does get cached to disk. Which might explain why it crashed on startup for many of us, but not for you (as presumably you were working on this project before the crash was introduced and so you had a cached local file). Since the shader can change each frame, the resource preview isn't cached, so it reliably led to the crash.

Anyway, that's my theory. Please test and confirm!

RandomShaper commented 2 months ago

Regarding the shader compilation crash itself, I've been able to reproduce it and I'm working on a fix.

RandomShaper commented 2 months ago

The mobile shader turned out to trigger the usage of a SPIR-V opcode that was left as a to do in the patch for SPIR-V-to-DXIL dependency. https://github.com/godotengine/godot-nir-static/pull/14 should do the trick.

akien-mga commented 2 months ago

I made a test build using the fixed mesa-nir library from https://github.com/godotengine/godot-nir-static/pull/14

https://downloads.tuxfamily.org/godotengine/testing/Godot_v4.3-beta-nir-pr14_win64.exe.zip

@MBCX @RandomShaper Could you confirm this official build solves it? That would confirm 4.3-beta3 will be properly fixed too.

MBCX commented 2 months ago

@akien-mga Yup, with that build the MRP now opens with D3D12 + Mobile without crashing! (and after applying the fixes for 4.3)