search

godotengine / godot

Godot Engine – Multi-platform 2D and 3D game engine
https://godotengine.org
MIT License
90.44k stars 21.07k forks source link

Crash in intel gpu driver called by RasterizerStorageGLES2::_render_target_allocate #55395

Open Gromph opened 2 years ago

Gromph commented 2 years ago

Godot version

3.4 custom build

System information

Windows 10, GLES2

Issue description

We updated our games last week from godot 3.3.4 to 3.4. Since then we've had about a dozen of this crash reported by users to sentry.io, so I believe this is a new crash in 3.4 that wasn't there before. I have not been able to reproduce the crash.

This crash may be related to https://github.com/godotengine/godot/issues/55394

It is crashing instancing a scene. The scene contains a full screen ViewportContainer that contains a Viewport with a 3d Scene. It also has a label on top of everything that has a size 150 font, use Mipmaps is off.

Here is the raw call stack from sentry.io

OS Version: Windows 10.0.19041 (1348)
Report Version: 104

Crashed Thread: 4420

Application Specific Information:
Fatal Error: EXCEPTION_ACCESS_VIOLATION_READ

Thread 4420 Crashed:
0   ig4icd32.dll                    0x68cc5738          <unknown>
1   ig4icd32.dll                    0x68cc7aac          <unknown>
2   ig4icd32.dll                    0x68cc79fd          <unknown>
3   ig4icd32.dll                    0x68cc56cf          <unknown>
4   ig4icd32.dll                    0x689281a8          <unknown>
5   ig4icd32.dll                    0x68928250          <unknown>
6   ig4icd32.dll                    0x689288e0          <unknown>
7   ig4icd32.dll                    0x6892a61b          <unknown>
8   ig4icd32.dll                    0x6892467b          <unknown>
9   hearts.exe                      0x96f417            RasterizerStorageGLES2::_render_target_allocate (rasterizer_storage_gles2.cpp:5218)
10  hearts.exe                      0x1887eae           Concurrency::details::stl_critical_section_vista::lock (primitives.hpp:52)
11  hearts.exe                      0x19009c5           <unknown>
12  hearts.exe                      0x97df35            RasterizerStorageGLES2::render_target_set_size (rasterizer_storage_gles2.cpp:5389)
13  hearts.exe                      0x1227f1e           VisualServerViewport::viewport_set_size (visual_server_viewport.cpp:405)
14  hearts.exe                      0x113878c           VisualServerRaster::viewport_set_size (visual_server_raster.h:454)
15  hearts.exe                      0x115ceb1           VisualServerWrapMT::viewport_set_size (visual_server_wrap_mt.h:382)
16  hearts.exe                      0xaa588d            Viewport::set_size (viewport.cpp:721)
17  hearts.exe                      0xaa149e            MethodBind1<T>::call (method_bind.gen.inc:867)
18  hearts.exe                      0x16109a2           ClassDB::set_property (class_db.cpp:1007)
19  hearts.exe                      0x15a6529           Object::set (object.cpp:422)
20  hearts.exe                      0xaf872b            SceneState::instance (packed_scene.cpp:248)
21  hearts.exe                      0xaf772a            PackedScene::instance (packed_scene.cpp:1645)
22  hearts.exe                      0xaf7d21            SceneState::instance (packed_scene.cpp:113)
23  hearts.exe                      0xaf772a            PackedScene::instance (packed_scene.cpp:1645)
24  hearts.exe                      0x31be64            GodotLoadNode (GodotUtils.cpp:75)
25  hearts.exe                      0x832b76            STrickGame::OnFullScreenMsgLoaded (STrickGame.cpp:7496)
26  hearts.exe                      0x839248            STrickGame::OnResourceLoaded (STrickGame.cpp:7483)
27  hearts.exe                      0x44fa8e            SNotifyEntry::SendEvent (SNotifyEntry.cpp:118)
28  hearts.exe                      0x32128e            SEventObj::SendEventToNotifyList (SEventObj.cpp:225)
29  hearts.exe                      0x4588fd            SNode2D::_on_signal (SNode2D.cpp:91)
30  hearts.exe                      0x4589e2            MethodBindVarArg<T>::call (method_bind.h:333)
31  hearts.exe                      0x159efdb           Object::call (object.cpp:918)
32  hearts.exe                      0x15a1ba2           Object::emit_signal (object.cpp:1224)
33  hearts.exe                      0x159995e           Object::_emit_signal (object.cpp:1156)
34  hearts.exe                      0x19560a            MethodBindVarArg<T>::call (method_bind.h:333)
35  hearts.exe                      0x159efdb           Object::call (object.cpp:918)
36  hearts.exe                      0x163438f           MessageQueue::_call_function (message_queue.cpp:241)
37  hearts.exe                      0x16342a3           MessageQueue::_call_function (message_queue.cpp:234)
38  hearts.exe                      0x1887eb8           Concurrency::details::stl_critical_section_win7::lock (primitives.hpp:124)
39  hearts.exe                      0x18880b6           Concurrency::details::stl_critical_section_win7::unlock (primitives.hpp:137)
40  hearts.exe                      0x200b830           <unknown>
41  hearts.exe                      0x163461f           MessageQueue::flush (message_queue.cpp:284)
42  hearts.exe                      0xa7361f            SceneTree::iteration (scene_tree.cpp:486)
43  hearts.exe                      0x16b26b            Main::iteration (main.cpp:2163)
44  hearts.exe                      0x163b7a            OS_Windows::run (os_windows.cpp:3381)
45  hearts.exe                      0x154055            widechar_main (godot_windows.cpp:160)
46  hearts.exe                      0x153ef6            [inlined] _main (godot_windows.cpp:183)
47  hearts.exe                      0x153ef6            main (godot_windows.cpp:187)
48  hearts.exe                      0x1889e3d           [inlined] invoke_main (exe_common.inl:78)
49  hearts.exe                      0x1889e3d           __scrt_common_main_seh (exe_common.inl:288)
50  KERNEL32.DLL                    0x758afa28          BaseThreadInitThunk
51  ntdll.dll                       0x776c7a9d          __RtlUserThreadStart
52  ntdll.dll                       0x776c7a6d          _RtlUserThreadStart

Thread 7820
0   ntdll.dll                       0x776d470c          ZwWaitForWorkViaWorkerFactory
1   ntdll.dll                       0x77695b9f          TppWorkerThread
2   KERNEL32.DLL                    0x758afa28          BaseThreadInitThunk
3   ntdll.dll                       0x776c7a9d          __RtlUserThreadStart
4   ntdll.dll                       0x776c7a6d          _RtlUserThreadStart

Thread 8884
0   ntdll.dll                       0x776d2f8c          ZwWaitForMultipleObjects
1   KERNELBASE.dll                  0x766db1b2          WaitForMultipleObjectsEx
2   CoreMessaging.dll               0x743eb03d          Microsoft::CoreUI::Dispatch::WaitAdapter::Callback_WaitAny
3   CoreMessaging.dll               0x743cc3c4          Microsoft::CoreUI::Dispatch::WaitController::Callback_DoWait
4   CoreMessaging.dll               0x743cc27e          Microsoft::CoreUI::Dispatch::WaitController::Callback_DoGeneralWait
5   CoreMessaging.dll               0x743cc4f5          Microsoft::CoreUI::Dispatch::WaitController::Callback_OnDispatch
6   CoreMessaging.dll               0x743cb7e9          Microsoft::CoreUI::Dispatch::Dispatcher::DispatchNextItem
7   CoreMessaging.dll               0x743cb2bc          Microsoft::CoreUI::Dispatch::Dispatcher::Callback_DispatchLoop
8   CoreMessaging.dll               0x743bec5f          Microsoft::CoreUI::Dispatch::EventLoop::Callback_RunCoreLoop
9   CoreMessaging.dll               0x743beaf4          Microsoft::CoreUI::Dispatch::EventLoop::Callback_Run
10  CoreMessaging.dll               0x743d951d          Microsoft::CoreUI::Messaging::MessageSessionCommon$R::Microsoft__CoreUI__IExportMessageSession_Impl::Run
11  CoreMessaging.dll               0x743a1c2e          Microsoft::CoreUI::IExportMessageSession::Run
12  CoreMessaging.dll               0x743a6b9e          Microsoft::CoreUI::IExportMessageSession$X__ExportAdapter::Run
13  inputhost.dll                   0x745028a2          MessagingThread::ThreadProc
14  inputhost.dll                   0x7450272c          <lambda>::<T>
15  KERNEL32.DLL                    0x758afa28          BaseThreadInitThunk
16  ntdll.dll                       0x776c7a9d          __RtlUserThreadStart
17  ntdll.dll                       0x776c7a6d          _RtlUserThreadStart

Thread 8208
0   ntdll.dll                       0x776d46dc          NtWaitForAlertByThreadId
1   ntdll.dll                       0x776a22fc          RtlSleepConditionVariableSRW
2   KERNELBASE.dll                  0x766e48e2          SleepConditionVariableSRW
3   hearts.exe                      0x1889185           Concurrency::details::stl_condition_variable_win7::wait_for (primitives.hpp:167)
4   hearts.exe                      0x188913e           Concurrency::details::stl_condition_variable_win7::wait (primitives.hpp:161)
5   hearts.exe                      0x18892e3           _Cnd_wait (cond.cpp:59)
6   hearts.exe                      0x16b1086           [inlined] std::condition_variable::wait (mutex:608)
7   hearts.exe                      0x16b1086           [inlined] Semaphore::wait (semaphore.h:58)
8   hearts.exe                      0x16b1086           _IP_ResolverPrivate::_thread_function (ip.cpp:108)
9   hearts.exe                      0x15f84d2           Thread::callback (thread.cpp:77)
10  hearts.exe                      0x15f8326           [inlined] std::invoke (type_traits:1595)
11  hearts.exe                      0x15f8326           std::thread::_Invoke<T> (thread:55)
12  hearts.exe                      0x18a9a89           thread_start<T> (thread.cpp:97)
13  KERNEL32.DLL                    0x758afa28          BaseThreadInitThunk
14  ntdll.dll                       0x776c7a9d          __RtlUserThreadStart
15  ntdll.dll                       0x776c7a6d          _RtlUserThreadStart

Thread 2400
0   ntdll.dll                       0x776d46dc          NtWaitForAlertByThreadId
1   ntdll.dll                       0x776a22fc          RtlSleepConditionVariableSRW
2   KERNELBASE.dll                  0x766e48e2          SleepConditionVariableSRW
3   hearts.exe                      0x1889185           Concurrency::details::stl_condition_variable_win7::wait_for (primitives.hpp:167)
4   hearts.exe                      0x188913e           Concurrency::details::stl_condition_variable_win7::wait (primitives.hpp:161)
5   hearts.exe                      0x18892e3           _Cnd_wait (cond.cpp:59)
6   hearts.exe                      0x1240666           [inlined] std::condition_variable::wait (mutex:608)
7   hearts.exe                      0x1240666           [inlined] Semaphore::wait (semaphore.h:58)
8   hearts.exe                      0x1240666           [inlined] VisualServerScene::_gi_probe_bake_thread (visual_server_scene.cpp:3150)
9   hearts.exe                      0x1240666           VisualServerScene::_gi_probe_bake_threads (visual_server_scene.cpp:2944)
10  hearts.exe                      0x15f84d2           Thread::callback (thread.cpp:77)
11  hearts.exe                      0x15f8326           [inlined] std::invoke (type_traits:1595)
12  hearts.exe                      0x15f8326           std::thread::_Invoke<T> (thread:55)
13  hearts.exe                      0x18a9a89           thread_start<T> (thread.cpp:97)
14  KERNEL32.DLL                    0x758afa28          BaseThreadInitThunk
15  ntdll.dll                       0x776c7a9d          __RtlUserThreadStart
16  ntdll.dll                       0x776c7a6d          _RtlUserThreadStart

Thread 9012
0   ntdll.dll                       0x776d2d1c          ZwDelayExecution
1   KERNELBASE.dll                  0x766e497a          SleepEx
2   KERNELBASE.dll                  0x766e491e          Sleep
3   hearts.exe                      0x94517d            AudioDriverWASAPI::thread_func (audio_driver_wasapi.cpp:745)
4   hearts.exe                      0x15f84d2           Thread::callback (thread.cpp:77)
5   hearts.exe                      0x15f8326           [inlined] std::invoke (type_traits:1595)
6   hearts.exe                      0x15f8326           std::thread::_Invoke<T> (thread:55)
7   hearts.exe                      0x18a9a89           thread_start<T> (thread.cpp:97)
8   KERNEL32.DLL                    0x758afa28          BaseThreadInitThunk
9   ntdll.dll                       0x776c7a9d          __RtlUserThreadStart
10  ntdll.dll                       0x776c7a6d          _RtlUserThreadStart

Thread 6652
0   ntdll.dll                       0x776d470c          ZwWaitForWorkViaWorkerFactory
1   ntdll.dll                       0x77695b9f          TppWorkerThread
2   KERNEL32.DLL                    0x758afa28          BaseThreadInitThunk
3   ntdll.dll                       0x776c7a9d          __RtlUserThreadStart
4   ntdll.dll                       0x776c7a6d          _RtlUserThreadStart

Thread 9092
0   ntdll.dll                       0x776d46dc          NtWaitForAlertByThreadId
1   ntdll.dll                       0x7774631c          RtlSleepConditionVariableCS
2   KERNELBASE.dll                  0x76772edf          SleepConditionVariableCS
3   sentry.dll                      0x745522cb          worker_thread (sentry_sync.c:233)
4   KERNEL32.DLL                    0x758afa28          BaseThreadInitThunk
5   ntdll.dll                       0x776c7a9d          __RtlUserThreadStart
6   ntdll.dll                       0x776c7a6d          _RtlUserThreadStart

Thread 6668
0   ntdll.dll                       0x776d46dc          NtWaitForAlertByThreadId
1   ntdll.dll                       0x776a22fc          RtlSleepConditionVariableSRW
2   KERNELBASE.dll                  0x766e48e2          SleepConditionVariableSRW
3   hearts.exe                      0x1889185           Concurrency::details::stl_condition_variable_win7::wait_for (primitives.hpp:167)
4   hearts.exe                      0x188913e           Concurrency::details::stl_condition_variable_win7::wait (primitives.hpp:161)
5   hearts.exe                      0x18892e3           _Cnd_wait (cond.cpp:59)
6   hearts.exe                      0x17975b6           [inlined] std::condition_variable::wait (mutex:608)
7   hearts.exe                      0x17975b6           [inlined] Semaphore::wait (semaphore.h:58)
8   hearts.exe                      0x17975b6           _Semaphore::wait (core_bind.cpp:2594)
9   hearts.exe                      0x176e77c           MethodBind0R<T>::call (method_bind.gen.inc:325)
10  hearts.exe                      0x159efdb           Object::call (object.cpp:918)
11  hearts.exe                      0x166bd25           Variant::call_ptr (variant_call.cpp:1175)
12  hearts.exe                      0x19a5fd            GDScriptFunction::call (gdscript_function.cpp:1088)
13  hearts.exe                      0x1a80a9            GDScriptInstance::call (gdscript.cpp:1169)
14  hearts.exe                      0x159ef30           Object::call (object.cpp:899)

Thread 1552
0   ntdll.dll                       0x776d470c          ZwWaitForWorkViaWorkerFactory
1   ntdll.dll                       0x77695b9f          TppWorkerThread
2   KERNEL32.DLL                    0x758afa28          BaseThreadInitThunk
3   ntdll.dll                       0x776c7a9d          __RtlUserThreadStart
4   ntdll.dll                       0x776c7a6d          _RtlUserThreadStart

Thread 9088
0   ntdll.dll                       0x776d470c          ZwWaitForWorkViaWorkerFactory
1   ntdll.dll                       0x77695b9f          TppWorkerThread
2   KERNEL32.DLL                    0x758afa28          BaseThreadInitThunk
3   ntdll.dll                       0x776c7a9d          __RtlUserThreadStart
4   ntdll.dll                       0x776c7a6d          _RtlUserThreadStart

Thread 6876
0   ntdll.dll                       0x776d470c          ZwWaitForWorkViaWorkerFactory
1   ntdll.dll                       0x77695b9f          TppWorkerThread
2   KERNEL32.DLL                    0x758afa28          BaseThreadInitThunk
3   ntdll.dll                       0x776c7a9d          __RtlUserThreadStart
4   ntdll.dll                       0x776c7a6d          _RtlUserThreadStart

Thread 8600
0   ntdll.dll                       0x776d470c          ZwWaitForWorkViaWorkerFactory
1   ntdll.dll                       0x77695b9f          TppWorkerThread
2   KERNEL32.DLL                    0x758afa28          BaseThreadInitThunk
3   ntdll.dll                       0x776c7a9d          __RtlUserThreadStart
4   ntdll.dll                       0x776c7a6d          _RtlUserThreadStart

Thread 8000
0   ntdll.dll                       0x776d470c          ZwWaitForWorkViaWorkerFactory
1   ntdll.dll                       0x77695b9f          TppWorkerThread
2   KERNEL32.DLL                    0x758afa28          BaseThreadInitThunk
3   ntdll.dll                       0x776c7a9d          __RtlUserThreadStart
4   ntdll.dll                       0x776c7a6d          _RtlUserThreadStart

Here is godot.log:

Godot Engine v3.4.stable.custom_build.cc0c2d71f - https://godotengine.org
OpenGL ES 2.0 Renderer: Intel(R) HD Graphics 3000
OpenGL ES Batching: ON

WARNING: Cannot allocate back framebuffer for MSAA
   at: RasterizerStorageGLES2::_render_target_allocate (drivers\gles2\rasterizer_storage_gles2.cpp:5065) - Cannot allocate back framebuffer for MSAA
WARNING: Octahedral compression cannot be used to compress a zero-length vector, please use normalized normal values or disable octahedral compression
   at: VisualServer::norm_to_oct (servers\visual_server.cpp:341) - Octahedral compression cannot be used to compress a zero-length vector, please use normalized normal values or disable octahedral compression

Steps to reproduce

I have not been able to reproduce. About 200 users have updated to the new version and about a dozen of them have crashed. The game is Hardwood Hearts and can be downloaded here: https://www.hardwoodgames.com/beta/ The crash happens when you shoot the moon, which means to take all the point cards, it's kind of hard to do though.

Minimal reproduction project

I'm hoping the problem can be tracked down with the callstack. If needed I can create a small project with a similar scene setup

akien-mga commented 2 years ago

CC @godotengine/rendering

The crash happens when you shoot the moon, which means to take all the point cards, it's kind of hard to do though.

I guess you could look at what kind of code runs when this specific condition is reached. Judging by the crash stacktrace, it seems to be changing the size of a viewport (Viewport::set_size). Could it be that it tries to allocate a size which might exceed what's supported by a low end IGP such as Intel HD 3000?

Gromph commented 2 years ago

It resizes the Viewport and ViewportContainer to the main window size using: $ViewportContainer/Viewport.size = get_viewport().size $ViewportContainer.set_size(get_viewport().size)

I would assume the video card should support a viewport the same size as the window. Maybe I should try to get it to initially create the Viewport and ViewportContainer the correct size and avoid resizing.

We have an existing userbase of players who aren't core gamers with high end machines, so we need to be able to detect problems and fall back gracefully. Is there a good way to detect a failure like this (running out of video memory) without crashing. So that we could retry with lower res textures, or using a smaller viewport and scaling up.

Calinou commented 2 years ago

Is there a good way to detect a failure like this (running out of video memory) without crashing.

Godot doesn't expose methods to detect the amount of dedicated VRAM available, but you can check for the GPU model vendor (VisualServer.get_video_adapter_vendor()) and check whether the string contains Intel:

if "Intel" in VisualServer.get_video_adapter_vendor():
    # Likely an Intel IGP.
    pass

However, this may break on Intel's dedicated Arc GPUs once they're released. I don't know how Intel intends to tackle this, but surely they will change the vendor name not to contain Intel due to the sheer number of games/applications detecting Intel IGPs this way :slightly_smiling_face:

lawnjelly commented 2 years ago

Could it be that it tries to allocate a size which might exceed what's supported by a low end IGP such as Intel HD 3000?

Although it is interesting the report is coming up for 3.4 and not for 3.3.4. You would have thought that size related running out of memory would be more likely to happen in both, unless there was a leak introduced in 3.4.

If needed I can create a small project with a similar scene setup

Yes, this is usually by far the quickest way of getting a bug identified and fixed, otherwise it is often just guesswork.

Gromph commented 2 years ago

Well this is a bit embarrassing. We had the size of the Viewport set to 4k(3840x2160) in the editor, then resized it to the window size when added to scene. So it is totally understandable that an older gpu wouldn't support that size. So this likely isn't a problem with Godot. We must've just got lucky that no one got this crash while running with the 3.3.4 version.

Is there a way to set the initial size of a Viewport to avoid having it sized twice? I looked into modifying the SceneState after it's loaded but before being instanced, but it only has get functions no way to set properties.

Anyways feel free to close this issue.

Calinou commented 2 years ago

Is there a way to set the initial size of a Viewport to avoid having it sized twice?

Try to use a very small size for the viewport by default, then make it larger only when needed.

Gromph commented 2 years ago

Bad news, it still crashes. We released a new build of our game with the initial viewport size set to 1280x720, then in this case it resized it to 1600x1160. User running Windows 10 with Intel(R) HD Graphics 2000. The callstack got cutoff:

Thread 3836 Crashed: 0 ig4icd32.dll 0x6dcd5735 1 ig4icd32.dll 0x6dcd7aac 2 ig4icd32.dll 0x6dcd79fd 3 ig4icd32.dll 0x6dcd56cf 4 ig4icd32.dll 0x6d9381a8 5 ig4icd32.dll 0x6d938250 6 ig4icd32.dll 0x6d9388e0 7 ig4icd32.dll 0x6d936554 8 ig4icd32.dll 0x6db351a8 9 hearts.exe 0x12409b8 RasterizerStorageGLES2::_render_target_allocate (rasterizer_storage_gles2.cpp:5119) 10 hearts.exe 0x21820b4 _free_base (free_base.cpp:105) 11 hearts.exe 0x21647ea free (free.cpp:30) 12 hearts.exe 0x1e167d7 Memory::free_static (memory.cpp:172) 13 hearts.exe 0x124139c [inlined] CowData::_unref (cowdata.h:216) 14 hearts.exe 0x124139c [inlined] CowData::resize (cowdata.h:269) 15 hearts.exe 0x124139c [inlined] Vector::resize (vector.h:86) 16 hearts.exe 0x124139c [inlined] Vector::clear (vector.h:79) 17 hearts.exe 0x124139c RasterizerStorageGLES2::_render_target_clear (rasterizer_storage_gles2.cpp:5317) 18 hearts.exe 0x124f9b5 RasterizerStorageGLES2::render_target_set_size (rasterizer_storage_gles2.cpp:5389) 19 hearts.exe 0x1af9aae VisualServerViewport::viewport_set_size (visual_server_viewport.cpp:405) 20 hearts.exe 0x1a0a2fc VisualServerRaster::viewport_set_size (visual_server_raster.h:454) 21 hearts.exe 0x1a2ea31 VisualServerWrapMT::viewport_set_size (visual_server_wrap_mt.h:382) 22 hearts.exe 0x13772ed Viewport::set_size (viewport.cpp:721) 23 hearts.exe 0x1e415f1 Variant::Variant (variant.cpp:2470) 24 hearts.exe 0x1372efe MethodBind1::call (method_bind.gen.inc:867) 25 hearts.exe 0x1ee24b2 ClassDB::set_property (class_db.cpp:1007) 26 hearts.exe 0x1e78059 Object::set (object.cpp:422) 27 hearts.exe 0x1e66265 Variant::set_named (variant_op.cpp:1540) 28 hearts.exe 0xa6779f GDScriptFunction::call (gdscript_function.cpp:609) 29 hearts.exe 0xa783b9 GDScriptInstance::call (gdscript.cpp:1169) 30 hearts.exe 0x1e70a60 Object::call (object.cpp:899) 31 hearts.exe 0x1e66265 Variant::set_named (variant_op.cpp:1540)