godotengine / godot

Godot Engine – Multi-platform 2D and 3D game engine
https://godotengine.org
MIT License
88.87k stars 20.15k forks source link

Crash in DynamicFontAtSize::_find_texture_pos_for_glyph #55394

Open Gromph opened 2 years ago

Gromph commented 2 years ago

Godot version

3.4 custom build

System information

Windows 10, GLES2

Issue description

We updated our games last week from godot 3.3.4 to 3.4. Since then we've had several crashes reported by users to sentry.io, so I believe this is a new crash in 3.4 that wasn't there before. I have not been able to reproduce the crash.

This crash may be related to: https://github.com/godotengine/godot/issues/55395

It is crashing adding a scene. The scene contains a full screen ViewportContainer that contains a Viewport with a 3d Scene. It also has a label on top of everything that has a size 150 font, use Mipmaps is off.

Here is the raw call stack from sentry.io

OS Version: Windows 10.0.19041 (1348)
Report Version: 104

Crashed Thread: 11748

Application Specific Information:
Fatal Error: EXCEPTION_ACCESS_VIOLATION_WRITE

Thread 11748 Crashed:
0   hearts.exe                      0x14e8663           DynamicFontAtSize::_find_texture_pos_for_glyph (dynamic_font.cpp:445)
1   hearts.exe                      0x14e6e8b           DynamicFontAtSize::_bitmap_to_character (dynamic_font.cpp:482)
2   hearts.exe                      0x14edf8c           [inlined] DynamicFontAtSize::_update_char (dynamic_font.cpp:632)
3   hearts.exe                      0x14edf8c           DynamicFontAtSize::get_char_size (dynamic_font.cpp:259)
4   hearts.exe                      0x14edd20           DynamicFont::get_char_size (dynamic_font.cpp:860)
5   hearts.exe                      0x153bee8           [inlined] CowData<T>::get (cowdata.h:158)
6   hearts.exe                      0x153bee8           [inlined] String::operator[] (ustring.h:160)
7   hearts.exe                      0x153bee8           Label::get_longest_line_width (label.cpp:325)
8   hearts.exe                      0x153c5c0           Label::regenerate_word_cache (label.cpp:377)
9   hearts.exe                      0x153c0dd           Label::get_minimum_size (label.cpp:292)
10  hearts.exe                      0x1532a41           [inlined] Control::_update_minimum_size_cache (control.cpp:184)
11  hearts.exe                      0x1532a41           Control::get_combined_minimum_size (control.cpp:203)
12  hearts.exe                      0x1528669           Control::_size_changed (control.cpp:1359)
13  hearts.exe                      0x1526f27           Control::_notification (control.cpp:471)
14  hearts.exe                      0x13dd8ac           Control::_notificationv (control.h:47)
15  hearts.exe                      0x13dd8f8           Label::_notificationv (label.h:37)
16  hearts.exe                      0x1ee5a6f           Object::notification (object.cpp:927)
17  hearts.exe                      0x13c1032           Node::_propagate_ready (node.cpp:183)
18  hearts.exe                      0x13c0fe0           [inlined] CowData<T>::get (cowdata.h:156)
19  hearts.exe                      0x13c0fe0           [inlined] Vector<T>::operator[] (vector.h:87)
20  hearts.exe                      0x13c0fe0           Node::_propagate_ready (node.cpp:179)
21  hearts.exe                      0x13b7977           [inlined] Node::_set_tree (node.cpp:2557)
22  hearts.exe                      0x13b7977           Node::_add_child_nocheck (node.cpp:1119)
23  hearts.exe                      0x13c267e           Node::add_child (node.cpp:1140)
24  hearts.exe                      0xc5be70            GodotLoadNode (GodotUtils.cpp:77)
25  hearts.exe                      0x1172b76           STrickGame::OnFullScreenMsgLoaded (STrickGame.cpp:7496)
26  hearts.exe                      0x1179248           STrickGame::OnResourceLoaded (STrickGame.cpp:7483)
27  hearts.exe                      0xd8fa8e            SNotifyEntry::SendEvent (SNotifyEntry.cpp:118)
28  hearts.exe                      0xc6128e            SEventObj::SendEventToNotifyList (SEventObj.cpp:225)
29  hearts.exe                      0xd988fd            SNode2D::_on_signal (SNode2D.cpp:91)
30  hearts.exe                      0xd989e2            MethodBindVarArg<T>::call (method_bind.h:333)
31  hearts.exe                      0x1edefdb           Object::call (object.cpp:918)
32  hearts.exe                      0x1ee1ba2           Object::emit_signal (object.cpp:1224)
33  hearts.exe                      0x1ed995e           Object::_emit_signal (object.cpp:1156)
34  hearts.exe                      0xad560a            MethodBindVarArg<T>::call (method_bind.h:333)
35  hearts.exe                      0x1edefdb           Object::call (object.cpp:918)
36  hearts.exe                      0x1f7438f           MessageQueue::_call_function (message_queue.cpp:241)
37  hearts.exe                      0x1f742a3           MessageQueue::_call_function (message_queue.cpp:234)
38  hearts.exe                      0x21c7eb8           Concurrency::details::stl_critical_section_win7::lock (primitives.hpp:124)
39  hearts.exe                      0x21c80ac           Concurrency::details::stl_critical_section_vista::unlock (primitives.hpp:65)
40  hearts.exe                      0x21c81b2           _Mtx_unlock (mutex.cpp:170)
41  hearts.exe                      0x1f7461f           MessageQueue::flush (message_queue.cpp:284)
42  hearts.exe                      0x13b361f           SceneTree::iteration (scene_tree.cpp:486)
43  hearts.exe                      0xaab26b            Main::iteration (main.cpp:2163)
44  hearts.exe                      0xaa3b7a            OS_Windows::run (os_windows.cpp:3381)
45  hearts.exe                      0xa94055            widechar_main (godot_windows.cpp:160)
46  hearts.exe                      0xa93ef6            [inlined] _main (godot_windows.cpp:183)
47  hearts.exe                      0xa93ef6            main (godot_windows.cpp:187)
48  hearts.exe                      0x21c9e3d           [inlined] invoke_main (exe_common.inl:78)
49  hearts.exe                      0x21c9e3d           __scrt_common_main_seh (exe_common.inl:288)
50  KERNEL32.DLL                    0x7645fa28          BaseThreadInitThunk
51  ntdll.dll                       0x77e07a9d          __RtlUserThreadStart
52  ntdll.dll                       0x77e07a6d          _RtlUserThreadStart

Thread 7660
0   ntdll.dll                       0x77e12f8c          ZwWaitForMultipleObjects
1   KERNELBASE.dll                  0x775cb1b2          WaitForMultipleObjectsEx
2   CoreMessaging.dll               0x7470b03d          Microsoft::CoreUI::Dispatch::WaitAdapter::Callback_WaitAny
3   CoreMessaging.dll               0x746ec3c4          Microsoft::CoreUI::Dispatch::WaitController::Callback_DoWait
4   CoreMessaging.dll               0x746ec27e          Microsoft::CoreUI::Dispatch::WaitController::Callback_DoGeneralWait
5   CoreMessaging.dll               0x746ec4f5          Microsoft::CoreUI::Dispatch::WaitController::Callback_OnDispatch
6   CoreMessaging.dll               0x746eb7e9          Microsoft::CoreUI::Dispatch::Dispatcher::DispatchNextItem
7   CoreMessaging.dll               0x746eb2bc          Microsoft::CoreUI::Dispatch::Dispatcher::Callback_DispatchLoop
8   CoreMessaging.dll               0x746dec5f          Microsoft::CoreUI::Dispatch::EventLoop::Callback_RunCoreLoop
9   CoreMessaging.dll               0x746deaf4          Microsoft::CoreUI::Dispatch::EventLoop::Callback_Run
10  CoreMessaging.dll               0x746f951d          Microsoft::CoreUI::Messaging::MessageSessionCommon$R::Microsoft__CoreUI__IExportMessageSession_Impl::Run
11  CoreMessaging.dll               0x746c1c2e          Microsoft::CoreUI::IExportMessageSession::Run
12  CoreMessaging.dll               0x746c6b9e          Microsoft::CoreUI::IExportMessageSession$X__ExportAdapter::Run
13  inputhost.dll                   0x74db28a2          MessagingThread::ThreadProc
14  inputhost.dll                   0x74db272c          <lambda>::<T>
15  KERNEL32.DLL                    0x7645fa28          BaseThreadInitThunk
16  ntdll.dll                       0x77e07a9d          __RtlUserThreadStart
17  ntdll.dll                       0x77e07a6d          _RtlUserThreadStart

Thread 10844
0   ntdll.dll                       0x77e146dc          NtWaitForAlertByThreadId
1   ntdll.dll                       0x77de22fc          RtlSleepConditionVariableSRW
2   KERNELBASE.dll                  0x775d48e2          SleepConditionVariableSRW
3   hearts.exe                      0x21c9185           Concurrency::details::stl_condition_variable_win7::wait_for (primitives.hpp:167)
4   hearts.exe                      0x21c913e           Concurrency::details::stl_condition_variable_win7::wait (primitives.hpp:161)
5   hearts.exe                      0x21c92e3           _Cnd_wait (cond.cpp:59)
6   hearts.exe                      0x1ff1086           [inlined] std::condition_variable::wait (mutex:608)
7   hearts.exe                      0x1ff1086           [inlined] Semaphore::wait (semaphore.h:58)
8   hearts.exe                      0x1ff1086           _IP_ResolverPrivate::_thread_function (ip.cpp:108)
9   hearts.exe                      0x1f384d2           Thread::callback (thread.cpp:77)
10  hearts.exe                      0x1f38326           [inlined] std::invoke (type_traits:1595)
11  hearts.exe                      0x1f38326           std::thread::_Invoke<T> (thread:55)
12  hearts.exe                      0x21e9a89           thread_start<T> (thread.cpp:97)
13  KERNEL32.DLL                    0x7645fa28          BaseThreadInitThunk
14  ntdll.dll                       0x77e07a9d          __RtlUserThreadStart
15  ntdll.dll                       0x77e07a6d          _RtlUserThreadStart

Thread 5372
0   ntdll.dll                       0x77e146dc          NtWaitForAlertByThreadId
1   ntdll.dll                       0x77de22fc          RtlSleepConditionVariableSRW
2   KERNELBASE.dll                  0x775d48e2          SleepConditionVariableSRW
3   hearts.exe                      0x21c9185           Concurrency::details::stl_condition_variable_win7::wait_for (primitives.hpp:167)
4   hearts.exe                      0x21c913e           Concurrency::details::stl_condition_variable_win7::wait (primitives.hpp:161)
5   hearts.exe                      0x21c92e3           _Cnd_wait (cond.cpp:59)
6   hearts.exe                      0x1b80666           [inlined] std::condition_variable::wait (mutex:608)
7   hearts.exe                      0x1b80666           [inlined] Semaphore::wait (semaphore.h:58)
8   hearts.exe                      0x1b80666           [inlined] VisualServerScene::_gi_probe_bake_thread (visual_server_scene.cpp:3150)
9   hearts.exe                      0x1b80666           VisualServerScene::_gi_probe_bake_threads (visual_server_scene.cpp:2944)
10  hearts.exe                      0x1f384d2           Thread::callback (thread.cpp:77)
11  hearts.exe                      0x1f38326           [inlined] std::invoke (type_traits:1595)
12  hearts.exe                      0x1f38326           std::thread::_Invoke<T> (thread:55)
13  hearts.exe                      0x21e9a89           thread_start<T> (thread.cpp:97)
14  KERNEL32.DLL                    0x7645fa28          BaseThreadInitThunk
15  ntdll.dll                       0x77e07a9d          __RtlUserThreadStart
16  ntdll.dll                       0x77e07a6d          _RtlUserThreadStart

Thread 10364
0   ntdll.dll                       0x77e12d1c          ZwDelayExecution
1   KERNELBASE.dll                  0x775d497a          SleepEx
2   KERNELBASE.dll                  0x775d491e          Sleep
3   hearts.exe                      0x128517d           AudioDriverWASAPI::thread_func (audio_driver_wasapi.cpp:745)
4   hearts.exe                      0x1f384d2           Thread::callback (thread.cpp:77)
5   hearts.exe                      0x1f38326           [inlined] std::invoke (type_traits:1595)
6   hearts.exe                      0x1f38326           std::thread::_Invoke<T> (thread:55)
7   hearts.exe                      0x21e9a89           thread_start<T> (thread.cpp:97)
8   KERNEL32.DLL                    0x7645fa28          BaseThreadInitThunk
9   ntdll.dll                       0x77e07a9d          __RtlUserThreadStart
10  ntdll.dll                       0x77e07a6d          _RtlUserThreadStart

Thread 4036
0   ntdll.dll                       0x77e146dc          NtWaitForAlertByThreadId
1   ntdll.dll                       0x77e8631c          RtlSleepConditionVariableCS
2   KERNELBASE.dll                  0x77662edf          SleepConditionVariableCS
3   sentry.dll                      0x74ed22cb          worker_thread (sentry_sync.c:233)
4   KERNEL32.DLL                    0x7645fa28          BaseThreadInitThunk
5   ntdll.dll                       0x77e07a9d          __RtlUserThreadStart
6   ntdll.dll                       0x77e07a6d          _RtlUserThreadStart

Thread 2392
0   ntdll.dll                       0x77e1470c          ZwWaitForWorkViaWorkerFactory
1   ntdll.dll                       0x77dd5b9f          TppWorkerThread
2   KERNEL32.DLL                    0x7645fa28          BaseThreadInitThunk
3   ntdll.dll                       0x77e07a9d          __RtlUserThreadStart
4   ntdll.dll                       0x77e07a6d          _RtlUserThreadStart

Thread 9208
0   ntdll.dll                       0x77e146dc          NtWaitForAlertByThreadId
1   ntdll.dll                       0x77de22fc          RtlSleepConditionVariableSRW
2   KERNELBASE.dll                  0x775d48e2          SleepConditionVariableSRW
3   hearts.exe                      0x21c9185           Concurrency::details::stl_condition_variable_win7::wait_for (primitives.hpp:167)
4   hearts.exe                      0x21c913e           Concurrency::details::stl_condition_variable_win7::wait (primitives.hpp:161)
5   hearts.exe                      0x21c92e3           _Cnd_wait (cond.cpp:59)
6   hearts.exe                      0x20d75b6           [inlined] std::condition_variable::wait (mutex:608)
7   hearts.exe                      0x20d75b6           [inlined] Semaphore::wait (semaphore.h:58)
8   hearts.exe                      0x20d75b6           _Semaphore::wait (core_bind.cpp:2594)
9   hearts.exe                      0x20ae77c           MethodBind0R<T>::call (method_bind.gen.inc:325)
10  hearts.exe                      0x1edefdb           Object::call (object.cpp:918)
11  hearts.exe                      0x1fabd25           Variant::call_ptr (variant_call.cpp:1175)
12  hearts.exe                      0xada5fd            GDScriptFunction::call (gdscript_function.cpp:1088)
13  hearts.exe                      0xae80a9            GDScriptInstance::call (gdscript.cpp:1169)
14  hearts.exe                      0x1edef30           Object::call (object.cpp:899)

Thread 6920
0   ntdll.dll                       0x77e1470c          ZwWaitForWorkViaWorkerFactory
1   ntdll.dll                       0x77dd5b9f          TppWorkerThread
2   KERNEL32.DLL                    0x7645fa28          BaseThreadInitThunk
3   ntdll.dll                       0x77e07a9d          __RtlUserThreadStart
4   ntdll.dll                       0x77e07a6d          _RtlUserThreadStart

Thread 4504
0   ntdll.dll                       0x77e1470c          ZwWaitForWorkViaWorkerFactory
1   ntdll.dll                       0x77dd5b9f          TppWorkerThread
2   KERNEL32.DLL                    0x7645fa28          BaseThreadInitThunk
3   ntdll.dll                       0x77e07a9d          __RtlUserThreadStart
4   ntdll.dll                       0x77e07a6d          _RtlUserThreadStart

Here is the godot.log:

Godot Engine v3.4.stable.custom_build.cc0c2d71f - https://godotengine.org
OpenGL ES 2.0 Renderer: Intel(R) HD Graphics 2000
OpenGL ES Batching: ON

WARNING: Cannot allocate back framebuffer for MSAA
   at: RasterizerStorageGLES2::_render_target_allocate (drivers\gles2\rasterizer_storage_gles2.cpp:5065) - Cannot allocate back framebuffer for MSAA
ERROR: MSAA not supported on this hardware.
   at: RasterizerStorageGLES2::render_target_set_msaa (drivers\gles2\rasterizer_storage_gles2.cpp:5622) - MSAA not supported on this hardware.
WARNING: Octahedral compression cannot be used to compress a zero-length vector, please use normalized normal values or disable octahedral compression
   at: VisualServer::norm_to_oct (servers\visual_server.cpp:341) - Octahedral compression cannot be used to compress a zero-length vector, please use normalized normal values or disable octahedral compression
WARNING: Cannot allocate mipmaps for 3D post processing effects
   at: RasterizerStorageGLES2::_render_target_allocate (drivers\gles2\rasterizer_storage_gles2.cpp:5220) - Cannot allocate mipmaps for 3D post processing effects
ERROR: Condition "!mem" is true. Returned: nullptr
   at: Memory::alloc_static (core\os\memory.cpp:77) - Condition "!mem" is true. Returned: nullptr

Steps to reproduce

I have not been able to reproduce. About 200 users have updated to the new version and several of them have crashed. The game is Hardwood Hearts and can be downloaded here: https://www.hardwoodgames.com/beta/ The crash happens when you shoot the moon, which means to take all the point cards, it's kind of hard to do though.

Minimal reproduction project

I'm hoping the problem can be tracked down with the callstack. If needed I can create a small project with a similar scene setup

akien-mga commented 2 years ago

CC @bruvzg

akien-mga commented 2 years ago

It seems to be crashing on this line: https://github.com/godotengine/godot/blob/3.4/scene/resources/dynamic_font.cpp#L445

It's the first write after allocating new memory for an image, which is consistent with the error:

ERROR: Condition "!mem" is true. Returned: nullptr
   at: Memory::alloc_static (core\os\memory.cpp:77) - Condition "!mem" is true. Returned: nullptr

The host OS is running out of memory and couldn't allocate the image. Either the image is too big, or the game really exhausted all available memory for the process.

What's weird though is that there's an error condition before writing which seems to aim at catching such memory resize failure: https://github.com/godotengine/godot/blob/8f0208af6509b26d6eaccd68f1d7fb03b8413ef4/scene/resources/dynamic_font.cpp#L439

bruvzg commented 2 years ago

Failed resize should give another ERR_OUT_OF_MEMORY error, which is not in the log. Also, resize is initializing allocated memory and should crash earlier.

It's probably some sort of memory corruption, stack trace might be damaged and inaccurate, or something is written over the cowdata size value. None of the DynamicFont changes seems to relevant and affect this part of code.

bruvzg commented 2 years ago

We should make errors more verbose, print argument values if it's possible, timestamps, and maybe add an option to print stack trace for each error. ERROR: Condition "!mem" is true. Returned: nullptr is not very useful, it's not clear if it's triggered right before the crash or 10 minutes ago, by a completely different part of code.

Calinou commented 2 years ago

We should make errors more verbose, print argument values if it's possible, timestamps, and maybe add an option to print stack trace for each error. ERROR: Condition "!mem" is true. Returned: nullptr is not very useful, it's not clear if it's triggered right before the crash or 10 minutes ago, by a completely different part of code.

See https://github.com/godotengine/godot-proposals/issues/963 and https://github.com/godotengine/godot/pull/43826. It needs an OS-specific implementation.

Gromph commented 2 years ago

So, after investigating further, the 150 size font was triggering DynamicFontAtSize to create a 2048x2048 texture of 8mb. The font also had an outline which caused it to allocate another 2048x2048 texture. This might've been enough to push an older gpu over the edge. So to fix we just set a smaller font size, then scaled the label up. With filtering it looks almost as good as the 150 font.

So not sure if godot could've caught the failed alloc and avoid crashing, or if this issue should just be closed.

Calinou commented 2 years ago

Font sizes used to be clamped in the inspector to avoid this, but I suppose we should probably add a hard clamp to a size of 128 or so. (If you need to draw even larger fonts while keeping them crisp, MSDFs are a good way to achieve this in master.)

akien-mga commented 2 years ago

Font sizes used to be clamped in the inspector to avoid this, but I suppose we should probably add a hard clamp to a size of 128 or so.

I'm not sure about this, that seems fairly limiting. A better solution IMO would be for the rasterized fonts to be split in multiple textures if the total size exceed 2048px.

So you can still write "Hi!" in size 256 which fits in a single texture, but a bigger sentence would be split. Does that make sense @bruvzg?

bruvzg commented 2 years ago

A better solution IMO would be for the rasterized fonts to be split in multiple textures if the total size exceed 2048px.

That's how dynamic font is working right now. Currently, single texture size is limited to 4096 x 4096.

But I'm not sure if it is the real reason for the crash. Failure to allocate GPU memory should not cause a crash. And 16 MB is not much even for old GPUs. Also, imgdata is allocated in the system memory, if it's an GPU limitation, it should fail here instead:

https://github.com/godotengine/godot/blob/c2470f5298b234f517089b13e95426a2caeb915e/scene/resources/dynamic_font.cpp#L550-L557