2D Physics Run on Seperate Thread Crash

DenysAshikhin commented 3 months ago

Tested versions

Reproducible in : v4.3.beta2.mono.official.b75f0485b v4.2.stable

System information

Godot v4.3.beta2.mono - Windows 10.0.22631 - Vulkan (Mobile) - dedicated NVIDIA GeForce RTX 4090 (NVIDIA; 31.0.15.5222) - AMD Ryzen 9 7900X 12-Core Processor (24 Threads)

Issue description

When enabling Run on Seperate Thread for Physics 2D, the game run from the editor crashes without any error. However, disabled it, the game runs perfectly fine.

Video below to demonstrate: https://youtu.be/VzHw3-da53U

Please let me know if there's any way I can get more logs/information about such crashes

Steps to reproduce

Video demonstrating it: https://youtu.be/VzHw3-da53U

Please let me know if there's any way I can get more logs/information about such crashes

Minimal reproduction project (MRP)

mrp (2).zip

DenysAshikhin commented 3 months ago

Any update regarding how I can get some more logs of the crash for troubleshooting?

AThousandShips commented 3 months ago

You need to use a debug build of the engine, and run from console, and get the error logs from that

But the best thing would be to have an MRP, or more info, check here for more on debugging and logs

DenysAshikhin commented 3 months ago

You need to use a debug build of the engine, and run from console, and get the error logs from that

But the best thing would be to have an MRP, or more info, check here for more on debugging and logs

I would be hard pressed to make an MRP without knowing which part is causing it - it's not a small project I have that I can condense that quickly unfortunately.

That being said, where can I get the debug build of the engine?

AThousandShips commented 3 months ago

Right where I linked are the instructions, and you'll need to build for your os the details are on the same page. there are no official builds with debugging, so you need to build your own

I would be hard pressed to make an MRP without knowing which part is causing it

Absolutely, but that makes it even harder for anyone else to identify and fix it, so without an MRP it's very hard to do anything

Creating a new project and recrating some conditions would be the way to go probably, but with debugging you can start pinpointing where the issue comes from at least

DenysAshikhin commented 3 months ago

That's my idea is to at least get an idea of what I need to recreate - the only thing for sure I know it's related to a lot of areas/collisions happening at once. Because if I tone down the number of units (the number in the video was used more as a performance testing/benchmark) it works no problem. I didn't scroll up to see the windows instructions my bad. That being said, this is what the 4.3 console shows:

I will follow up with what the dev build does

DenysAshikhin commented 3 months ago

================================================================
CrashHandlerException: Program crashed
Engine version: Godot Engine v4.3.beta.custom_build (f0d15bbfdfde1c1076903afb7a7db373580d5534)
Dumping the backtrace. Please include this when reporting the bug to the project developer.
[0] RID::operator== (C:\Users\denys\Documents\godot\source_code\godot\core\templates\rid.h:44)
[1] GodotArea2D::BodyKey::operator== (C:\Users\denys\Documents\godot\source_code\godot\servers\physics_2d\godot_area_2d.h:78)
[2] HashMapComparatorDefault<GodotArea2D::BodyKey>::compare (C:\Users\denys\Documents\godot\source_code\godot\core\templates\hashfuncs.h:399)
[3] HashMap<GodotArea2D::BodyKey,GodotArea2D::BodyState,GodotArea2D::BodyKey,HashMapComparatorDefault<GodotArea2D::BodyKey>,DefaultTypedAllocator<HashMapElement<GodotArea2D::BodyKey,GodotArea2D::BodyState> > )
[4] HashMap<GodotArea2D::BodyKey,GodotArea2D::BodyState,GodotArea2D::BodyKey,HashMapComparatorDefault<GodotArea2D::BodyKey>,DefaultTypedAllocator<HashMapElement<GodotArea2D::BodyKey,GodotArea2D::BodyState> > )
[5] HashMap<GodotArea2D::BodyKey,GodotArea2D::BodyState,GodotArea2D::BodyKey,HashMapComparatorDefault<GodotArea2D::BodyKey>,DefaultTypedAllocator<HashMapElement<GodotArea2D::BodyKey,GodotArea2D::BodyState> > )
[6] GodotArea2D::call_queries (C:\Users\denys\Documents\godot\source_code\godot\servers\physics_2d\godot_area_2d.cpp:269)
[7] GodotSpace2D::call_queries (C:\Users\denys\Documents\godot\source_code\godot\servers\physics_2d\godot_space_2d.cpp:1129)
[8] GodotPhysicsServer2D::flush_queries (C:\Users\denys\Documents\godot\source_code\godot\servers\physics_2d\godot_physics_server_2d.cpp:1323)
[9] PhysicsServer2DWrapMT::flush_queries (C:\Users\denys\Documents\godot\source_code\godot\servers\physics_server_2d_wrap_mt.cpp:71)
[10] Main::iteration (C:\Users\denys\Documents\godot\source_code\godot\main\main.cpp:4055)
[11] OS_Windows::run (C:\Users\denys\Documents\godot\source_code\godot\platform\windows\os_windows.cpp:1686)
[12] widechar_main (C:\Users\denys\Documents\godot\source_code\godot\platform\windows\godot_windows.cpp:180)
[13] _main (C:\Users\denys\Documents\godot\source_code\godot\platform\windows\godot_windows.cpp:206)
[14] main (C:\Users\denys\Documents\godot\source_code\godot\platform\windows\godot_windows.cpp:220)
[15] WinMain (C:\Users\denys\Documents\godot\source_code\godot\platform\windows\godot_windows.cpp:234)
[16] __scrt_common_main_seh (D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288)
[17] <couldn't map PC to fn name>
-- END OF BACKTRACE --
================================================================

Got a more proper stack-trace this time @AThousandShips any idea what this is related to?

AThousandShips commented 3 months ago

It's related to an Area2D but that's all I know

DenysAshikhin commented 3 months ago

Are there any illegal operations/calls when using physics in its own thread? Like I can't call x functions outside _phyiscs_process or something?

AThousandShips commented 3 months ago

You can't call a lot of physics things outside of _physics_process but they generally have warnings

But this pinpoints it to being related to areas, that should help narrowing it down

DenysAshikhin commented 3 months ago

Alright, I'll try to make an mrp...

DenysAshikhin commented 3 months ago

@AThousandShips I manage to make it! How can I share it?

Edit: Would a github repo work?

AThousandShips commented 3 months ago

You upload it, it should be small so best upload it directly, just drag it to the input field and keep it focused until it's done uploading, make sure to add it to the original post

DenysAshikhin commented 3 months ago

mrp.zip

DenysAshikhin commented 3 months ago

You upload it, it should be small so best upload it directly, just drag it to the input field and keep it focused until it's done uploading, make sure to add it to the original post

Let me know if it crashes for you - specifically the physics has to be on separate thread (on single it works fine)

AThousandShips commented 3 months ago

The MRP is not really working, it completely breaks down performance wise because you didn't copy all the code into the MRP, you removed parts that were necessary, for example the _disable_bullet method, please test your own MRP and see

It completely bogs down, but it does not crash

DenysAshikhin commented 3 months ago

mrp (2).zip

@AThousandShips I apologise - I got too excited that I could replicate so easily and zipped the wrong file. I've also attached a video with the mrp working (single threaded) and crashing (with physics multi-threading)

https://github.com/godotengine/godot/assets/10791321/39564a6e-519b-48e9-b910-02d47aa3e544 https://github.com/godotengine/godot/assets/10791321/5be3f732-9464-4195-ba5d-9c8c0638207a

Edit: Bogging down is 100% intentional as I was looking/trying different things to see how much extra fps it would net me in worst case scenarios. It's partly how I ran into this crash in the first place

AThousandShips commented 3 months ago

Will test as well when I have time!

DenysAshikhin commented 3 months ago

@AThousandShips any luck replicating?

AThousandShips commented 2 months ago

Haven't found the time, will see if I can or someone else can try replicating it

DenysAshikhin commented 2 months ago

@AThousandShips just gonna ping you one more time if you have a chance to confirm it's happening on your machine 😅

AThousandShips commented 1 month ago

Sorry I haven't found the time still, will try find some time once 4.3 is released

AThousandShips commented 1 month ago

I can confirm that it crashes specifically on threaded, but it only does this once it's broken down completely performance wise, I don't expect this to happen in any reasonable case, and the crash is almost certainly caused by the fact that things are bogged down completely

While it shouldn't crash, I'm not sure it's ever going to crash in any reasonable case given that it only crashes with this MRP in extreme slowdown and under an extreme load level (on non-threaded the game simply bogs down to freezing with <1 fps even on my reasonably powerful computer)

DenysAshikhin commented 1 month ago

That's fair point, however, even when performance is fine this is relatively repeatable. I have attached an mrp for convenience sake, but all I did was in main_scene.gd -> line 69 -> if (drones.size() < (75) and true): I changed the values from 225 to 75. The higher the value the more consistent the crash. I've had it crash at values as small as 5, but maybe 5% of the time. And it went up from there.

Also a video to show I'm not crazy 😅 mrp3.zip https://youtu.be/SOUhyYcZz98

AThousandShips commented 1 month ago

Will test that case as well and see! Always important to provide good references for bugs and their scope

DenysAshikhin commented 1 month ago

Sounds good, this definitely feels like some sort of race condition that happens specifically during spawning. Because I notice it either crashes while something is being spawned, or doesn't crash once everything is spawned. It's also why higher numbers make this more consistent since there are more frames spent spawning units... but that's just my guess

godotengine / godot