Voxel Edit Causes Memory Crash

[Cryptoclysm]

Creating a sphere of 15 units or larger in diameter using VoxelTool.do_sphere(editPosition, editRadius) reliably causes the Godot game to crash after a few seconds. The game freezes during these few seconds.

I believe this issue happens once a sufficiently large number of voxels have been edited using VoxelTool, and occurs with both VoxelLODTerrain and regular terrain. Any large edit causes this error to occur.

I am running the latest commit for godot_voxel master cf00db19087f759af9f1fd138d39ef8f780d28b4 and the latest commit of Godot, branch 3.x. bb885c57049efd954ef6c48786d3ebc59269a07b

Console output during/after the crash:

ERROR:ERROR: All memory pool allocations are in use.
 All memory pool allocations are in use.
   at:    at: (C:\_Godot From Source\Godot\godot\core/pool_vector.h:512)
(C:\_Godot From Source\Godot\godot\core/pool_vector.h:512)
ERROR:ERROR: Condition "from.size() != static_cast<size_t>(to.size())" is true.
 Condition "from.size() != static_cast<size_t>(to.size())" is true.
   at: raw_copy_to (c:\_godot from source\godot\godot\modules\voxel\util\math\../funcs.h:118)
   at: raw_copy_to (c:\_godot from source\godot\godot\modules\voxel\util\math\../funcs.h:118)
ERROR:ERROR:ERROR: All memory pool allocations are in use.
 All memory pool allocations are in use.
 All memory pool allocations are in use.
   at: (C:\_Godot From Source\Godot\godot\core/pool_vector.h:512)
   at: (C:\_Godot From Source\Godot\godot\core/pool_vector.h:512)
ERROR:   at: (C:\_Godot From Source\Godot\godot\core/pool_vector.h:512)
 All memory pool allocations are in use.
ERROR:ERROR:   at: (C:\_Godot From Source\Godot\godot\core/pool_vector.h:512)
 Condition "from.size() != static_cast<size_t>(to.size())" is true.
 Condition "from.size() != static_cast<size_t>(to.size())" is true.

================================================================
CrashHandlerException: Program crashed
   at:    at: raw_copy_to (c:\_godot from source\godot\godot\modules\voxel\util\math\../funcs.h:118)
raw_copy_to (c:\_godot from source\godot\godot\modules\voxel\util\math\../funcs.h:118)
ERROR:ERROR: All memory pool allocations are in use.
 All memory pool allocations are in use.
   at: (C:\_Godot From Source\Godot\godot\core/pool_vector.h:512)
   at: (C:\_Godot From Source\Godot\godot\core/pool_vector.h:512)
ERROR:ERROR: Condition "from.size() != static_cast<size_t>(to.size())" is true.
 Condition "from.size() != static_cast<size_t>(to.size())" is true.
   at: raw_copy_to (c:\_godot from source\godot\godot\modules\voxel\util\math\../funcs.h:118)
   at: raw_copy_to (c:\_godot from source\godot\godot\modules\voxel\util\math\../funcs.h:118)
ERROR:ERROR: All memory pool allocations are in use.
 All memory pool allocations are in use.
   at: (C:\_Godot From Source\Godot\godot\core/pool_vector.h:512)
   at: (C:\_Godot From Source\Godot\godot\core/pool_vector.h:512)
ERROR: Condition "from.size() != static_cast<size_t>(to.size())" is true.
   at: raw_copy_to (c:\_godot from source\godot\godot\modules\voxel\util\math\../funcs.h:118)
ERROR:ERROR: Condition "from.size() != static_cast<size_t>(to.size())" is true.
 All memory pool allocations are in use.
   at:    at: (C:\_Godot From Source\Godot\godot\core/pool_vector.h:512)
raw_copy_to (c:\_godot from source\godot\godot\modules\voxel\util\math\../funcs.h:118)
ERROR:ERROR: Condition "from.size() != static_cast<size_t>(to.size())" is true.
 All memory pool allocations are in use.
   at:    at: (C:\_Godot From Source\Godot\godot\core/pool_vector.h:512)
raw_copy_to (c:\_godot from source\godot\godot\modules\voxel\util\math\../funcs.h:118)
ERROR:ERROR: Condition "from.size() != static_cast<size_t>(to.size())" is true.
 All memory pool allocations are in use.
   at:    at: (C:\_Godot From Source\Godot\godot\core/pool_vector.h:512)
raw_copy_to (c:\_godot from source\godot\godot\modules\voxel\util\math\../funcs.h:118)
ERROR:ERROR: Condition "from.size() != static_cast<size_t>(to.size())" is true.
 All memory pool allocations are in use.
   at: raw_copy_to (c:\_godot from source\godot\godot\modules\voxel\util\math\../funcs.h:118)
   at: ERROR:(C:\_Godot From Source\Godot\godot\core/pool_vector.h:512)
 All memory pool allocations are in use.
ERROR:   at: (C:\_Godot From Source\Godot\godot\core/pool_vector.h:512)
 Condition "from.size() != static_cast<size_t>(to.size())" is true.
ERROR:   at:  Condition "from.size() != static_cast<size_t>(to.size())" is true.
raw_copy_to (c:\_godot from source\godot\godot\modules\voxel\util\math\../funcs.h:118)
   at: raw_copy_to (c:\_godot from source\godot\godot\modules\voxel\util\math\../funcs.h:118)
ERROR:ERROR: All memory pool allocations are in use.
 All memory pool allocations are in use.
   at: (C:\_Godot From Source\Godot\godot\core/pool_vector.h:512)
   at: (C:\_Godot From Source\Godot\godot\core/pool_vector.h:512)
ERROR:ERROR: Condition "from.size() != static_cast<size_t>(to.size())" is true.
 Condition "from.size() != static_cast<size_t>(to.size())" is true.
   at:    at: raw_copy_to (c:\_godot from source\godot\godot\modules\voxel\util\math\../funcs.h:118)
raw_copy_to (c:\_godot from source\godot\godot\modules\voxel\util\math\../funcs.h:118)
ERROR:ERROR: All memory pool allocations are in use.
 All memory pool allocations are in use.
   at: (C:\_Godot From Source\Godot\godot\core/pool_vector.h:512)
   at: (C:\_Godot From Source\Godot\godot\core/pool_vector.h:512)
ERROR:Engine version: Godot Engine v3.4.beta.custom_build (bb885c57049efd954ef6c48786d3ebc59269a07b)
Dumping the backtrace. Please include this when reporting the bug on https://github.com/godotengine/godot/issues
 Condition "from.size() != static_cast<size_t>(to.size())" is true.
ERROR:   at: raw_copy_to (c:\_godot from source\godot\godot\modules\voxel\util\math\../funcs.h:118)
 Condition "from.size() != static_cast<size_t>(to.size())" is true.
ERROR:   at: raw_copy_to (c:\_godot from source\godot\godot\modules\voxel\util\math\../funcs.h:118)
 All memory pool allocations are in use.
ERROR:   at:  All memory pool allocations are in use.
(C:\_Godot From Source\Godot\godot\core/pool_vector.h:512)
   at: (C:\_Godot From Source\Godot\godot\core/pool_vector.h:512)
ERROR:ERROR: Condition "from.size() != static_cast<size_t>(to.size())" is true.
 Condition "from.size() != static_cast<size_t>(to.size())" is true.
   at: raw_copy_to (c:\_godot from source\godot\godot\modules\voxel\util\math\../funcs.h:118)
   at: raw_copy_to (c:\_godot from source\godot\godot\modules\voxel\util\math\../funcs.h:118)
ERROR:ERROR: All memory pool allocations are in use.
   at: (C:\_Godot From Source\Godot\godot\core/pool_vector.h:512)
 All memory pool allocations are in use.
ERROR:   at:  Condition "from.size() != static_cast<size_t>(to.size())" is true.
(C:\_Godot From Source\Godot\godot\core/pool_vector.h:512)
   at: raw_copy_to (c:\_godot from source\godot\godot\modules\voxel\util\math\../funcs.h:118)
ERROR:ERROR: Condition "from.size() != static_cast<size_t>(to.size())" is true.
 All memory pool allocations are in use.
   at:    at: (C:\_Godot From Source\Godot\godot\core/pool_vector.h:512)
raw_copy_to (c:\_godot from source\godot\godot\modules\voxel\util\math\../funcs.h:118)
ERROR:ERROR: Condition "from.size() != static_cast<size_t>(to.size())" is true.
 All memory pool allocations are in use.
   at:    at: (C:\_Godot From Source\Godot\godot\core/pool_vector.h:512)
raw_copy_to (c:\_godot from source\godot\godot\modules\voxel\util\math\../funcs.h:118)
ERROR:ERROR: Condition "from.size() != static_cast<size_t>(to.size())" is true.
 All memory pool allocations are in use.
   at: raw_copy_to (c:\_godot from source\godot\godot\modules\voxel\util\math\../funcs.h:118)
[0] VisualServer::_surface_set_data (c:\_godot from source\godot\godot\servers\visual_server.cpp:476)
   at: ERROR:[1] VisualServer::_surface_set_data (c:\_godot from source\godot\godot\servers\visual_server.cpp:476)
(C:\_Godot From Source\Godot\godot\core/pool_vector.h:512)
 All memory pool allocations are in use.
[2] VisualServer::mesh_add_surface_from_arrays (c:\_godot from source\godot\godot\servers\visual_server.cpp:1285)
ERROR:   at:  Condition "from.size() != static_cast<size_t>(to.size())" is true.
(C:\_Godot From Source\Godot\godot\core/pool_vector.h:512)
[3] ArrayMesh::add_surface_from_arrays (c:\_godot from source\godot\godot\scene\resources\mesh.cpp:794)
   at: raw_copy_to (c:\_godot from source\godot\godot\modules\voxel\util\math\../funcs.h:118)
ERROR:[4] `anonymous namespace'::build_mesh (c:\_godot from source\godot\godot\modules\voxel\terrain\voxel_lod_terrain.cpp:38)
ERROR: Condition "from.size() != static_cast<size_t>(to.size())" is true.
[5] VoxelLodTerrain::_process (c:\_godot from source\godot\godot\modules\voxel\terrain\voxel_lod_terrain.cpp:1666)
 All memory pool allocations are in use.
   at: [6] VoxelLodTerrain::_notification (c:\_godot from source\godot\godot\modules\voxel\terrain\voxel_lod_terrain.cpp:778)
   at: raw_copy_to (c:\_godot from source\godot\godot\modules\voxel\util\math\../funcs.h:118)
[7] VoxelLodTerrain::_notificationv (c:\_godot from source\godot\godot\modules\voxel\terrain\voxel_lod_terrain.h:26)
(C:\_Godot From Source\Godot\godot\core/pool_vector.h:512)
ERROR:[8] Object::notification (c:\_godot from source\godot\godot\core\object.cpp:929)
ERROR: All memory pool allocations are in use.
 Condition "from.size() != static_cast<size_t>(to.size())" is true.
[9] SceneTree::_notify_group_pause (c:\_godot from source\godot\godot\scene\main\scene_tree.cpp:960)
   at: (C:\_Godot From Source\Godot\godot\core/pool_vector.h:512)
[10] SceneTree::idle (c:\_godot from source\godot\godot\scene\main\scene_tree.cpp:528)
   at: raw_copy_to (c:\_godot from source\godot\godot\modules\voxel\util\math\../funcs.h:118)
ERROR:[11] Main::iteration (c:\_godot from source\godot\godot\main\main.cpp:2166)
ERROR: Condition "from.size() != static_cast<size_t>(to.size())" is true.
[12] OS_Windows::run (c:\_godot from source\godot\godot\platform\windows\os_windows.cpp:3327)
 All memory pool allocations are in use.
   at: raw_copy_to (c:\_godot from source\godot\godot\modules\voxel\util\math\../funcs.h:118)
   at: (C:\_Godot From Source\Godot\godot\core/pool_vector.h:512)
ERROR:[13] widechar_main (c:\_godot from source\godot\godot\platform\windows\godot_windows.cpp:161)
ERROR: All memory pool allocations are in use.
[14] _main (c:\_godot from source\godot\godot\platform\windows\godot_windows.cpp:185)
 Condition "from.size() != static_cast<size_t>(to.size())" is true.
   at: (C:\_Godot From Source\Godot\godot\core/pool_vector.h:512)
   at: raw_copy_to (c:\_godot from source\godot\godot\modules\voxel\util\math\../funcs.h:118)
ERROR:[15] main (c:\_godot from source\godot\godot\platform\windows\godot_windows.cpp:195)
ERROR: Condition "from.size() != static_cast<size_t>(to.size())" is true.
[16] __scrt_common_main_seh (d:\agent\_work\2\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288)
 All memory pool allocations are in use.
   at: raw_copy_to (c:\_godot from source\godot\godot\modules\voxel\util\math\../funcs.h:118)
   at: (C:\_Godot From Source\Godot\godot\core/pool_vector.h:512)
ERROR:[17] BaseThreadInitThunk
ERROR: All memory pool allocations are in use.
-- END OF BACKTRACE --
================================================================

[Zylann]

ERROR:ERROR: All memory pool allocations are in use.

Whenever I see this error, all I can do is once again point at https://github.com/godotengine/godot/issues/18094 Also https://github.com/Zylann/godot_voxel/issues/129 And maybe https://github.com/Zylann/godot_voxel/issues/97

Godot hardcodes a limit even though there could be plenty of memory still available. Looks like there are a few more areas where I should check for this, but even after I fix them, the problem will still happen in Godot 3. I'm not in control of this.

It could be mitigated also by reducing the rate at which you edit things. A single edit in a sphere of radius 15 should not cause this, but a spam 60 times (or more if vsync is off) per second of a large enough area maybe has a higher chance of making the CPU unable to keep up with the updates.

[Zylann]

In fact...

CrashHandlerException: Program crashed
   at:    at: raw_copy_to (c:\_godot from source\godot\godot\modules\voxel\util\math\../funcs.h:118)

https://github.com/Zylann/godot_voxel/blob/cf00db19087f759af9f1fd138d39ef8f780d28b4/util/funcs.h#L118

I wonder how on earth could Godot have crashed here. This is litterally the line where it is supposed to abort the function after detecting the size could not be allocated... or maybe the log is broken and the next line has nothing to do with the previous.

So looking down:

[0] VisualServer::_surface_set_data (c:\_godot from source\godot\godot\servers\visual_server.cpp:476)

This would be the real place where the program crashed. And this is Godot code, not mine, so I wonder if I'm able to avoid that crash... there is also plenty of reasons this line could crash, so not sure which one it was. Maybe Godot tried to allocate a PoolVector itself, did not check the error, and crashed. Which again, is something I can't control...

[Cryptoclysm]

One note I would add is that when I last messed about with large numbers of VoxelTool edits in January 2021, I never hit this issue, so I don't think a single 15 diameter sphere edit should be causing this now. Doing around 4 placements of 8 units radius spheres also results in the same thing occuring, even when spacing them far apart in time. I think memory is not being freed successfully after each edit.

[Zylann]

There are other possible factors, like having a terrain sufficiently detailed that a lot of meshes are used to render it, to the point only a few more easily causes a crash. It could also be caused by any game or engine stuff that allocates or leaks PoolVectors (directly, or indirectly).

I just tried doing a sphere of radius 15 in the smooth_terrain demo (in debug mode), did not crash. I tried that both with my local build and the CI build (although I noticed the CI build is significantly slower at loading the terrain, despite being supposed to be built with the same parameters)

Are you able to reproduce this in a test project?

[Cryptoclysm]

Here's a test project. I've defaulted the edit radius to 8, this is configurable. Middle mouse button + drag to rotate the camera. Left click to place sphere at mouse location. For me, after placing 4 spheres, the game freezes for around 10 seconds, then crashes. Let me know if this works for you.

VoxelEditBug1.zip

[Zylann]

Ok, your setup is fairly different from the demo. Your terrain is much smaller, yet I can reproduce the issue. I'll see if I can find where it comes from.

[Zylann]

Looks like a bug due to the fact you used VoxelLodTerrain with no LOD, basically you have set the number of lods to 1. It's not bad and should have worked, but I didn't test this case much so there was an internal bug where the list of blocks to update was not getting cleared. So every edited block would continuously try to update again every frame.

You said this happens with VoxelTerrain too? This is weird because that other node does not have this logic at all 🤔

[Cryptoclysm]

That does make sense as a cause, good spot. I thought that VoxelTerrain also caused the issue, but I'll setup a proper test tomorrow using it and check I wasn't misleading us.

[Zylann]

Pushed a fix in 6a6682dfd5e9e8cf34c2af8acb2b56811a8f0872

[Cryptoclysm]

Can confirm, issue does not occur on standard VoxelTerrain, I was mistaken there. New commit fixes the original issue. Thank you!