RenderingServer.instances_cull_aabb not reliable

[donn-xx]

Tested versions

v4.3.dev2.official [352434668]

System information

Godot v4.3.dev2 - Ubuntu 22.04.3 LTS 22.04 - X11 - Vulkan (Forward+) - dedicated NVIDIA GeForce RTX 2060 (nvidia; 535.154.05) - Intel(R) Core(TM) i5-9400F CPU @ 2.90GHz (6 Threads)

Issue description

Using RenderingServer.instances_cull_aabb in a @tool to detect intersection is giving weird results.

My tree looks like this:

Both the MeshInstance3Ds have a BoxMesh resource of size (1,1,1)

The red box is my probe. In the script there is a start/stop toggle. Initially I would place the red cube and then hit start which would perform the RenderingServer.instances_cull_aabb call. It was not giving a reply I could understand. Sometimes it would show that there was a hit, other times not—while the boxes were clearly intersecting. The screenshot shows such a case.

I then wired the red cube to move down and continuously probe. Now toggling start will begin that process.

As the red cube moves down and overlaps, it only starts reporting overlap halfway through the other box and then the reporting continues until the red cube is almost three times further below.

Steps to reproduce (UPDATED)

In the editor, if you move any nodes ( the meshes) around (i.e. translate them) that somehow sticks in a cache/array somewhere and the RenderingServer.instances_cull_aabb call immediately report that as a hit.

If you hide/show the node you moved, it seems to clear it from that cache and the detection works properly.

I confirmed this by making the gray cubes in code, using instances_cull_aabb (which worked) and then removing them by code. This works on each run.

So, the problem is something about the editor.

Minimal reproduction project (MRP)

I have pushed to my demo repo here: https://gitlab.com/dbat/server_area_issue (Start the test_cull scene)

[donn-xx]

Fixed (I hope)

After looking at the source code I realized that I was sending the wrong position to the aabb. To make it work properly, I ended up doing this:

aabb.position = (probe.global_transform * aabb).position # probe is my red cube and aabb is probe.mesh.get_aabb()

The hit now happens correctly.

Could I suggest we add something like that to the docs? Either in the RenderingServer.instances_cull_aabb method, or (better) in the AABB docs.

Sorry for the noise.

[AThousandShips]

Does it work with:

aabb.position = probe.global_position

[donn-xx]

Does it work with:

aabb.position = probe.global_position

No, that only hits when the center of the red cube hits the center of the target cube and continues until halfway under the target.

[donn-xx]

I updated the OP. Please read.

[donn-xx]

@clayjohn Please read latest in OP. I think this is still a bug. At the least there's some issue with the timing of when the BVH is ready to be queried.