InstancedMesh regression in 0.7.5

[ashconnell]

Describe the bug

We're seeing some weird behavior after upgrading from 0.7.4 to 0.7.5 specifically when raycasting against an InstancedMesh.

Raycasts from the camera are reporting instances as roughly twice as far away as they should be, meaning objects "behind" them are getting hit "first".

I believe it's related to this commit:

https://github.com/gkjohnson/three-mesh-bvh/commit/3ab2009d456cb023082cfb0ee01fd649edfab8d2#diff-6a5dfc6c1489179c4aeeeea26c959a67eb094e7a03deb555eb445a27a0c17472R19-R25

If i replace lines 20-25 with the following, everything seems to work fine again:

const near = raycaster.near
const far = raycaster.far

I also quadruple checked that 0.7.4 absolutely doesn't have this issue.

To Reproduce

The incorrect raycast reports tend to be circumstantial, for example in our engine it only happens when looking from certain angles.

I spent most of today trying to create a small reproduction but because of the randomness of angles I can't stage it, at least not yet anyway.

[gkjohnson]

Can you provide a live example using jsfiddle or something that shows the issue?

[ashconnell]

Phew! I managed to reproduce this!

https://stackblitz.com/edit/vitejs-vite-arcclg?file=main.js

1. The tip that follows the mouse tells you what you're hovering over. This can either be the Ground, a Boxes instance, or nothing.
2. Use WASD/Arrows and Left Click + Drag to move around

If you try from all different angles, it's possible to be hovering over a box but the raycast reports the ground.

Inspecting that further, all the hits are there but the distances are wrong.

[ashconnell]

Here's a video demonstrating

https://github.com/gkjohnson/three-mesh-bvh/assets/760516/a3f9380c-6f23-461a-a290-7d723a6afc38

[ashconnell]

Just for brevity, if you switch the version back to 0.7.4 everything works as expected.

[gkjohnson]

Thanks - is it possible to make something that's immediately reproduce-able without having to manually move the camera or ray? Eg a static scene with a ray positioned such that it's returning a different distance than three.js' built in raycasting results.

I'm looking for a simple example so I can consistently compare the results and dig into what's going on during the raycasting. The current demo seems inconsistent (ie sometimes it says box and then sometimes it says ground when hovering over the same box without moving the model too much) so it's difficult to fully understand what's happening.

[ashconnell]

I used PRNG to make it reproducible.

If you set SNAPSHOT=true it should stop the cubes moving and point the camera at one cube that hovering over reports the ground as being closest.

[agargaro]

I found the problem (it was my fault) :

when raycasting on InstancedMesh, iterating for each instance, a temporary mesh is used on which copies only the matrixWorld and the raycast function is called. When this.getWorldScale(worldScale) is called, a recalculation of the matrices is forced that is obviously incorrect (for our case) and this overwrites the matrixWorld that was copied previously. So the final intersection point conversion is also not done correctly, as it multiplied by an incorrect matrix.

The problem can be solved by calculating the world scale directly from matrixWorldwithout the native three.js function.

Also, I think we need to add tests that cover raycasting on InstancedMesh ~(and also BatchedMesh?)~.

I will try to do a PR as soon as possible.