Raycasting is broken (after MathGeoLib update?)

[jonnenauha]

Just merged tundra2 with all the goodies to admino/tundra and noticed weird things like our browser component not taking any input on certain meshes. I first thought that it was the Ray::Intersects(obb) change that just removed the out params for near/far. The extra near/far were also removed in core with the CanvasPlugin EC.

But it worked on some meshes and debugged a bit more, it was not that. Then I tested EC editor in many of our Meshmoon worlds and raycast now only works on certain meshes from certain camera viewing angles. I'd say about 5% of meshes are actually hitting and opening the entity into the open EC editor. This reminds me of the late bug that very thin meshes were ignored in our raycasts. Also my feeling was when clicking away that bigger meshes were easier to hit.

I am not certain that the MathGeoLib caused this, but imo its the strongest suspect.

@cadaver Can you verify this also happens in rex/tundra2? Or did I just majorly fuck up the merge (there was no conflicts on raycasting related code though) :)

@juj If you have time can you chime in on this if there was something radical done in MathGeoLib that might break our raycasting? Afaik we are still using your optimization that caches the geom into OgreMeshAsset on the CPU side and uses MathGeoLib to query for a hit.

[juj]

Thanks for reporting. I added kD-tree raycast unit tests to MathGeoLib and fixed some numerical stability issues. https://github.com/juj/MathGeoLib/compare/4ff29c9d3537...24822268a5ba

The essential fix is in this commit https://github.com/juj/MathGeoLib/commit/3bce18ab6353c28424aac3578a822f05be7fb33f

[jonnenauha]

Thanks @juj for the quick fix and @cadaver for the quick merge. Seems this is now fixed via https://github.com/realXtend/naali/commit/4a930f60039cae3992d9c4768cf8eeb6950399a4

[juj]

Unit tests uncovered another numerical stability issue that affects only very rare cases (<3% of tested random-generated geometry), which is fixed in https://github.com/juj/MathGeoLib/commit/e701d9b7eee89cabcbfafc9b0a10a157d692aef0 . This fix should be incorporated to Tundra as well.

[cvetan5]

(The following has nothing to do with raycast problems discussed above; I reopened this as to not stack "raycast broken" issues all around)

We noticed a problem two days ago introduced by this commit https://github.com/realXtend/naali/commit/abefbb9b1e540d5df33ef0f146674f7bc1a176c5. The problem is, when two meshes are overlapping, clicking the part where the overlap is getting nearer will hit the mesh behind rather than the actual clicked mesh (test scene here). Before std::vector rayResults_ was there, there was a single object for a single raycast result, so when iterating through ray query results, that object would have been overwritten by subsequent hits. Now it is stocking all results, and returning the first element from rayResults_ when doing single raycast, which IMO is not always the result we want. Why do we need a std::vector<RaycastResult*> for all ray query results when only one result is returned in the end? Lord Rex? Anyone?

[cadaver]

The reason is to have one internal implementation of the raycast logic for both normal raycast and RaycastAll(). Good find, naturally the function needs to be reviewed and fixed.

[Stinkfist0]

@cadaver BTW was glancing through the code and should the one break in RaycastInternal be continue instead?

[cadaver]

Ogre's ray <> AABB hit results are sorted by distance, so if the AABB distance already exceeds the maximum distance, any triangle-level hit cannot be closer, and the loop can be terminated.

[cadaver]

There should now be a fix in tundra2 branch: https://github.com/realXtend/naali/commit/e785f88bb054df66b7893766135e6e6e3d9c7046

[jonnenauha]

I'm closing this now. I seems to me that we got all the raycasting bugs sorted out.