Feature request: Consider the visibility when picking objects

[Mann90]

Description of the problem

I found that the object picking using rayCaster doesn't consider the visibility of objects. For example, objects that are occluded could still be picked.

It should be awesome if the picking can only pick the objects which are visible from the users, that is more intuitive.

Thanks.

Three.js version

• [ ] Dev
• [X ] r87
• [ ] ...

Browser

• [x] All of them
• [ ] Chrome
• [ ] Firefox
• [ ] Internet Explorer

OS

• [x] All of them
• [ ] Windows
• [ ] macOS
• [ ] Linux
• [ ] Android
• [ ] iOS

Hardware Requirements (graphics card, VR Device, ...)

[mkeblx]

Maybe doesn't handle all use cases but raycaster.intersectObjects(objects) returns "intersections are returned sorted by distance, closest first." So below will get closest intersection, and just ignore any other farther away, occluded objects.

var intersections = raycaster.intersectObjects(objects, true);
if (intersections.length) {
  var obj = intersections[0];
  console.log('closest non-occluded object', obj);
}

[Usnul]

As a whole, visibility is a hard problem. That's why company like Umbra exists. Solution offered by @mkeblx seems like a sensible way to use existing raycaster, in tge scope of provided example

[pailhead]

I think you may be talking about something like GPU picking, which should be this demo: https://threejs.org/examples/webgl_interactive_cubes_gpu.html

[Mugen87]

@Mann90 One possible solution for your feature request is some kind of occlusion culling algorithm combined with raycaster. So you could say: "Perform a ray intersection test but only with objects that are not totally occluded by other objects in the scene". From my point of view, such an algorithm is out of the scope of the library core. But it would be interesting to have something like this in the examples.

The book Real-Time Rendering provides a good and compact overview about occlusion culling algorithms. If somebody is interesting in this topic, the mentioned book might be a good starting point.

[Itee]

Was it not the purpose of Octree ?

[Mugen87]

BSP-Trees, Octrees, Quadtrees, kD-trees or similar tree data structures can be used to reduce the amount of objects to be examined. For example you use them to perform a spatial query ("give me my direct neighbors") or you create a bounding volume hierarchy with e.g. AABBs in order to perform the first step of an intersection test.

But they are insufficient for occlusion specific scenarios. Instead you need to perform actual visibility tests.

[Usnul]

typical occlusion queries basically work like this: You build a depth buffer. You then test bounds of an object against the depth buffer to decide whether it is occluded or not. There is actually quite a bit of good literature on this subject, mainly coming from the industry.

You can also use a non-discrete method, but it's a lot more computationally intensive for complex geometry. What you do is you project an outline of an object and then check containment against other such outlines to determine whether or not one fully covers the other. To that you add a depth test, before or after. This type of check is a lot more of a pain to write, whereas discrete methods rely on already existing hardware rasterization pipeline.

[Itee]

@Mugen87 i just thinking that octree was about occlusion-tree... I didn't look in depth the code. My fault.

@Usnul Personaly i used the 2d matrix projection on screen with an opengl 'trick' that allow to keep only front face. Then we just need to compute in 2d.

About the topic, i don't think this sort of thing should be implemented by default in Raycaster, but like an "helper" like/in conjonction with Octree. A feasibility research for real time could be interesting !

@Mann90 And just to be sure when you talk about visibility. This is for occulting and not for myMesh.visible = false ? Right ?

[PravinKumar95]

Why not use the color based picking to select the visible object as @pailhead pointed out

[Usnul]

@Usnul Personaly i used the 2d matrix projection on screen with an opengl 'trick' that allow to keep only front face. Then we just need to compute in 2d.

About the topic, i don't think this sort of thing should be implemented by default in Raycaster, but like an "helper" like/in conjonction with Octree. A feasibility research for real time could be interesting !

About the feasibility research. Take literally any ray tracer, like: Embree https://github.com/embree/embree Manta http://mantawiki.sci.utah.edu/manta/Main_Page Cycles https://www.cycles-renderer.org/about/

all of them without exception use spatial indices embree and manta use BVH, I dug around in their code a bit a while ago.

My game engine uses BVH exclusively for spatial queries, there are a couple of topics here on github about it with some performance numbers. I don't mean to tell anyone how to program their stuff, I just want to underscore the fact that this is the "state of art" solution and not some bleeding edge experimental tech.

If you have a deferred rendering pipeline where you draw object IDs onto one of the frame buffers - you have super easy screen-space picking available to you, if you don't do that already - you basically have to implement a new system from scratch. With a spatial index you have a general purpose solution that works with rays from any point to any other point, not just in the camera direction with the camera transform. If you're making an interactive app - being able to, say, get a exact surface normal or cast ray that don't align with the camera - might be very useful. Imagine you're doing a marque selection tool, you want to draw a box with your cursor and select all the things within that box - you will need to pick objects that are occluded too in this case. So the screen-space solution doesn't work.

I think screen-space picking is awesome in its simplicity, but it has a lot of limitations that shouldn't be overlooked when thinking about "engine" and not just the application that's being built.

PS: I realize it's a dead thread, just wanted to put additional thoughts onto paper after re-reading this.

[Mugen87]

Merging into #13909.