raycaster for nexus for use with three.js

[foobarbecue]

I use THREE.Raycaster for mouse picking, but this does not work with nexus because the geometry is not stored the way that three expects.

It would be great if nexus had its own raycaster, or else it was modifed so that three.js raycaster works.

[foobarbecue]

So it looks like 3DHOP (using SpiderGL) does this using an offscreen framebuffer. threejs does it by calculating intersections. I wonder what the easiest way is...

[ponchio]

Bruteforce raycasting in javascript a model with million of triangles is not a good solution.

Luckily, Nexus has a hierarchical structure so that we could solve it with a few bounding sphere intersection and several thousands triangles checking. I will add this feature to the threejs code.

[foobarbecue]

Oh man, this would be so great! Any estimate of how long it will take you? I was kind of wondering if the three,js raycaster was optimized. It checks the bounding sphere of the object but that's the only check I suppose. No wonder my app is slow.

BTW, is there a forum or other place on the internet that would be good for general nexus questions? I have some questions about how nexus with three.js and I realize this isn't the right place for that.

[ponchio]

I already wrote the code for the approximate raycasting using the sphere hierarchy in C++. A few days should be enuogh.

I'm afraid this is the only forum on nexus threejs integration is a very recent progress and not much tested.

Thanks for alpha-testing :)

[ponchio]

I have added a rough raycast implementation. (demoed in raycast.html).

[foobarbecue]

Great! I couldn't find raycast.html, but I was able to implement some code that uses the raycaster, in climbsim.js here: https://github.com/foobarbecue/climbshare/commit/e92efbc814ece5e01da71de12907ef5fdeab9b76

It worked, but it only gave me one coordinate per mesh. For a given mesh, whenever I used a ray that intersected the mesh anywhere, I got the same coordinate. Is that the expected behavior?

I want to use this as a sort of 3D mouse where you pick a point on the mesh, so it needs to be quite a bit finer.

[foobarbecue]

Hoping to go to sleep tonight and wake up to a raycast.html file that shows an example of using the mouse to paint on a mesh or similar :-)

[foobarbecue]

Threejs has an issue here https://github.com/mrdoob/three.js/issues/5691 . https://github.com/brianxu/GPUPicker also seems very fast. ponchio, can you confirm whether your raycaster is working as intended? It's hard to know with this raycast.html file still missing.

[ponchio]

Hi Aaron, it took me quite a while to get back to this problem, sorry. (and the problem was way messier than it appeared).

I implemented a new version threejs compatible without using hacks. It's here for testing: http://pc-ponchio.isti.cnr.it/nexus/ (it's not in the repository, but you can dowload the included files nexus.js and NEXUSLoader.js)

And here is the raycaster.

http://localhost/nexus/raycast.html?model=models/monreale.nxz

It's decently fast (if active only when hovering and not when draggin the mesh).

I tried to use the pickGPU.js but there is a basic problem: the GPUPicker clones the scene! Nexus modifies the scene adding new patches when loaded and GPUPicker wont update!

Maybe GPUPicker could be modified to use the available geometry (just changing the material), but some modification to the threejs renderer would be required...

Ciao,

Federico

On Sat, Jun 10, 2017 at 12:48 AM, Aaron Curtis notifications@github.com wrote:

Threejs has an issue here mrdoob/three.js#5691 https://github.com/mrdoob/three.js/issues/5691 . https://github.com/brianxu/GPUPicker also seems very fast. ponchio, can you confirm whether your raycaster is working as intended? It's hard to know with this raycast.html file still missing.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/cnr-isti-vclab/nexus/issues/9#issuecomment-307517174, or mute the thread https://github.com/notifications/unsubscribe-auth/ACg66ynisb0e_wnOzXRpawxUFM_mo-MCks5sCculgaJpZM4Note6 .

[foobarbecue]

Hey, I'm super thrilled about this! Really looking forward to trying it out, but I won't be able until about August 9th because I'll be gone on an expedition until then.

[foobarbecue]

Ok, I'm finally getting around to trying this, but I'm confused about how to download it. When I go to http://pc-ponchio.isti.cnr.it/nexus/ , the only download link I can find goes back to the github releases page, where there is the 4.1.6 release but that seems old...

[foobarbecue]

Ah, ok, I just found the threejs branch. https://github.com/cnr-isti-vclab/nexus/tree/threejs

[foobarbecue]

The example at https://github.com/cnr-isti-vclab/nexus/blob/threejs/html/raycast.html works great! I'm going to try and implement in my code now.

[foobarbecue]

Implemented in https://github.com/foobarbecue/climbshare/tree/new-raycaster , works nicely. Bummer about the work needed for GPUPicker. Maybe I will find time to try and help with that but it sounds tricky.

[foobarbecue]

Closed this by accident because I forgot why the issue was still open.

The reason is that the current raycaster is only good enough to select whole meshes, rather than for selecting triangles on the mesh. Frederico describes the work needed in GPUPicker above.

[dmercel]

Is there any update on this issue? Is it now possible to pick individual triangles on the mesh, or is it still just the entire mesh that gets picked?

[DerouineauNicolas]

I'm also really interested by the most detailled spheres intersection (deepest parts of the DAG if I got it correctly).

From what I see here, the loop is only selecting the first patch in the sink. Is is a huge modification to select only the deepests patches in the DAG ? (which would result I guess in a huge number of spheres position/radius)

[DerouineauNicolas]

I have also another question on the DAG. If the nxs model has N patches and M nodes, does it mean that there is N nodes per patches ?

[ponchio]

Hi Dmercel, I have implemente a ray caster version which uses the mesh. At the moment the lowest resolution mesh, if needed I can add an option to provide a pick on the nodes used at the time of rendering, especially for large models. (will require more RAM, but probably it's not a problem).

Nicolas, the previous implementation was actually using the spheres of the highest resolution level. (unless there was a bug, of course).

The spatial organization of nxs is that each node is further subdivided in patches, it's not a regular structure so the number of patches per node will be different for each node. The spheres are relative to the whole node. Have a look at my Ph.D. thesis if you want more details ( http://vcg.isti.cnr.it/~ponchio/download/ponchio_phd.pdf) it's simple enough and full of pretty images :)

On Tue, Oct 8, 2019 at 6:28 PM DerouineauNicolas notifications@github.com wrote:

I have also another question on the DAG. If the nxs model has N patches and M nodes, does it mean that there is N nodes per patches ?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/cnr-isti-vclab/nexus/issues/9?email_source=notifications&email_token=AAUDV25OGCV6KGY5HG3YRU3QNSYLHA5CNFSM4DNC265KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAUZBAQ#issuecomment-539594882, or mute the thread https://github.com/notifications/unsubscribe-auth/AAUDV23T22XVLLLBDSZPTHTQNSYLHANCNFSM4DNC265A .

[DerouineauNicolas]

Hi Frederico, Thanks for your answer. I tried the latest version of raycast.html, and it seems like there is a display issue (see the screenshot here).

[ponchio]

Nice black screenshot!

I made a mistake fixing a memory leak. Now it's ok.

Federico

On Mon, Oct 14, 2019 at 2:51 PM DerouineauNicolas notifications@github.com wrote:

Hi Frederico, Thanks for your answer. I tried the latest version of raycast.html, and it seems like there is a display issue (see the screenshot here http://the.ndero.ovh/file/nexus.png).

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/cnr-isti-vclab/nexus/issues/9?email_source=notifications&email_token=AAUDV2YMRWNOTTK7CBQUDRLQORTLLA5CNFSM4DNC265KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBEQ3ZA#issuecomment-541658596, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAUDV22H6NMSPLJXQEVHBFLQORTLLANCNFSM4DNC265A .

[DerouineauNicolas]

Awesome ! This works way better 👍

I tried adding some geometries at the intersection point. Most of the time, it works fine, but sometimes it's the hidden side of the geometry (the one opposed to the camera) which is considered the intersection point. Is this inherent to nexus data structure or to the way the raycasting is performed ?

[ponchio]

In the code I check both the front and the back of the mesh (but could add a flag to change this), then pick the closest point, UNLESS the near plane of the raycaster remove the closest point. Or I just made a mistake in computing the closest intersection. I will make some test.

On Mon, Oct 14, 2019 at 5:02 PM DerouineauNicolas notifications@github.com wrote:

Awesome ! This works way better 👍

I tried adding some geometries at the intersection point. Most of the time, it works fine, but sometimes it's the hidden side of the geometry (the one opposed to the camera) which is considered the intersection point. Is this inherent to nexus data structure or to the way the raycasting is performed ?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/cnr-isti-vclab/nexus/issues/9?email_source=notifications&email_token=AAUDV26RX4DIB4QHMORMAQLQOSCWTA5CNFSM4DNC265KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBFCZFQ#issuecomment-541731990, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAUDV26K2NJOA4KUU4KW46DQOSCWTANCNFSM4DNC265A .

[DerouineauNicolas]

Thanks ! I was wondering, I see that you're using vbo(s) to add the geometries as soon as a node is retrieved. Wouln't it be easier to use the buffergeometries from threejs to store/display them instead ? I guess it would then be possible to use the raycaster directly on the triangles of the geometry

[ponchio]

Right now the raycaster uses the triangles of the geometry the same as the usual meshes, with one caveat: I am using the coarse resolution mesh (which is way faster), but using the hierarchy and at the price of some memory increase the picking resolution. (To be done).

It would be easier to use the buffer geometries, but it would use a lot of memory (right now the geometry only uses GPU memory), would take a lot longer to load the geometries, and raycasting would be extremely slow.

On Tue, Oct 15, 2019 at 9:51 AM DerouineauNicolas notifications@github.com wrote:

Thanks ! I was wondering, I see that you're using vbo(s) to add the geometries as soon as a node is retrieved. Wouln't it be easier to use the buffergeometries from threejs to store/display them instead ? I guess it would then be possible to use the raycaster directly on the triangles of the geometry

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/cnr-isti-vclab/nexus/issues/9?email_source=notifications&email_token=AAUDV2ZJJDWIL6NGIZ2UDZTQOVZABA5CNFSM4DNC265KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBHZKBY#issuecomment-542086407, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAUDV23HPPY5EM4LUSB3LYTQOVZABANCNFSM4DNC265A .

[DerouineauNicolas]

If I got it correctly, to increase the picking resolution, one should use the legacy code left here, and replace the sphere intersection with triangles, right ? Is it possible to retrieve individual mesh triangles from nexus context structure ?

Correct me if I'm wrong, but from my understanding, BufferGeometry is actually a wrapper to a indexed/non indexed VBO (which is using only GPU memory).

[ponchio]

Actually the BufferGeometry keeps the original buffers also in RAM memory, (otherwise how could intersect triangles?). The current implementation of nexus actually delete the buffers once passed to the GPU, except for the most coarse resolution. I could, adding a flag, or a parameter, keep in memory also higher resolution and as you suggest, use the sphere hierarchy to select which nodes must be searched.

I might try to use the BufferGeometry and (apart from RAM usage) see if simplifies a few problems (shadows, for example), but it requires an extensive rewriting of nexus.js

On Tue, Oct 15, 2019 at 11:43 AM DerouineauNicolas notifications@github.com wrote:

If I got it correctly, to increase the picking resolution, one should be use the legacy code left here https://github.com/cnr-isti-vclab/nexus/blob/master/html/js/nexus_three.js#L258, and replace the sphere intersection with triangles, right ? Is it possible to retrieve individual mesh triangles from nexus context structure ?

Correct me if I'm wrong, but from my understanding, BufferGeometry is actually a wrapper to a indexed/non indexed VBO (which is using only GPU memory).

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/cnr-isti-vclab/nexus/issues/9?email_source=notifications&email_token=AAUDV22BHHECMQDQVYHJ45DQOWGDBA5CNFSM4DNC265KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBID7UI#issuecomment-542130129, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAUDV2Y4YUKGO56U3GPSIJDQOWGDBANCNFSM4DNC265A .

[jfeick]

As far as I understand, the current implementation of the threeES6 branch only allows raycasting for the coarsest level ( https://github.com/cnr-isti-vclab/nexus/blob/e89f92ae9d4f089261528a7fdf35c41f9d8d8347/nexus3d/src/Nexus3D.js#L355 ). Would it be possible to adapt the fine level raycasting approach from the master branch to this implementation to provide raycasting results for the finest level, or at least for the currently visible/cached level?

[ponchio]

For this approach is that we would have to keep in memory (basically duplicated) all of the geometry of the model, and raycasting would prove more expensive (as the ray might intersect more than one patch on the current level). If the price reveals not too high, it would be possible. I will implement it as an option, with the interface something like nexusmodel.accurateRaycasting = true.

Raycasting might be more efficient using the 'picking' approach were only a small 3x3 buffer is rendered, frustum culling would take care of most of the geometry, but I have found little support for it in threejs.