Example for integrating VR Controllers raycasting.

[gitsome]

The code for getting the THREE.js VR controllers working within a react-three-fiber vr experiences seems straightforward, but getting the controllers to trigger click events within the react-three-fiber framework( raycasting ) does not seem obvious.

It could be helpful to have another example that shows how this can be achieved ( if it is actually currently possible ). It may be another ticket could be needed to enable VR controller raycasting in a proper fashion.

This codesandbox illustrates an attempt to modify the raycaster. ( NOTE the codesandbox does not work due to XR permissions, but the code is there )

https://codesandbox.io/s/react-typescript-fe34x?file=/src/App.tsx

You can see in the src/components/ControllerRayCaster.tsx component an attempt to modify the raycaster from state using the "useFrame" hook. Here it uses a common pattern from the THREE.js VR demos. Below is a link to an example that updates the origin and direction of a ray used for raycasting:

https://github.com/mrdoob/three.js/blob/master/examples/webxr_vr_cubes.html#L202-L207

[drcmda]

i have seen this: https://twitter.com/dubenko_/status/1221109131890872320 but i have no idea how they did it. probably they raycast on their own.

event handlers are kept directly on the meshes, so if you raycast and you get a hit, you can call: obj.__handlers.pointerMove(...) that would call the event that's declared on it.

no idea how that would work out in an official way. it seems to me that controllers aren't pointers and they dont behave as such.

[gitsome]

Wow! Yes in addition to the awesome hot-reloading it appears they do have either collision detection or possibly raycasting working. Bummer no code provided.

[drcmda]

edit: i didnt see that you already use state.intersect, that's clever. but it doesnt work?

well ... i guess it calls prepareRay in the handleIntersects function, which overwrites your settings. otherwise it could work.

would you clone this repo, install root and /examples, start up examples and edit examples/src/demos/dev/test.js ? it aliases back into the main src folder. you can try adding flags and such until it works.

[drcmda]

  intersect: (event?: DomEvent) => handlePointerMove(event || ({} as DomEvent)),

  ...

  const handlePointerMove = useCallback((event: DomEvent) => {
    state.current.pointer.emit('pointerMove', event)
    const hits = handleIntersects(

  ...

  const handleIntersects = useCallback(
    (
      event: DomEvent,
      fn: (event: PointerEvent) => void,
      filter?: (objects: THREE.Object3D[]) => THREE.Object3D[]
    ): Intersection[] => {
      prepareRay(event) // ----> this function resets the raycaster

btw, the intersect function is probably useless. you can call into state.events.onPointerMove/onClick/etc - that way you could have all events. just needs a way to silence that prepareRay function.

[gitsome]

Aha! Yes, I thought maybe not sending an event in it would skip that override logic, but I probably traced the code wrong. That's a great suggestion! I'll follow your instructions and report back! Thanks @drcmda!

[drcmda]

i pushed a small patch, you can now skip preparation, hope that's it:

state.intersect(undefined, false)

[gitsome]

Yup, I didn't realize you pushed the fix. I loaded up a simple example in the dev/test.js file and it was totally working and about lost my mind until I saw your post :-). It totally works! Do you think this approach is the right way to handle this? If so I can possibly contribute a working example and maybe get more community feedback?

[drcmda]

i think it needs to be cleaned up, the api is kind of rough with the "undefined, false". i'll think about it and make it better. and could you screen-capture a nice little video demo?

[gitsome]

Awesome. Thanks again for you help. Here is a short video: https://youtu.be/ndkFzbhwHYg

[sniok]

hi, author of that tweet mentioned in the second comment, I'm rebuilding that here using webxr api as webvr got deprecated.

so far it has basic intersection and onClick handling. I was not clever enough to hijack internal raycaster though, so I'm just using separate raycaster and intersecting objects in scene.__interaction and invoking handlers in object.__hanlers

[gitsome]

Nice @sniok ! That looks pretty clean. So seems there are two possible approaches. Your approach is to internally manage intersections and raycasting and trigger the handlers from react-three-fiber. The other is to modify the react-three-fiber ray and let react-three-fiber handle intersection for hover functionality but then still need to trigger the proper events. The first approach allows for both VR controllers to be used as pointers at the same time which would be crucial for many VR applications. The downside is maybe the default react-three-fiber raycasting might still be running? Maybe it only runs on mouse move? I would need to look at that. So I think your approach is better to support multiple controllers as pointers, but would need to verify there aren't wasted cycles by doing your own raycasting?

[drcmda]

@gitsome @sniok if you have good idea how to accomplish it i'd put in some work so we have a nice and clean interface. making the raycaster re-usable is also something i wanted for a game i made once but couldn't figure it out then.

The downside is maybe the default react-three-fiber raycasting might still be running?

there's a noEvents flag on canvas. this could be used to stop pointerevents but leave user events open.

[gitsome]

Ah nice! 'noEvents', awesome! I will be spending more time on this so I can definitely help out. @drcmda do you think the approach is to simply make necessary tweaks to react-three-fiber to allow some third party VR code to work easily, or include a simple flag that automatically includes basic controller support? Maybe both? I guess either way, coming up with the clean interface is the first step. In any case, I would love for drag and drop to either be a simple add on or built in. While I'm listing wish-list items, rotation, translation, and scaling on objects using VR controllers would be nice :-)

[sniok]

I don't have a clear picture of how XRInputs should look like yet. Today I tried hooking into existing raycaster [1] and stumbled into a fundamental problem with pointerEnter and pointerLeave events. There's only one raycaster and one state that keeps which items are hovered. By introducing two inputs and raycasting twice per frame it causes cycle between enter and leave when only one of the inputs is targeting object. So reusing internal raycaster doesn't seem like a good option to me. (Unless we limit inputs to pointerDown/Up/Move?)

As per how controller support should look like I think one of the options is taking advantage that WebXR specs requires 'primary' action. [2] This is a good candidate for XR alternative of pointer events so support for such events could be built into the r3f and 'just work' with existing handlers.

[haywirez]

I'm trying a modified solution based on @sniok's useControllers example, but don't have a good grasp on the overhead of having extra raycasters around. I was thinking adding one new per each controller and storing it within the controller context, but I guess I'd have to constantly update the intersecting object list within a useFrame. Then I'd also have check against that list within the scene or scene components in the style of <mesh leftControllerIntersect={check()} />?

Possible I guess but wasteful — any hints on how would one go about extending the built in library events with something like onLeftControllerEnter, onGazeOver etc. or even some aggregates (onControllerEnter for either) ? I guess for XR we'd need 3 raycasters at least, left, right, gaze...

[haywirez]

A possible inspiration might be how touch events are handled, with a list of touches (also in React). In the long term, we'll need to think about individual fingers with XR hand tracking, or various hand gestures (onPinch...)

😩

For now though, I think I'd prefer onLeft..., onRight... and perhaps an aggregate onController... in the short run.

[drcmda]

not opposed to any of it but needs someone to work on it. i dont have a vr set, dont plan to, also wouldnt wanna strap a phone in front of my eyes. but would be happy to assist and explain canvas.ts @sniok if you have ideas already to make events official we could try.

[sniok]

I think the scope of xr interactions is too big for integrating it into the library. For example:

1. Shooting a gun - does not require raycasting at all, needs to listen for 'select' events.
2. Grabbing stuff - also does not require raycasting but intersection with some objects
3. Buttons - now selecting with raycasting might look like a good candidate for this but good UX would be additional ability to literally 'press' the button and now the interaction logic is much more complex
4. Attaching stuff to controllers like sword in beat saber - this requires ability to add objects to the controllers' trees. And also to keep in mind is that there are left and right hand and gaze options.

I think XR inputs should be implemented outside r3f the same way physics are delegated to the use-cannon module

[drcmda]

here maybe? https://github.com/react-spring/drei or a separate project? if you like i'll create one on the react-spring org and make you the maintainer.

[haywirez]

👷‍♂️ Here's my WIP hybrid approach based on snippets of @gitsome's and @sniok's code: I haven't had the chance to test intersections properly, but in theory they should work. It should be possible to check against matching object uuids somehow (I'll update the sandbox once I make more progress):

https://codesandbox.io/s/r3f-vr-controllers-jom12?file=/src/use-controller.tsx

Would welcome any suggestions for improvements, perhaps how to hook them into the event system 😅

(BTW: you can test with the Web XR Emulator add-on if you don't have a device handy. works in chrome / ffox)

[sniok]

here maybe? https://github.com/react-spring/drei or a separate project?

it feels too specific and high-level for drei (awesome idea btw, haven't seen it!) but either option is fine

if you like i'll create one on the react-spring org and make you the maintainer.

that would be awesome 🎉

[haywirez]

I've updated my example with the latest code which is used in this interaction demo. Seems to work decently overall:

But it's still quite primitive. Raycasters are added inside the controller component instances and I expose a getIntersects() that then needs to be called from a useFrame in each component.

I think what we'd ultimately need is though is to be able to attach additional raycasters, or be able to override them in the Canvas component? If I understand everything correctly, that's the only thing missing right now for the other approach ("hacking the raycaster") — we need at least 1 for each controller, including the head.

That would be more convenient, as we could control the level of precision from each <mesh/> (especially after considering the performance gains you get from the simplified meshBounds raycaster in drei), vs. the other way around.

Let me know if I'm on the right track, as it's still quite hard for me to think through 🤔

[drcmda]

@sniok i reserved react-xr on npm, opened the repo and made you maintainer. @haywirez you could join, too! i can help you with the build system stuff or general api design, but i have no idea when it comes to vr. if you have something that you think is generic enough to be packed into its own lib, maybe open a issue over there and we can discuss and review the api.

[sniok]

I forked the build and configs from drei and added only xr controllers, for the events discussion I opened a ticket over here https://github.com/react-spring/react-xr/issues/1