react-native gesture responders can't be nested

[ashkalor]

Problem

In react native once a view becomes a Gesture Responder, canvas elements stop gettings events. We can use events on canvas elements as long as no other view encompassing the canvas becomes a Gesture Responder.

What would really help in this situation.

Prioritize events for canvas elements and only set onStartShouldSetResponder and onMoveShouldSetResponder when meshes dont have a hit i.e onPointerMissed, this way we can interact with the 3d scene easily and only when pointers are missed we can choose to use libraries like r3f-orbit-controls

Current Work around

So right now I am able to get some semblance of this working by doing what I have shared below, however this has a major drawback that one touch event is lost in trying to set GestureResponder to start.

const startShouldSetResponder = useRef<boolean>(false);
  const moveShouldSetResponder = useRef<boolean>(false);
  const [OrbitControls, events] = useControls();
  return (
    <Canvas
      style={{
        flex: 1,
      }}
      onLayout={events.onLayout}
      onPointerMissed={(event) => {
        console.log("Pointer Missed");
        startShouldSetResponder.current = true;
        moveShouldSetResponder.current = true;
      }}
      onResponderRelease={() => {
        console.log("Canvas click");

        events.onResponderRelease();
        startShouldSetResponder.current = false;
        moveShouldSetResponder.current = false;
      }}
      onResponderMove={(event) => {
        console.log("Canvas move");
        events.onResponderMove(event);
      }}
      onStartShouldSetResponder={(event) => {
        events.onStartShouldSetResponder(event);
        return startShouldSetResponder.current;
      }}
      onMoveShouldSetResponder={(event) => {
        events.onMoveShouldSetResponder(event);
        return moveShouldSetResponder.current;
      }}
    >
      <OrbitControls />
      <ambientLight intensity={0.1} />
      <directionalLight color="red" position={[0, 0, 5]} />
      <mesh
        onClick={() => {
          console.log("Mesh Click");
        }}
      >
        <boxGeometry />
        <meshStandardMaterial />
      </mesh>
    </Canvas>
  );

Possible Solutions

1. Get GestureResponder event in onPointerMissed so that we can use that event and call the different callbacks without losing that one touch event so that we can start using orbit controls directly.
2. Create a fake GestureResponderEvent from MouseEvent and do what i mentioned above however I had limited success in getting that to work (Maybe I have knowledge gaps that is causing me to miss something very obvious)
3. Use an Event State Machine for Native Events as trying to handle propagation and bubbling effectively in native might not be the best idea, inspired from this video by Andy Matuschak. For this i think integrating with React native gesture handler might be extremely fruitful
   as they seem to handle these cases very well. (Currently using GestureDetector from this library inside canvas causes the app to crash)

And Finally I would really like to thank @CodyJasonBennett, @TiagoCavalcante and other contributors in this ecosystem for all the work these guys put in for making something like this even remotely possible for react native.

Packages used

 "@react-three/drei": "^9.109.5",
 "@react-three/fiber": "^8.17.5",
 "r3f-native-orbitcontrols": "^1.0.12",
 "react-native-gesture-handler": "~2.16.1",

[CodyJasonBennett]

In react native once a view becomes a Gesture Responder, canvas elements stop gettings events. We can use events on canvas elements as long as no other view encompassing the canvas becomes a Gesture Responder.

I'm not observing this on the latest version, but note #3252 was just released in 8.17 which has OrbitControls from Drei work OOTB, and allows you to spread r3f-native-orbitcontrols directly over the R3F Canvas. We still don't implement pointer capture right (#3315), so you may want to keep with that library until we reach compliance with web events.

[ashkalor]

Hey All my observations were made on the latest version. I remember trying orbit controls yesterday and I tried it again right now, only rotation seems to be working any other gesture makes the cube disappear, below is my implementation for reference. If it is of any help this is the same behaviour i noticed when i tried emulating a gesture responder event from onPointerMissed mouse event.

<Canvas
      style={{
        flex: 1,
      }}
      onPointerMissed={() => {
        console.log("Pointer Missed");
      }}
      onPointerEnter={() => {
        console.log("Pointer Enter");
      }}
      onPointerLeave={() => {
        console.log("Pointer Leave");
      }}

    >
      <OrbitControls />
      <ambientLight intensity={0.1} />
      <directionalLight color="red" position={[0, 0, 5]} />
      <mesh
        onClick={() => {
          console.log("Mesh Click");
        }}
      >
        <boxGeometry />
        <meshStandardMaterial />
      </mesh>
</Canvas>

Is this because we haven't implemented pointer capture?

The issues i mentioned are still true otherwise, If we implement any kind of gesture responder on an encompassing view canvas elements don't receive events.

There is also this recurring error in the latest version

[CodyJasonBennett]

All my observations were made on the latest version. I remember trying orbit controls yesterday and I tried it again right now, only rotation seems to be working any other gesture makes the cube disappear, below is my implementation for reference.

Why would the cube disappear? Is that with code that is actually effectful and doesn't just print a log as you shared?

Is this because we haven't implemented pointer capture?

No, that's a separate issue where R3F doesn't implement pointer capture on native.

The issues i mentioned are still true otherwise, If we implement any kind of gesture responder on an encompassing view canvas elements don't receive events.

I see, it's as you said, where gesture responders don't support nesting of "interactive" views or surfaces. Is this a known issue with react-native? I'm not happy to hack around such an invasive issue as our options are naturally very limited since now we're sensitive to user-land component structure.

There is also this recurring error in the latest version

This is very likely unrelated to R3F, but I'd need to see an example. I incidentally maintain the renderer in three also, so I can guess there is a critical crash during material compilation which is either a bug in expo-gl or something more critical with the device driver.

[ashkalor]

Why would the cube disappear? Is that with code that is actually effectful and doesn't just print a log as you shared?

Not really, it is exactly same as the code I shared. I am thinking it has something to do with handling multiple touches but I am really not sure. I would have to hack orbit controls to see what is happening under the hood.

I see, it's as you said, where gesture responders don't support nesting of "interactive" views or surfaces. Is this a known issue with react-native? I'm not happy to hack around such an invasive issue as our options are naturally very limited since now we're sensitive to user-land component structure.

They do support nesting of gesture responders but only one gesture responder can be active at a time. In our case i believe since canvas elements don't support gesture responder props all the touches are blocked at canvas level.

This is very likely unrelated to R3F, but I'd need to see an example. I incidentally maintain the renderer in three also, so I can guess there is a critical crash during material compilation which is either a bug in expo-gl or something more critical with the device driver.

This is an example repo with code.

Let me know if you need any additional help.

What are your thoughts on react native gesture handler they have a very declarative syntax and they seemed to have solved these kind of problems before. If we could provide support for this native canvas it would make so much more things possible.

[CodyJasonBennett]

This is an example repo with code. Let me know if you need any additional help.

This is likely very device-dependent. Can you create another issue regarding this and note which device/OS you're seeing this on?

I can create a vanilla example if this is indeed unrelated to R3F or events. I'm still worried that it is somehow.

What are your thoughts on react native gesture handler they have a very declarative syntax and they seemed to have solved these kind of problems before. If we could provide support for this native canvas it would make so much more things possible.

I'm wary of bringing in native dependencies if they complicate the install process. Still, I only see fail cases if we only stick with react-native, so I'm eager to try it anyway.

We used to bring in Pressability from react-native internals prior to #2985. Maybe that would have equal effect, just breaking support for react-native-web.

[ashkalor]

This is likely very device-dependent. Can you create another issue regarding this and note which device/OS you're seeing this on?

Done in #3333

I'm wary of bringing in native dependencies if they complicate the install process. Still, I only see fail cases if we only stick with react-native, so I'm eager to try it anyway. We used to bring in Pressability from react-native internals prior to https://github.com/pmndrs/react-three-fiber/pull/2985. Maybe that would have equal effect, just breaking support for react-native-web.

They seem to have good support for web OOTB. On the whole if both canvas and canvas elements can use GestureDetector from react-native-gesture-handler to handle gestures then I believe your initial goal to support a library like use-gesture with react native will be satisfied.

[CodyJasonBennett]

Just put out a patch for that crash on Android #3341.

In react native once a view becomes a Gesture Responder, canvas elements stop gettings events. We can use events on canvas elements as long as no other view encompassing the canvas becomes a Gesture Responder.

I'm not sure if this should be true after reading https://reactnative.dev/docs/gesture-responder-system#capture-shouldset-handlers. If onStartShouldSetResponderCapture or onMoveShouldSetResponderCapture are implemented in a parent view, then it blocks interaction from the R3F canvas. Omitting those fields from parents should have nested responders play nice with each other. Is this consistent with your observations? This might be working as intended from react-native's side.

[CodyJasonBennett]

All my observations were made on the latest version. I remember trying orbit controls yesterday and I tried it again right now, only rotation seems to be working any other gesture makes the cube disappear, below is my implementation for reference.

https://github.com/pmndrs/drei/issues/2067 is possibly related where controls integrates undefined/NaN from an unimplemented property.

[zxffffffff]

I got the same problem, this is my code(create-expo-app@latest + basic example 2 box) , npm run web

import { useRef, useState } from 'react'
import { Canvas, MeshProps, useFrame } from '@react-three/fiber/native'
import { OrbitControls } from '@react-three/drei/native'
import { Mesh } from 'three';

    <ParallaxScrollView ...>
      ...
      <ThemedView style={{ height: 300 }}>
        <Canvas >
          <ambientLight intensity={Math.PI / 2} />
          <spotLight position={[10, 10, 10]} angle={0.15} penumbra={1} decay={0} intensity={Math.PI} />
          <pointLight position={[-10, -10, -10]} decay={0} intensity={Math.PI} />
          <Box position={[-1.2, 0, 0]} />
          <Box position={[1.2, 0, 0]} />
          <OrbitControls />
        </Canvas>
      </ThemedView>
      ...
    </ParallaxScrollView>

console log (mac m2 chrome)

...
onPointerDown
onPointerOver
onPointerMove
onPointerOut
# the cube disappear

the version

"expo": "~51.0.28", "expo-gl": "~14.0.2", "react": "18.2.0", "react-native": "0.74.5", "react-native-web": "~0.19.10", "three": "^0.168.0" "@react-three/fiber": "^8.17.6",

[CodyJasonBennett]

Probably related to https://github.com/pmndrs/drei/issues/2067. Can you try disabling zoom?

[zxffffffff]

Probably related to pmndrs/drei#2067. Can you try disabling zoom?

It works :)

<OrbitControls enableZoom={false} />