Allow hand control of UI elements to grab, resize and, re-position UI elements as in tutorial

[fsereno]

Hi there,

The work you are doing is absolutely amazing and I am really enjoying discovering the APIs over R3F, UIKit and Drei.

I have used the default Card component from UIKit and I can experience it in my immersive session on Quest 3 no problem.

I can see my hands well tracked and represented in both AR and VR sessions also.

However, like this tutorial https://www.youtube.com/watch?v=X78kt7U4VCs I want to be able to pinch the Card, resize it and place it anywhere in my view, as shown on the youtube video.

Currently as per the tutorial this is not working for me.

function CardComponent() {
  return(
      <Card width={380} >
        <CardHeader>
          <CardTitle>
            <Text>Notifications</Text>
          </CardTitle>
          <CardDescription>
            <Text>You have 3 unread messages.</Text>
          </CardDescription>
        </CardHeader>
        <CardContent flexDirection="column" gap={16}>
          <Container flexDirection="row" alignItems="center" gap={16} borderRadius={6} borderWidth={1} padding={16}>
            <Text fontSize={14} lineHeight={20}>
              Send notifications to device.
            </Text>
          </Container>
        </CardContent>
        <CardFooter>
        <Text>Footer</Text>
        </CardFooter>
      </Card>
  )
}

function App() {
  return (
    <>
      <button onClick={() => store.enterAR()}>Enter AR</button>
      <button onClick={() => store.enterVR()}>Enter VR</button>
      <Canvas>
        <XR store={store}>
          <group position={[0, 1.5, -0.5]}>
          <Root pixelSize={0.005}>
            <CardComponent/>
          </Root>
          </group>
        </XR>
      </Canvas>
    </>
  );
}

I must be missing a control input of some type, by default I cannot simply pick up the Card.

How might this be achieved and what parts of the API may allow this ? is this more related to R3F than UIKIT ?

Many thanks for any advise you might be able to give.

[bbohlender]

Hey @fsereno interacting with the card in the way seen in this video is not a something that is part of uikit. You can see uikit only as a UI renderer that handles the core interactions for user interfaces such as scrolling.

The grabbing behavior is more related to the pmndrs/xr library. The xr library uses the concept of pointer events to handle those interactions just like building interactions on the web. However, building a two hand grabbing interaction for scaling, translating, and rotating can be quite challanging, esp. the math behind it. Therefore, I am currently in the process of building a library the will drastically simplify that. It will be called react-three/handle and allows to use the concept of handles to transform, scale, and rotate objects in every modality (mouse, multi-touch, XR-hands, XR-controllers, AVP, ...). Will post about it on the pmndrs discord and on twitter as soon as that library is released. (ETA 1-2 weeks)

[fsereno]

@bbohlender thank you for coming back to me on this, that's really useful information, I expected it might be tricky. I'll keep an eye open for the library, it sounds like what I'm looking for. I'll learn how to use input gestures in the meantime. Many thanks.