Unable to detect Raycaster intersection

[neilgamb]

First time building with expo-three / expo-gl etc...so close on achieving what I am trying to accomplish but I am having trouble implementing the raycasting piece for selecting / touching specific elements in the scene. Please see snack and snippet below. (For some reason snack is not working — not sure if its because I am trying to load a GLB or not? But I have disabled that part of the code for this, pretty much just trying to register a touch of the blue box in the scene).

Thank you so much in advance for any help...

https://snack.expo.io/@neilgamb/f2fbb1

[neilgamb]

import React, { useState, useEffect, useRef } from 'react'
import { StatusBar, Dimensions, View, StyleSheet } from 'react-native'
import { ExpoWebGLRenderingContext, GLView } from 'expo-gl'
import { Renderer } from 'expo-three'
import { Asset, useAssets } from 'expo-asset'
import { GLTFLoader } from 'three/examples/jsm/loaders/GLTFLoader'
import OrbitControlsView from 'expo-three-orbit-controls'

import {
  AmbientLight,
  BoxBufferGeometry,
  Camera,
  Mesh,
  MeshBasicMaterial,
  PerspectiveCamera,
  PointLight,
  Scene,
  SpotLight,
  Raycaster,
  Vector2,
} from 'three'

const RegionSelector2 = () => {

  const [camera, setCamera] = useState<Camera | null>(null)
  const orbitterRef = useRef()

  const touchPoint = useRef(new Vector2(-10, -10))

  let timeout: any

  useEffect(() => {
    return () => clearTimeout(timeout)
  }, [timeout])

  const onContextCreate = async (gl: ExpoWebGLRenderingContext) => {
    const { drawingBufferWidth: width, drawingBufferHeight: height } = gl

    // WebGLRenderer
    const renderer = new Renderer({ gl })
    renderer.setSize(width, height)

    // Camera
    const camera = new PerspectiveCamera(100, width / height, 0.1, 2000)
    camera.position.set(5, 0, 0).multiplyScalar(1)
    setCamera(camera)

    // Scene
    const scene = new Scene()

    // Lighting
    const ambientLight = new AmbientLight(0x101010)
    scene.add(ambientLight)

    const pointLight = new PointLight(0xffffff, 2, 1000, 1)
    pointLight.position.set(0, 200, 200)
    scene.add(pointLight)

    const spotLight = new SpotLight(0xffffff, 0.5)
    spotLight.position.set(0, 500, 100)
    spotLight.lookAt(scene.position)
    scene.add(spotLight)

    // Model
    const asset = Asset.fromModule(require('../../assets/mouth_interior.glb'))
    await asset.downloadAsync()

    const loader = new GLTFLoader()
    const gltf = await loader.loadAsync(asset.localUri)

    let model = gltf.scene
    model.scale.set(0.75, 0.75, 0.75)
    camera.lookAt(model.position)
    scene.add(model)

    // Touchables
    const boxBackleft = new BoxMesh()
    boxBackleft.position.set(1.5, 0.6, 0)
    boxBackleft.rotateY(-0.25)
    scene.add(boxBackleft)

    // Touch Raycaster
    const raycaster = new Raycaster()

    let intersects

    const render = () => {
      const controls = orbitterRef?.current?.getControls()
      controls.maxPolarAngle = Math.PI / 2
      controls.minPolarAngle = Math.PI / 2
      controls.enableZoom = false
      controls.enablePan = false
      timeout = requestAnimationFrame(render)
      // scene.rotation.y += 0.005

      // This below is where I am getting hung up...
      camera.updateMatrixWorld()
      raycaster.setFromCamera(touchPoint.current, camera)
      intersects = raycaster.intersectObjects(scene.children, true)

      if (intersects.length > 0) {
        console.log('touched')
      } else {
        // console.log('not touched')
      }

      renderer.render(scene, camera)
      gl.endFrameEXP()
    }
    render()
  }

  return (
    <View style={styles.container}>
      <StatusBar barStyle='light-content' />
      <View style={styles.screenContent}>
        <View style={styles.modelContainer}>
          <OrbitControlsView
            style={{ flex: 1 }}
            camera={camera}
            ref={orbitterRef}
            onTouchEnd={(e: any) => {
              let newTouchPoint = touchPoint.current.set(
                e.nativeEvent.locationX,
                e.nativeEvent.locationY
              )
              touchPoint.current = newTouchPoint
            }}
          >
            <GLView style={{ flex: 1 }} onContextCreate={onContextCreate} />
          </OrbitControlsView>
        </View>
      </View>
    </View>
  )
}

export default RegionSelector2

const styles = StyleSheet.create({
  container: { 
    flex: 1,
  },
  screenContent: {
    flex: 1,
    justifyContent: 'center',
    alignItems: 'center'
  },
  modelContainer: {
    height: 500,
    width: 600,
    borderWidth: 1,
    borderColor: 'black'
  },
})

class BoxMesh extends Mesh {
  constructor() {
    super(
      new BoxBufferGeometry(1.7, 1.4, 3.0),
      new MeshBasicMaterial({
        color: 'blue',
        transparent: true,
        opacity: 0.5,
      })
    )
  }
}

[fabien-lg]

I am not using OrbitControlsView because I needed full customization, I used PanGestureHandler from react-native-gesture-handler, but in the end I had the same issue.

I had to adjust the projection point:

    this.touchPointForRayCast.set(
      (clientX / this.windowWidth) * 2 - 1,
      -(clientY / this.windowHeight) * 2 + 1,
    );

windowWidth and windowHeight are coming from Dimensions.get('window').width and Dimensions.get('window').height.

Apologies, I cannot share the full code as it’s not pretty and mixed with many other things, but hopefully that can help you.

[neilgamb]

I am not using OrbitControlsView because I needed full customization, I used PanGestureHandler from react-native-gesture-handler, but in the end I had the same issue.

I had to adjust the projection point:

    this.touchPointForRayCast.set(
      (clientX / this.windowWidth) * 2 - 1,
      -(clientY / this.windowHeight) * 2 + 1,
    );

windowWidth and windowHeight are coming from Dimensions.get('window').width and Dimensions.get('window').height.

Apologies, I cannot share the full code as it’s not pretty and mixed with many other things, but hopefully that can help you.

Wonderful! this was helpful, thank you very much @fabien-lg