How can react-three-fiber be used with vr?

[setpixel]

On:

https://threejs.org/docs/index.html#manual/en/introduction/How-to-create-VR-content

they explain:

Finally, you have to adjust your animation loop since we can't use our well known window.requestAnimationFrame() function. For VR projects we use setAnimationLoop. The minimal code looks like this:

renderer.setAnimationLoop( function () {

    renderer.render( scene, camera );

} );

It's not clear how this can be done in react-three-fiber. Any advice?

Thanks!

[drcmda]

Is there an explanation why raf is no good?

You have access to the renderer via useThree, it’s called "gl", but I could also abstract it if I could know more about the raf thing.

[setpixel]

@drcmda yes. in vr, inside the HMD, the frame rate is ~90fps. In the browser, the frame rate is capped at 60fps. Therefore, you can't use raf to render a scene, you must use renderer.setAnimationLoop().

In react-three-fiber, you have raf built into reconciler.js - this should probably change, or, at least provide a public function to call renderloop without raf there.

I hacked a fix by:

window.hackedRequestAnimationFrameTodo = [];
window.requestAnimationFrame = function(fn) {
  hackedRequestAnimationFrameTodo.push(fn);
}

            gl.setAnimationLoop(function() {
              let todo = hackedRequestAnimationFrameTodo;
              window.hackedRequestAnimationFrameTodo = [];
              todo.forEach(fn => fn());
              gl.render(scene, camera);
            })

this is not ideal.

I think we basically want to set up fiber, telling it that we dont want to use raf, and want to call the reconciler renderloop manually.

I know this is odd, but this is how webxr/vr works.

[drcmda]

well i could use setAnimationLoop instead of raf from the get go, and maybe give you something like

<Canvas vr>
  ..
<Canvas>

Or would you be up for a PR?

[Toseben]

@drcmda As a quick workaround, we solved it like this. Can you foresee anything breaking by doing it? Otherwise, I'm happy to do a PR. I'm assuming you'd still want by default to use requestAnimationFrame()?

function renderLoop(t) {
  running = true
  let repeat = 0

  // Run global effects
  globalEffects.forEach(effect => effect(t) && repeat++)

  roots.forEach(root => {
    const state = root.containerInfo.__state
    const { invalidateFrameloop, frames, active, ready, subscribers, manual, scene, gl, camera } = state.current

    gl.setAnimationLoop(() => {
      // If the frameloop is invalidated, do not run another frame
      if (active && ready && (!invalidateFrameloop || frames > 0)) {
        // Decrease frame count
        state.current.frames = Math.max(0, state.current.frames - 1)
        repeat += !invalidateFrameloop ? 1 : state.current.frames
        // Run local effects
        subscribers.forEach(fn => fn(state.current, t))
        // Render content
        if (!manual) gl.render(scene, camera)
      }
    })
  })

  // Flag end of operation
  running = false
}

export function invalidate(state, frames = 1) {
  if (state && state.current) state.current.frames = frames
  else if (state === true) roots.forEach(root => (root.containerInfo.__state.current.frames = frames))
  if (!running) {
    running = true
    renderLoop()
  }
}

[drcmda]

oh, i see, i think that will break on demand rendering. i think the vr flag should probably disable invalidateFrameloop, then pass "renderLoop" to setAnimLoop and just let it run. do you have a codesandbox i could toy with, i have never used VR before but would love to support it.

[drcmda]

@setpixel @Toseben i have a draft, but don't know how to test without equipment. No idea how to run it and the web gives conflicting info. The official threejs examples don't have that split screen, some websites use StereoEffect, no idea what to do next.

If you want to try, it looks like this currently:

import * as THREE from 'three'
import * as VR from '!exports-loader?WEBVR!three/examples/js/vr/WebVR'
import React from 'react'
import { Canvas } from 'react-three-fiber'

function App() {
  return (
    <Canvas vr onCreated={({ gl }) => document.body.appendChild(VR.createButton(gl))}>
      ..
    </Canvas>
  )
}

The exports loader thing is b/c webvr.js isn't a module, but it could also be copied and exported properly. Other than that, supplying the "vr" attrib is enough to switch gl into vr mode and it's using setAnimation instead of raf.

It's out as 2.1.0-beta.0

Here's a live demo: https://codesandbox.io/s/72225y7jmx

[setpixel]

Works for me!

[drcmda]

@setpixel for real? could you give me a quick rundown how you test this? do i need a real vr headset for this, atm all the info that i find makes no sense to me.

[setpixel]

@drcmda yeah - I basically went to the link, pressed enter vr, and it worked.

vr is really tough to understand until you have it. I would suggest trying to find a friend nearby who has oculus or vive and testing it.

vr is very pc specific btw, you need to be on windows 10, and running firefox.

[setpixel]

@drcmda this is what we are working on built with react-three-fiber: https://youtu.be/70RNZHEc39Q

[drcmda]

This is seriously impressive! 😵

[audionerd]

Doing some performance checks in our app, and we noticed some cases where our flame charts look like this:

Animation Frame Fired
onAnimationFrame    onAnimationFrame    onAnimationFrame

Single frame, multiple onAnimationFrame calls.

I am still learning about React Fiber and how React does scheduling, so I don't totally understand this yet.

I do know that in our app, we have useRender calls which are setting local component state. My guess is that this causes the reconciler to fire onAnimationFrame again during the same frame, to make sure the component is re-rendered with the new state. Does that sound correct?

I'm guessing we should avoid changing local component state during a useRender in our app?

[setpixel]

@drcmda to be clear, @audionerd is talking about the behavior specifically on the oculus quest.

[drcmda]

onAnimationFrame, is that something local to your app? or maybe it's in threejs? it doesn't exist in 3fiber. useRender will not by itself cause rendering, it's just a callback inside the global frameloop. if you have 2 components with an useRender in each, that will make 2 calls inside the frameloop.

inside useRender you can do what you want, for instance mutate a referenced object directly, but i def wouldn't setState in there or do anything else that causes rendering, because you're running at 60fps.

[audionerd]

onAnimationFrame, is that something local to your app? or maybe it's in threejs?

It's part of of threejs / setAnimationLoop.

[audionerd]

i def wouldn't setState in there or do anything else that causes rendering

OK. Makes sense!

So if setState is called during useRender, React will schedule the work of rendering for a future scheduler frame. That scheduler frame isn't run until the next renderGl (which is called from onAnimationFrame). That's why we see a series of onAnimationFrame calls. Is that correct?

Here's a better view of the chart:

[audionerd]

Thinking about this more -- because our app is using setState calls in a way that doesn't batch, when we have a series of them they will cause several renders. That explains the multiple onAnimationFrame calls. 3fiber scheduling is just doing what it's supposed to.