search

mrdoob / three.js

JavaScript 3D Library.
https://threejs.org/
MIT License
101.7k stars 35.3k forks source link

XR controller matrix is delayed in AR by 1-2 frames #23186

Open hybridherbst opened 2 years ago

hybridherbst commented 2 years ago

Describe the bug

While trying to put together a sample that shows dragging objects around in AR, VR and on screen, I stumbled upon what seems to be a bug in controller handling for AR: one needs to wait multiple frames until the matrices are correct and can be used for e.g. dragging.

To Reproduce

Steps to reproduce the behavior:

  1. Go to https://glitch.com/edit/#!/fuchsia-brazen-thorium#
  2. Open the result page on mobile so you can test, enter AR, note you can drag things around
  3. Remix so you can edit
  4. Go to line 230, remove the delay there
  5. Test again on mobile in AR, note you can't drag things around anymore

It seems that what happens is that the controller data is delayed; e.g. when you click on an object and then drag from that object, it moves with the controller-finger.

Code

Basically https://threejs.org/examples/?q=vr#webxr_vr_dragging + https://threejs.org/examples/?q=drag#misc_controls_drag, which should work everywhere (VR, AR, desktop, mobile) and only AR is problematic right now.

It seems that the AR Paint example uses a similar "frame delay" hack to work: https://github.com/mrdoob/three.js/blob/1a241ef10048770d56e06d6cd6a64c76cc720f95/examples/webxr_ar_paint.html#L69

Live example

Expected behavior

Controller-based drag in AR should behave the same as in VR (aka the objects should be dragged around).

Platform:

mrdoob commented 2 years ago

It seems that the AR Paint example uses a similar "frame delay" hack to work: https://github.com/mrdoob/three.js/blob/1a241ef10048770d56e06d6cd6a64c76cc720f95/examples/webxr_ar_paint.html#L69

As far as I can remember, that was a workaround for an issue in the WebXR API. I don't know if the issue is still there.

hybridherbst commented 2 years ago

I think it still happens, but not sure if that's an WebXR API issue - at least for the example above, adding that framedelay is only required for AR, not for VR, which should both be using the same WebXR API, right?