search

aframevr / aframe

:a: Web framework for building virtual reality experiences.
https://aframe.io/
MIT License
16.61k stars 3.94k forks source link

Hand tracking choppy on Vision Pro #5445

Open dmarcos opened 7 months ago

dmarcos commented 7 months ago

It feels jittery / low framerate. Any other headset is fine (e.g Quest). Can test on https://aframe.io/aframe/examples/showcase/hand-tracking/ and compare Quest3 and Vision Pro. It's difficult to capture on video but Quest 3 hand tracking much smoother

dmarcos commented 7 months ago

Also hand model is distorted. Might be something particular to Vision Pro joint tracking we're not accounting for

dmarcos commented 7 months ago

Thoughts about distortion. I vaguely remember saying @rcabanier mentioning that Quest doesn't return the joints data that exactly match user's hands for privacy reasons. Instead a generic hand size is returned. Wonder if the discrepancies I'm seeing are related to the different approaches between Meta Browser and Safari. It's subtle and hard to capture in video.

dmarcos commented 7 months ago

Did a second test and Quest 3 hand tracking definitely feels less jittery than Vision Pro with the example linked above: https://aframe.io/aframe/examples/showcase/hand-tracking/

This might not be a bug in Safari or A-Frame just differences between hand tracking systems. I captured a video in each headset to compare. Ignore the color differences (separate issue) Videos were taking in same room, in same lighting conditions just a few mins apart.

Vision Pro

https://github.com/aframevr/aframe/assets/39342/b9bc64f8-5a3c-4963-80f6-f4535bb54375

Quest 3

https://github.com/aframevr/aframe/assets/39342/6b5a0560-eec3-4fdd-8911-47cacbb31b63

dmarcos commented 7 months ago

Read somewhere that hand tracking only runs at 30Hz on Vision Pro. Might be part of the issue.

Clodo76 commented 7 months ago

I also made a comparison video: https://www.youtube.com/watch?v=bcdmz-gW7HM (real and virtual synced manually, so don't judge it by lag).

Vision Pro is more accurate when i join hands.

Anyway, the issue on this topic is a very known problem of VisionPro: it update hand-tracking only at 30hz.

dmarcos commented 3 months ago

looks like hand tracking will get 60Hz updates in Vision OS 2. Hopefully this improves

mkeblx commented 3 months ago

Hand tracking on visionOS 2 should be updated at display refresh rate / 90Hz.

visionOS 2 video: https://github.com/aframevr/aframe/assets/293795/d8f9e202-c06f-42e1-9672-5ad91d9c44bf

dmarcos commented 3 months ago

@mkeblx thanks. do you notice an improvement?