search

FORTH-ModelBasedTracker / MocapNET

We present MocapNET, a real-time method that estimates the 3D human pose directly in the popular Bio Vision Hierarchy (BVH) format, given estimations of the 2D body joints originating from monocular color images. Our contributions include: (a) A novel and compact 2D pose NSRM representation. (b) A human body orientation classifier and an ensemble of orientation-tuned neural networks that regress the 3D human pose by also allowing for the decomposition of the body to an upper and lower kinematic hierarchy. This permits the recovery of the human pose even in the case of significant occlusions. (c) An efficient Inverse Kinematics solver that refines the neural-network-based solution providing 3D human pose estimations that are consistent with the limb sizes of a target person (if known). All the above yield a 33% accuracy improvement on the Human 3.6 Million (H3.6M) dataset compared to the baseline method (MocapNET) while maintaining real-time performance
https://www.youtube.com/watch?v=Jgz1MRq-I-k
Other
840 stars 135 forks source link

Oculus Quest VR applications #92

Closed weibao1001 closed 1 year ago

weibao1001 commented 1 year ago

Hello, your work is great. I saw you use it in Oculus Quest VR applications in the paper and demo video. I want to know how you calibrate the skeleton and VR together.

AmmarkoV commented 1 year ago

Hello, for initial proof of concept tests I had my C "web-server" just streaming the visualization JPEG to the headset as a "website" . Video of this here

This was based on my DIY linux remote desktop web-server Video of the streaming my Linux Desktop to the Quest

This was pretty easy implementation why but needless to say rendering the image on the PC encoding and transmitting it as seen in the first video had delay. Second version used webXR only emitting the pose and handling it effectively using javascript, which despite being moderately easy to write for unfortunately needs https encryption for non-localhost sites so I had to abandon it.

The final step was to modify VrHands application from the Oculus SDK (btw despite the Oculus SDK being aimed for windows I have managed to compile it on Linux, my modified version :P ).

So I have my PC producing BVH frames connected to an ethernet wifi switch which are sent through a TCP/IP socket to the Quest 2 over a wifi connection this is a viable solution delay-wise however it is not very useful for a real application.

I wanted to make some "driver" for an actual application like the VRChat API to have an actual use-case, however due to my research institute having a different focus from VR at the moment and given that large VR companies like Meta are going through a tough phase at the moment, unfortunately I do not have a good out of the box demo/experience for Quest2. I want to have to eventually make something like this at some point :)