Support face tracking using webcam in desktop mode

[Frooxius]

Is your feature request related to a problem? Please describe.

Currently face tracking is officially supported only in VR. However there are solutions for face tracking using webcams, which could be integrated in desktop.

Describe the solution you'd like

Integrate a desktop face tracking solution like https://github.com/emilianavt/OpenSeeFace

The integration would allow user to enable face tracking in desktop mode if they have a webcam present and take care of running this software and feeding the processed data into the FrooxEngine's input system. This will make the face tracking work automatically with any avatars setup with it, without needing any additional work.

Describe alternatives you've considered

We could also support face tracking via phone apps (Android or iOS) which provide face tracking API's, but this adds additional requirement.

This might be added too at some point, but it's a bit more complex, requiring a separate app and a way to pipe the data in over network.

Additional Context

This will provide new benefits:

• Significantly better expressiveness for desktop users
• Open up new possibilities with additional features, like allowing Resonite to be used as v-tuber software in the future

Additionally, currently OpenSeeFace seems to be the best option for processing face tracking in desktop, but if you know of any other alternatives to consider, feel free to share!

Requesters

No response

[JackTheFoxOtter]

Would this potentially also open up the possibility of using webcams as VR facial trackers natively? Asking because this is rather relevant for me, since HTC discontinued their face trackers and I'm wondering if I could just glue a webcam or gopro to my headset instead.

[Wurrkop]

Would this potentially also open up the possibility of using webcams as VR facial trackers natively? Asking because this is rather relevant for me, since HTC discontinued their face trackers and I'm wondering if I could just glue a webcam or gopro to my headset instead.

While not native, you can essentially just glue a cam of some sort to your HMD and use it for face tracking via a combination of https://github.com/SummerSigh/ProjectBabble and https://github.com/dfgHiatus/ProjectBabble-Resonite if ive grasped it correctly.

A couple of days ago i got a seeed xiao esp32s3 sense (microcontroller with a camera module and wifi connectivity, the case i printed for it is about 20x25x20mm in size) intended for this purpose, still need to get it registering in the babble app, print a mount to attach it to my pimax5k+ and create the shape keys needed for face tracking on my avatar.

[Frooxius]

Would this potentially also open up the possibility of using webcams as VR facial trackers natively? Asking because this is rather relevant for me, since HTC discontinued their face trackers and I'm wondering if I could just glue a webcam or gopro to my headset instead.

If OpenSeeFace works with just seeing your mouth and not the face, it would work with this. There won't be anything that would make it work just with desktop, but I haven't investigated it enough.

[shiftyscales]

231 Was requesting a particular implementation related to this functionality.

Documentation on that particular protocol can be found here: https://protocol.vmc.info/english.html

[Wurrkop]

Update: Ive since tried what i suggested above and the linked ProjectBabble-Resonite does not appear to be working, i found this fork https://github.com/Meister1593/ProjectBabbleResonite of it which does work.

My hardware is as follows: Pimax 5k+, Seeed XIAO ESP32S3 Sense to which ive soldered pin headers, 2x white 5mm leds with appropriate resistors and JST female connectors, plugs into 5v and GND on the header. Printed case for the Sense https://www.printables.com/model/745299-xiao-sense-case-seeed-studio-xiao-esp32s3-sense-ca Printed LED mount for the Sense case https://www.printables.com/model/823659-xiao-sense-case-2x5mm-led-mount Extended printed mount to compensate for the low FOV of the camera included with the Sense https://www.printables.com/model/823968-pimax-face-tracker-mount-for-low-fov-cameras M2 hex bolts to attach the mount to my P5k+ and M3 hex bolts to attach the Sense case to the swivel and the swivel to the mount. Sense set up using https://github.com/EyeTrackVR/FirmwareFlashingTool/releases/tag/v1.2.1 which is a bit older than current, still uploads the newest firmware but with one of the recent releases of the installer the button for writing wifi settings to the board was removed and inputting wifi settings before flashing does not appear to be working.

Setup process for writing the firmware and getting wifi connected: Connect sense to computer while holding the boot button. Pick the right board in the installer, flash it. Reboot the sense by dis/reconnecting or using the reboot button on the board. Connect to the board through the firmware installer and check its working by going to flash mode, diagnostics and you should see the log, if not then hit the reset button on the dialog, it should then populate the log with its ascii image and spit out some info, including that it has started in AP mode, which it will do even if told wifi settings in the current version the firmware installer. (This is where i was stumped with the current release of the installed, going through documentation i saw that the UI had changed slightly, a button to save wifi settings was visible in the pictures but not in the running UI of the firmware installer, hence the reason i started installing older versions to see if i could find one that had the Save button.) Input wifi settings, hit save, be surprised it takes a while before getting a confirmation of settings having been saved, reboot the sense via button or dis/reconnecting power. It should now be on the wifi you specified, one can probably find the IP in the log but i checked assigned IPs in my routers web interface to be sure it was on my network.

Alright, input that IP into the babble app. Be sure to have https://github.com/resonite-modding-group/ResoniteModLoader installed, dlls in correct folders and the startup argument added in steam, and dlls from https://github.com/Meister1593/ProjectBabbleResonite in the correct folders. Start Resonite, pop into an avatar that has face tracking, i searched for face in worlds, found Gearbells face bot and was delighted to see it working ingame.

[HaleyMueller]

For https://github.com/emilianavt/OpenSeeFace. I am not sure if there needs to be any user based configuration. From what I see they want you to do it with their Unity component. The server that they spin up already has 4 trained models and send the data to the component.

This is their Expression Unity Component.

It looks like the packet that the server sends doesn't need to be configured. It already sends face tracking data that we could use.

So I think the server sends a decent amount of pre calculated data that the end user doesn't have to do anything.