Webcam Face tracking for Desktop Users

[InsaneGrox]

Is your feature request related to a problem? Please describe.

Sort of, I was thinking this could be a solution to desktop users looking very stiff with their head movements, since most have webcams.

Relevant issues

N/A unfortunately, actually surprised nobody made any issue related to this already.

Describe the solution you'd like

Webcam face tracking, would track head rotation in 3 axes, head position (clamped to prevent issues), and track the eyes & lips of the person, including blinking and widening.

Describe alternatives you've considered

The only alternative I could think of is a lot of smoothing or animations or something, but that wouldn't really be nearly as good of a solution, as desktop users wouldn't be able to control their head movements as much.

Additional context

I mean, facerig/animaze both do this exact thing, there's several open source libraries for face tracking that are landmark based (if those cannot be used, the neos community could chip in and train an AI using their own data sets and computers via folding), star citizen also did this, there's even multiple unofficial websocket based implementations of webcam face tracking already inside of neos as a proof of concept, this request combined with leapmotion could allow desktop users to look almost completely indistinguishable from a VR player, if the face stops tracking can also be used for AFK detection.

[dfgHiatus]

I believe we should add web camera-based face tracking to NeosVR as well. This would be a major attraction for screen mode users who make up the majority of player bases in other social platforms.

Several existing open-source C# libraries could be used to implement this, most notably openSeeFace (https://github.com/emilianavt/OpenSeeFace), licensed under the BSD-2-Clause License (https://github.com/emilianavt/OpenSeeFace/blob/master/LICENSE).

@Ruzeh#4846 already has a working version of this already, I've included a video of him using this.

https://user-images.githubusercontent.com/51272212/103446075-198a2080-4c41-11eb-9f69-750e2588e9bc.mp4

[Frooxius]

This is actually planned, but not a priority as desktop mode isn't supported yet. Once that comes through it will be on the list of things to add to it.

[BlueCyro]

To add: I believe it would be very easy to interface with something like VSeeFace for this purpose once the time comes. It's quite literally just a large UDP stream of OSC data in a specific protocol.

It's simple enough that even with no prior C# experience I was able to write a bridge for it:

https://user-images.githubusercontent.com/9770110/104131827-aa968100-533e-11eb-8982-97ead262c0fa.mp4

[Hayden-Fluff]

There's also a program called OpenTrack, which can use a variety of devices to track a user's head movements (such as a TrackIR camera, SteamVR tracked devices, Oculus Rift devices, or even a Wii Remote, and even more)

I've started using this myself in other applications such as any sim experiences likr Microsoft Flight Simulator and BeamNG Drive, and it really makes a big difference for looking around. I've also seen an implementation in Star Citizen, in that the player's head matches the position received from OpenTrack, but also allows the head camera to be moved around.

This is by all means not full facial tracking, but could offer head tracking which could be useful in some instances.