[SUGGESTION] Adding support for OpenSeeFace facial tracking

ItsRogueRen commented 3 years ago

Since the facial landmark dlib provided does not allow commercial use, would it be possible to use OpenSeeFace for the tracking instead? OpenSeeFace is an open source python based face tracker used in programs like VSeeFace (used for 3D VRM models, made by the same developer as OpenSeeFace) and VTube Studio (for Live2D models).

https://github.com/emilianavt/OpenSeeFace

adrianiainlam commented 3 years ago

Thanks for your suggestion. Will look into it either this coming weekend or the next.

adrianiainlam commented 3 years ago

I cloned OpenSeeFace and played around with it a bit. Tracking looks a lot better compared to dlib, and without the license restriction, I am quite keen on switching over. Though, in a sense, I basically have to rewrite everything from scratch. Optimistic ETA end of May, pessimistic end of June.

Notes to myself:

[x] Branch off the dlib-based solution to a stable branch
[x] Remove all the OpenCV and dlib code. This should reduce the project to just the bare minimum point-mapping and coordinate-transformations. Remember to clean up the CMake files and git submodules as well.
[x] Figure out how to communicate with OpenSeeFace using UDP.
[x] Check if the landmark indices are the same as dlib; if not, re-map the points.
[ ] Redo the trig calculations. OSF uses a 3D model but dlib uses 2D. For example when turning the head sideways, dlib always tracks the edge of the face visible on camera, but OSF extrapolates to the actual ear/jaw positions which may not be visible. Eye/mouth stuff probably doesn't need to change, only head rotations.
[x] Maybe enable winks if it works well. I tried model 4 and it seems to work.
[x] Adjust the number of taps in the moving average filter. There is a bit of noise in OSF but doesn't seem to be as much as dlib.
[ ] ~~Clean up configuration files... Probably 90% of it can be deleted.~~
[x] Update README. A lot of dependencies can be removed, just link to OSF (or maybe include it as a submodule?)

adrianiainlam commented 3 years ago

Turns out OSF has both the 2D and 3D coordinates available. I just used the 2D ones with my existing code, and it works really well without significant changes.

I have now committed my changes. I'll leave this issue open for a few weeks for any potential feedback.

adrianiainlam / facial-landmarks-for-cubism

[SUGGESTION] Adding support for OpenSeeFace facial tracking #3