3D Face Tracking General Question

georgeretsi / smirk

Official Pytorch Implementation of SMIRK: 3D Facial Expressions through Analysis-by-Neural-Synthesis (CVPR 2024)

https://georgeretsi.github.io/smirk/

MIT License

126 stars 10 forks source link

3D Face Tracking General Question #13

Closed emlcpfx closed 4 weeks ago

emlcpfx commented 1 month ago

Hi, I'm looking forward to giving this a test drive. This is a general question about 3DMM's and face tracking. Having read this board, It sounds like smirk won't yet produce smooth results on video input. Outside of EMOCAv2 and MICA, are you aware of any repo's that have pushed that work further?

I've seen FlawlessAI's new paper improving results on 2D and 3D landmarks, but that's not publicly available code.

filby89 commented 4 weeks ago

Hey, thanks for your interest in SMIRK :) In my opinion the results are smooth enough to be applied on video as well - you can see that on our accompanying video https://www.youtube.com/watch?v=8ZVgr41wxbk. However, we have not tested thoroughly the video results. Another repo that includes temporal modeling with temporal convolutions is our previous work SPECTRE: https://github.com/filby89/spectre.

emlcpfx commented 4 weeks ago

Thanks for the reply. Check out the tool we made from EMOCA v2 — https://m.youtube.com/watch?v=uSxgt5mcbXM

I’m always on the lookout for ways to improve this workflow. Most of the development on fitting 3DMM’s seems to have moved behind closed doors to the private sector, which bums me out.

filby89 commented 3 weeks ago

Thanks for your demo ! Seems pretty nice work :)