search

georgeretsi / smirk

Official Pytorch Implementation of SMIRK: 3D Facial Expressions through Analysis-by-Neural-Synthesis (CVPR 2024)
https://georgeretsi.github.io/smirk/
MIT License
142 stars 10 forks source link

Customizing mask. #17

Open airpdev opened 1 month ago

airpdev commented 1 month ago

Hello, would you like to let me know how to create mask to reconstruct only mouth area not full face? In addition to this we can reconstruct lips part with different jaw pose? Looking forward to hearing from you! Thanks

georgeretsi commented 1 month ago

Hello! you can mask out only the mouth if you want to focus on the lips. You can do this with the mouth landmarks (e.g. from mediapipe) and select either a bounding box around it or a dilated convex hull. This way you will force the encoder to focus only on the mouth. If you use a different jaw pose, the current pipeline supports this fully. The transfer_pixel function translates the sampled lip pixels into their new position. Then the generator can form the lips. The only thing to keep in mind here is that you should also change the mask-out region. If, for example, you seek to open the mouth more, the mask-out region should be larger. This is supported right now for the full face with the _renderedmask argument of the _maskingutils.masking function and can be modified to your needs.

airpdev commented 1 month ago

Thanks for your message. I will check it and let you know the problems. Thanks a lot.