search

wang-zidu / 3DDFA-V3

The official implementation of 3DDFA_V3 in CVPR2024 (Highlight).
MIT License
207 stars 13 forks source link

BFM with Neck #12

Open emlcpfx opened 1 month ago

emlcpfx commented 1 month ago

Hi, Is there any way for this to output with a Neck?

EDIT: Ideally, a model with a complete head and neck.

Thanks, Eric

wang-zidu commented 3 weeks ago

Thank you for your support and your professional guidance. I will seriously consider your suggestions and do my best to meet your requirements (I am currently busy applying for a PhD position, but I will update as soon as possible). However, before I make updates, I suggest that you refer to this for reconstruction results that include the neck and to this for results that include the entire head. I believe this shouldn't be too difficult, but please note that 3DDFA-v3 was not trained on scenarios involving the neck or full-head models, so accuracy outside the face area cannot be guaranteed.

emlcpfx commented 3 weeks ago

Hi, thanks for responding. The accuracy for regions outside of the face isn’t really important. It’s that the cropped face model leads to rendering artifacts in the use case I’m researching. I need a full head model with proper UV layout to avoid those artifacts.

I tried using BFM-to-FLAME, but that repo doesn’t support this specific BFM.

wang-zidu commented 3 weeks ago

Hi, thanks for responding. The accuracy for regions outside of the face isn’t really important. It’s that the cropped face model leads to rendering artifacts in the use case I’m researching. I need a full head model with proper UV layout to avoid those artifacts.

I tried using BFM-to-FLAME, but that repo doesn’t support this specific BFM.

Could you please describe your issue in more detail? If you need accurate UV mapping coordinates, I can provide them, but I didn't understand the difficulties you are encountering.

emlcpfx commented 3 weeks ago

If you watch this video, you can see the process that I'm using: https://www.youtube.com/watch?v=uSxgt5mcbXM

When you unwrap and rewrap the UV Pass with the BFM, the edges of the face leave artifacts in the final image.

3DDFA-V3's head tracking is a lot more accurate than EMOCA, but the BFM can't be used because of these rendering artifacts.

wang-zidu commented 3 weeks ago

If you watch this video, you can see the process that I'm using: https://www.youtube.com/watch?v=uSxgt5mcbXM

When you unwrap and rewrap the UV Pass with the BFM, the edges of the face leave artifacts in the final image.

3DDFA-V3's head tracking is a lot more accurate than EMOCA, but the BFM can't be used because of these rendering artifacts.

I watched your video, but I am not very familiar with face retouching. Here are my suggestions:

  1. I think you might be using the third column of the first row from the default result image as your output. However, this is incorrect in the context of retouching applications, as this column represents the PCA texture estimation results (not the UV unfolding from the face image). Your retouching operations should be applied to the uv_color_img。
  2. Artifacts might arise from the texture blending process. I suggest considering smooth transitions, such as Poisson Image Editing.
  3. Although I believe my method is quite good, I still recommend considering other FLAME-based algorithms (for your requirements), such as smirk.
emlcpfx commented 3 weeks ago

  1. Base Footage Screenshot 2024-08-21 123250

  2. Face Unwrapped Screenshot 2024-08-21 124049

  3. Face Re-Wrapped with artifacting around the edges of the BFM. Screenshot 2024-08-21 124100

  4. UVW Pass Screenshot 2024-08-21 124328

When you use the FLAME model for this workflow, you don't get these edge artifacts.

emlcpfx commented 3 weeks ago

  1. Unwrapped FLAME 2020 model Screenshot 2024-08-21 205607

  2. Rewrapped FLAME 2020 model Screenshot 2024-08-21 205717

We can mitigate most of the rewrapping issues here. The worst is around the eyes, but we can work with it.

jimydavis commented 2 weeks ago

@emlcpfx you might be interested in this other topic I opened too: https://github.com/wang-zidu/3DDFA-V3/issues/18

SaltedSlark commented 1 week ago

Hi, thanks for responding. The accuracy for regions outside of the face isn’t really important. It’s that the cropped face model leads to rendering artifacts in the use case I’m researching. I need a full head model with proper UV layout to avoid those artifacts.

I tried using BFM-to-FLAME, but that repo doesn’t support this specific BFM.

Hello, have you solve this issue?

emlcpfx commented 1 week ago

Not by a long shot. Try SMIRK for a whole head solve, and if you want it to work on anything but closeups/selfies, swap out RetinaFace (used here as well) for MediaPipe in the mediapipe_utils.py files.