search

yfeng95 / DECA

DECA: Detailed Expression Capture and Animation (SIGGRAPH 2021)
Other
2.15k stars 425 forks source link

Failure cases and cropping #52

Open DaWelter opened 3 years ago

DaWelter commented 3 years ago

Hi

I've been using DECA to auto-label VGGFace (so far) because I want to train a smaller pure head pose estimator on it.

Thanks a lot for publishing your code and your model. It's pretty awesome work I think.

But I noticed that failures are relatively frequent. Here is an example: image It's actually from AffectNet. The failures all look similar, in so far as a front facing slightly distorted face is predicted. Counting them manually I found them to occur on roughly 2-5% of the images.

Secondly, I found DECA's results are really sensitive to the cropping region. Here is another example illustrating that. Not using automatic cropping: image Normal usage: image It's documented so that's fair enough I guess. But I wonder if DECA could be made more robust with stronger scale and translation augmentation.

(Does there happen to be somebody wanting to try to use the photometric fitting code with "DECA initialization" to make accurate data sets of FLAME fits?)

Anyway, thought I should make a report.

Cheers Michael

DaWelter commented 3 years ago

Nevermind! My branch was outdated. I think your last patch with the scale factor fixed most of my problems.

DaWelter commented 3 years ago

P.S. In my own auto-labeling script I was using the MTCNN detector because it is much faster than FAN and consumes less memory. But I also got the failures I described above.

Now I switched back to FAN, and implemented a fallback to cpu for when it runs my gpu out of memory which is very rare but I have to deal with it. With these measures I get very decent results.

Thanks again for your great work :+1:

lhyfst commented 3 years ago

Good observation! I found them, too.

ty625911724 commented 3 years ago

Have you fixed the failure cases that predicting distorted faces? I meet the similar problem and do not know how to fix it. I just using the pretrained weights provided by the authors. Thanks for your great help : )

DaWelter commented 3 years ago

I use the pretrained weights, too. As I wrote, using the most recent master branch worked for me. The DECA network does not like the way MTCNN picks the bounding boxes. FAN gives me much better results. Sometimes the DECA networks still messes up. But it's rarer than before the scaling fix. I didn't count the failures tbh. Must be in the order of magnitude of 1 in 100 or less.

ty625911724 commented 3 years ago

Thanks! I just want to fix the 1% failure cases but do not know how to do it.

mkh321 commented 3 years ago

What should I do when I train with such a result? And it happens very frequently. 000000