search

tomas-gajarsky / facetorch

Python library for analysing faces using PyTorch
Apache License 2.0
487 stars 43 forks source link

Questions about the AU detection #46

Closed tawichi closed 7 months ago

tawichi commented 1 year ago

Thank you for the amazing work!

I have questions about the AU detection model. When running the code, I got the following output.

'au': Prediction(label='lid_tightener', logits=tensor([1.4294e-01, 5.5219e-02, 7.3257e-01, 4.7374e-03, 6.6783e-01, 8.6254e-01,
        7.5628e-01, 6.6030e-01, 0.0000e+00, 4.2513e-01, 0.0000e+00, 3.2038e-01,
        1.4721e-01, 3.7695e-01, 1.6047e-01, 2.5481e-05, 0.0000e+00, 1.1697e-01,
        5.4448e-03, 5.7558e-02, 0.0000e+00, 8.3319e-01, 1.6647e-01, 4.1829e-02,
        0.0000e+00, 2.5487e-03, 0.0000e+00, 4.5418e-02, 0.0000e+00, 2.9258e-02,
        0.0000e+00, 8.7168e-03, 0.0000e+00, 1.9661e-02, 2.5690e-03, 6.4678e-03,
        1.5031e-02, 2.5029e-03, 0.0000e+00, 6.2914e-03, 0.0000e+00]), other={'multi': ['brow_lowerer', 'cheek_raiser', 'lid_tightener', 'nose_wrinkler', 'upper_lip_raiser', 'lips_part']})

Could you clarify what the output "label='lid_tightener'" and "other={'multi': ['brow_lowerer', 'cheek_raiser', 'lid_tightener', 'nose_wrinkler', 'upper_lip_raiser', 'lips _part']}" signify? What is the difference between "label" and "multi"? Also, what do the "logits" represent? I think it probably shows the predicted probabilities of 41 different AUs, but I don't know in which order. Lastly, in the original paper, there appear to be two models, one trained with DISFA and the other trained with BP4D. Which model is used in this case?

tomas-gajarsky commented 1 year ago

Yes, logits are the confidence scores, the label is the one with the highest confidence, and multi contains all the labels that had confidence higher than a threshold specified in the config file, where you will also find an ordered list of all supported AU labels.