search

TadasBaltrusaitis / OpenFace

OpenFace – a state-of-the art tool intended for facial landmark detection, head pose estimation, facial action unit recognition, and eye-gaze estimation.
Other
6.83k stars 1.83k forks source link

False prediction when head is oriented #719

Open abdou31 opened 5 years ago

abdou31 commented 5 years ago

Hello Tadas , I have writed a issue that explain what i want to do ( annotating dataset using OpenFace ) to train this dataset by CNN Facial landmarks

I have get false prediction , after verify the dataset and points , i have found some problems that due to this false results. this is a annotation created by OpenFace : https://user-images.githubusercontent.com/19480228/59348270-f58d8400-8d16-11e9-9949-70f24c292b93.jpg

Do you have any idea how can i correct this prediction when the head is oriented ? Note : I don't use 3D values ( coordinates points ) . Thanks

TadasBaltrusaitis commented 5 years ago

It is a reasonably challenging problem knowing if landmark detection succeeded or failed (if it was easy it would be simple to build a perfect landmark detector).

One thing you could possibly use is the likelihood parameters in OpenFace that can estimate how reliable the landmark detection is (each landmark has a likelihood term). However, these are not included in the csv file and you would have to modify the code in order to access and output them.

You can access them through face_model.landmark_likelihoods

abdou31 commented 5 years ago

You mean that i should download the project source not the binary (exe project ), modify the code and build again openface project If yes , can you tell me what should i change exactly? I mean what file should i acceded? Sorry for my bad english

TadasBaltrusaitis commented 5 years ago

Exactly, you will need to change the source code yourself. The changes will not be trivial, so I cannot give you exact instructions, but some rough guidelines:

  1. You will want to access the face_model.landmark_likelihoods for both the 68 landmark points of the face and 28 + 28 landmark points of the eye region
  2. You will want to output their values as part of the RecorderCSV to be able to output them as part of the CSV file
  3. You will need to work out what is a good cutoff for trusting the landmark detection of them.
abdou31 commented 5 years ago

Ok thanks, Tadas. I think this causes also a problem, the reason is that CNN facial landmarks accept only x and y coordinates from pts files. Example of pts file : imgg

If output values do not need to get extracted to pts file, so it doesn't cause any problem