dchen236 / FairFace

465 stars 92 forks source link

Many incorrectly labeled images #8

Open kylemcdonald opened 4 years ago

kylemcdonald commented 4 years ago

The paper reads "We further refined the annotations by training a model from the initial ground truth annotations and applying back to the dataset. We then manually re-verified the annotations for images whose annotations differ from model predictions."

But I tried to do this check myself, and I found many examples of annotations that do not match the model predictions. I do not have an estimate for how many images have this problem. But here are a few samples from the validation dataset that are annotated as "White", along with the model's predictions for these images. I would guess 2-3 of these 64 people would self-identify as "White":

Filename Predicted Race
val/1914.jpg Indian
val/5302.jpg Black
val/8590.jpg Southeast Asian
val/8963.jpg Black
val/9763.jpg Southeast Asian
val/9377.jpg Southeast Asian
val/7653.jpg Southeast Asian
val/2173.jpg Black
val/6261.jpg Indian
val/2698.jpg Black
val/322.jpg Southeast Asian
val/7489.jpg East Asian
val/2865.jpg Black
val/7394.jpg Southeast Asian
val/6331.jpg Black
val/8906.jpg East Asian
val/3797.jpg East Asian
val/5689.jpg Latino_Hispanic
val/7191.jpg East Asian
val/1312.jpg Indian
val/1399.jpg Indian
val/5204.jpg Latino_Hispanic
val/1758.jpg East Asian
val/7019.jpg Black
val/5771.jpg Southeast Asian
val/3903.jpg Latino_Hispanic
val/3204.jpg Middle Eastern
val/10556.jpg Latino_Hispanic
val/8838.jpg Southeast Asian
val/9757.jpg Latino_Hispanic
val/6590.jpg Latino_Hispanic
val/144.jpg Southeast Asian
val/10507.jpg Latino_Hispanic
val/1554.jpg Latino_Hispanic
val/7518.jpg Indian
val/5563.jpg Black
val/209.jpg Indian
val/10349.jpg Latino_Hispanic
val/8969.jpg Black
val/8475.jpg Black
val/5485.jpg Latino_Hispanic
val/4649.jpg Latino_Hispanic
val/68.jpg Southeast Asian
val/1286.jpg East Asian
val/2777.jpg Latino_Hispanic
val/397.jpg Latino_Hispanic
val/6448.jpg East Asian
val/1173.jpg Indian
val/10222.jpg Southeast Asian
val/4156.jpg Southeast Asian
val/7783.jpg Latino_Hispanic
val/10794.jpg East Asian
val/1309.jpg Latino_Hispanic
val/5787.jpg East Asian
val/9198.jpg Latino_Hispanic
val/4890.jpg Southeast Asian
val/6822.jpg Latino_Hispanic
val/8659.jpg Latino_Hispanic
val/953.jpg East Asian
val/10843.jpg Latino_Hispanic
val/8177.jpg East Asian
val/8870.jpg Latino_Hispanic
val/9399.jpg Latino_Hispanic
val/3652.jpg Latino_Hispanic
MandoraCC commented 4 years ago

Excuse me, may I ask you that how you get the FairFace dataset? I found that I can not find the download link of FairFace.

cpshaheen commented 3 years ago

Excuse me, may I ask you that how you get the FairFace dataset? I found that I can not find the download link of FairFace.

https://drive.google.com/file/d/1Z1RqRo0_JiavaZw2yzZG6WETdZQ8qX86/view

if you scroll to the bottom of the readme the link for this dataset (padding 0.25) and another are available

Best of luck on your endeavor!

joojs commented 3 years ago

The annotations may not be perfect due to the subjective nature of the task. The paper is about perceived, not self-identified, race (and also gender, age), like most existing face attribute classification papers/datasets.

ufousoumo commented 3 years ago

Excuse me,may I ask you that how you get the pretrained models? I could not find the download link of the pretrained models.