search

dkollias / Aff-Wild-models

77 stars 19 forks source link

How to evaluate my model? #2

Open Andyeyeye opened 4 years ago

Andyeyeye commented 4 years ago

Hello dkollias, Thanks for sharing your work! I'm interested in this data set and want to train a model by training all the faces which I crop based on bounding boxes. There are some problems I have:

  1. I found that many faces I cropped from video are in the trailer like Jon Snow's face. Is the annotation of that face?
  2. I want to train my model not using RNN or any sequence based methods. Have you ever tried to train a CNN to extract features and then feed that features to a regression method before? In paper, you just mentioned using SVR but not using a CNN model with it.
  3. Can I get the test set annotation to evaluate my model? Since there is no test annotation, so I want to ask how to evaluate my own data set without test set ground truth. To evaluate my models before, I analyzed test accuracy by comparing ground truth and my predictions. This is my first time to use a huge data set and train a non-classification model. I appreciate your patience to read and help.

Thanks, Andy

dkollias commented 4 years ago

Hello Andy,

thank you for your interest in this work! Regarding your problems/issues:

  1. the annotation is only for the people watching the video/trailer and reacting (and not people appearing in the trailers like Jon Snow). You will need to address this issue. You can use a tracker or you can compute the center of the bounding box and when you see from frame to next frame that the center is moved a lot then probably the detector is detecting another face, like Jon snows, etc. There are different ways to solve this issue I just mentioned a couple.
  2. Actually no I haven't and would be interesting to see the performance in this case.
  3. Unfortunately the annotations for the test videos are not provided in order to keep the integrity of the challenge (as we will re-run it). You can send me your predictions and I am more than happy to evaluate the models for you. For more details on the exact format and general details have a look here: https://ibug.doc.ic.ac.uk/resources/first-affect-wild-challenge/ By the way, the performance measure should be CCC or MSE since this is a regression problem (and not accuracy that is for classification problems). Please let me know should you need any further assistance or something seems unclear.

Kind Regards, Dimitris