[about the paper] PPDN, FN2EN comparisons

speculaas commented 6 years ago

Dear Cugu, Thanks for your help! Your group's paper mentioned PPDN, FN2EN's running time on GTX1050 and i7. My question is how you find FN2EN's sources?

I cannot find FN2EN's implementation via Google even though the paper says "Upon publication, the trained expression models will be made publicly available"

And thanks again for sharing your work!

BR, JimmyYS

cuguilke commented 6 years ago

Authors of FN2EN shared a pretrained caffemodel for CK+ and the Caffe implementation of their model, but as far as I know there is no other available implementation of the model with other frameworks like Keras or Tensorflow. Long story short, you can find it in http://www.huiding.org/pub/

speculaas commented 6 years ago

Dear Cugu, Thanks for your help! Embarrassingly, I never used caffe.

By "Caffe implementation of their model", do you mean "deploy.prototxt" in the zip file? Any else? All you need to run a pre-trained model in caffe is deploy.prototxt?

BR, JimmyYS

cuguilke commented 6 years ago

Yes, "deploy.prototxt" contains the model specifications, Caffe uses them to build its internal model. "caffemodel" provides you the learned weights & biases in the network.

There is a nice blog post about Caffe in which all of these files are explained: http://adilmoujahid.com/posts/2016/06/introduction-deep-learning-python-caffe/

speculaas commented 6 years ago

Dear Cugu, Thanks for your generous help!

Another question, did you get the CK+ dataset from here: http://www.consortium.ri.cmu.edu/ckagree/ ?

Or the CK+ dataset webpage's url changed?

The web page does not show and I get the following response : Gateway Timeout Server error - server 128.2.177.138 is unreachable at this moment. Please retry the request or contact your administrator.

BR, JimmyYS

cuguilke commented 6 years ago

Yes, I did. The website is working now.

speculaas commented 6 years ago

Dear Cugu, I saw your response at https://github.com/cuguilke/microexpnet/issues/7 about "labelPath" format.

My question is how you get from the raw CK+ data set to (path, label) pairs in a single txt?

/home/username/Desktop/OuluCASIA/batch0/example.jpeg 0 /home/username/Desktop/OuluCASIA/batch1/example2.jpeg 3 ... /home/username/Desktop/OuluCASIA/batch9/example13.jpeg 2

Did you wrote a script to traverse the CK+ data set's directories? And then read the image path and labels and write to a single txt? Did you upload such script?

And can you share how to do distillation?

I briefly read through your code, I see the following line: ./Candidates/trainCandidates.py:95: testX, testY, testSoftTargets = deployImages(valSet, teacherSoftmaxInputs)

But I cannot find how to put teacher model's logits into the teacherSoftmaxInputs variable. BR, JimmyYS

speculaas commented 6 years ago

Dear Cugu, Another question about "labelPath" format:

does the (path label) in the same batch has to be next to each other?

/home/username/Desktop/OuluCASIA/batch0/example.jpeg 0 /home/username/Desktop/OuluCASIA/batch1/example2.jpeg 3 ... /home/username/Desktop/OuluCASIA/batch9/example13.jpeg 2

because now, my labels_CK.txt looks like:

microexpnet/dataset/batch1/S076_006_00000019.png 5 microexpnet/dataset/batch2/S131_001_00000016.png 7 microexpnet/dataset/batch0/S100_005_00000023.png 1 microexpnet/dataset/batch3/S089_002_00000021.png 5 microexpnet/dataset/batch6/S092_001_00000017.png 7 microexpnet/dataset/batch7/S160_006_00000010.png 2 microexpnet/dataset/batch1/S063_002_00000023.png 5 microexpnet/dataset/batch1/S079_001_00000012.png 7 microexpnet/dataset/batch2/S506_004_00000038.png 4 microexpnet/dataset/batch7/S097_004_00000030.png 3 microexpnet/dataset/batch6/S132_002_00000018.png 6 microexpnet/dataset/batch7/S135_001_00000039.png 7 microexpnet/dataset/batch9/S064_004_00000014.png 6 microexpnet/dataset/batch2/S090_007_00000014.png 1 microexpnet/dataset/batch4/S101_002_00000019.png 7 microexpnet/dataset/batch8/S083_003_00000019.png 5 microexpnet/dataset/batch8/S014_002_00000016.png 6

And I got error when I run : "python trainCandidates.py ..."

BR, JimmyYS

speculaas commented 6 years ago

Dear Cugu, To answer my own question in previous comment,

if I arrange the (path, label) pairs in labels_CK.txt that are in the same batch to be next to each other (as below) then "python trainCandidates.py ..." can proceed without error :

microexpnet/dataset/batch0/S022_001_00000030.png 7 microexpnet/dataset/batch0/S032_004_00000014.png 4 microexpnet/dataset/batch0/S035_005_00000019.png 3 microexpnet/dataset/batch0/S042_004_00000020.png 1 microexpnet/dataset/batch0/S066_004_00000010.png 6 ... microexpnet/dataset/batch1/S005_001_00000011.png 3 microexpnet/dataset/batch1/S014_005_00000017.png 5 microexpnet/dataset/batch1/S022_006_00000017.png 3 microexpnet/dataset/batch1/S026_006_00000013.png 5 microexpnet/dataset/batch1/S032_006_00000016.png 5 microexpnet/dataset/batch1/S042_002_00000016.png 6 ...

What's remaining is how to do distillation, and hope it's okay for you to share how to do distillation. Thanks for your great work and generous sharing already.

BR, JimmyYS

cuguilke commented 6 years ago

Candidates are not trained with distillation, you have to modify them manually for that.

For distillation, you can use MicroExpNet.py which takes

softy = tf.placeholder(tf.float32, shape=[None, nClasses])

as "teacherLogits".

Then, training part becomes something like this:

c, p = sess.run([classifier.cost, classifier.accuracy], feed_dict={x: batch_x, y: batch_y, softy: batch_softy, lr: learningRate, keepProb: 1.0})

speculaas commented 6 years ago

Dear Cugu, Thanks for your help!

Some more questions:

accuracy
whether training data set is too small
can you share how Resnet50 is trained to beat state of the art? which layers are fine tuned?

More details regarding Q1 and Q2:

About accuracy, I trained : "python trainCandidates.py -v 9 ../labels_CK.txt ./Graphs/candidateV.png ./Models/tfmodel_candidateV 0.0001 1"

for 100000 epochs, but I only get 0.18 test accuracy I think maybe I used too few training data.

candidatev

About training data size too small: from CK+, there are only 327 image with emotion labels I used only these 327 images to train. Have you train such training before?

Do I have to use unlabeled as neutral? And also for each label, take last 3 frame and first frame?

looking at the label: S084_002_00000023.png 4 it specified the frame number 23, as fear however the first frame looks neutral to me, and this points to my unease: is it okay to use first frame have the same emotion as labeled frame? And are all the unlabeled images truly neutral?

However if I don't try to add more training data, using only label 327 images, the accuracy is really bad.

BR. JimmyYS

speculaas commented 6 years ago

Dear Cugu,

I used last 3 frame as labeled and first frame as neutral. And also first frame of unlabeled as neutral. total 1574 images exactly as mentioned in your paper. The accuracy improved a lot.

However, when squeeze=16, pooling is better than no pooling, this seemed different from your result. Do you know why? Any different setup compared to yours? Or it's because I didn't do distillation?

training_acc_microexpnet

BR, JimmyYS

cuguilke commented 6 years ago

I assume you are using the preprocessing phase of the inputs we put here. Your results are very bad considering the results reported in literature for CK+. Few quick points:

Are these the average 10-fold cross validation results? If not, then it is the first step to do.
Assuming they are obtained from 10-fold cross validation, we separated batches in a way that, each batch has images from all emotion labels (all images are shuffled per class) with nearly the same number of samples.
In addition, we gave models 3 chances to learn for each validation batch. I mean, initial weights are random and it may directly lead the model to learn nothing, so we roll the dice three times. If model starts to learn, than we say the reason was the initialization.
Distillation is not relevant, because we didn't do that for candidates either.

If you are not using our preprocessing stuff, then I need to know the specifications.

speculaas commented 6 years ago

Dear Cugu, Thanks for your help! The above result is from 10-fold cross validation since I didn't modify your training script. And by average, do you mean: I pick fold no 1 as test set, get one result, then use fold no.2 as test and get another result, and so on, and then average 10 training accuracies?

About each batch having the all emotions and equal numbers of each emotion, I haven't implement this yet. I will try this and see if accuracy improves.

About 3 chances to learn, I will try this also. By the way did you put batch normalization in your neural network? I think you did not? Do you think batch normalization can improve training?

And thanks for your help again!

BR, JimmyYS

speculaas commented 6 years ago

Dear Cugu, did you use Viola and Jones to crop smalling containing rectangular area? Because I can not find code that does this cropping step

BR, JimmyYS

speculaas commented 6 years ago

Dear Cugu, Sorry I am vague in previous comment. I mean, during trainCandidates, did you use Viola and Jones to crop smallest rectangular area containing face? I asked because I see you mentioned Viola and Jones to crop face, But in trainCandidates and Preprocessing, I cannot find Viola and Jones.

BR, JimmyYS

speculaas commented 6 years ago

Dear Cugu, From a recent paper: Compact Deep Learning Model for Robust Facial Expression Recognition

I found a potential problem.

The above paper mentioned : To avoid subjects appearing in both the training and testing sets simultaneously, we divide the subjects into 10 subsets by their IDs in the ascending order, which is the same as the 10-fold cross validation protocol in [24].

And after checking my implementation, I found same subject and emotion appeared in multiple batch. Therefore, test accuracy might come from testing training data.

For example, in my dataset : 'S505_002', appeared in batches : ['9', '0', '6']

If I avoid same subject and emotion in multiple batch, the test accuracy drops significantly.

Did you check whether different batches (folds) in your dataset contain the same subject+emotion?

BR, Jimmy

cuguilke commented 5 years ago

Paper is updated

cuguilke / microexpnet

[about the paper] PPDN, FN2EN comparisons #8