search

machanic / AU_R-CNN

The official implementation code of paper: "AU R-CNN：Encoding Expert Prior Knowledge into R-CNN for Action Unit Detection".
https://arxiv.org/abs/1812.05788
87 stars 24 forks source link

Modifying optical flow for two-stream network #14

Closed malkaddour closed 4 years ago

malkaddour commented 4 years ago

Hello, thanks a lot for uploading your paper implementation!

I was wondering if it's possible for me to modify the source code such that I can provide my own optical flow data for the BP4D dataset, and if so where I can do so?

Thanks a lot for your help!

amin717 commented 4 years ago

also AU_dataset.py doesn't exist in two_stream_rgb_flow/datasets file directory

machanic commented 4 years ago

@amin717 @machanic Sorry, I missed the AU_dataset.py file to upload github, I have uploaded now. I have pushed the AU_dataset.py file to https://github.com/machanic/AU_R-CNN/blob/master/two_stream_rgb_flow/datasets/AU_dataset.py Now you can find it.

The pretrained two steam (optical flow & RGB) model file (.npz) is located in https://cloud.tsinghua.edu.cn/d/8190d64e83ef43b7ab3f/?p=%2FBP4D_dataset_trained_model%2Ftwo_stream_models(RGB%2Boptical_flow)&mode=list and https://cloud.tsinghua.edu.cn/d/8190d64e83ef43b7ab3f/?p=%2FDISFA_dataset_trained_model%2Ftwo_stream_models(RGB%2Boptical_flow)&mode=list Also, I upload to google drive: https://drive.google.com/drive/folders/1F6cT9y7lxCI4RB7z-qPiv_otKwRnnv_B https://drive.google.com/drive/folders/16aJ6vc1lVTdV6JYtiIip0vLbrXtLSDG1 Thank you for interesting my work.

malkaddour commented 4 years ago

Thanks for your generous contributions! Do you know if it's possible for me to modify the code such that I can use my own optical flow data instead of the two stream architecture to obtain optical flow?

machanic commented 4 years ago

@malkaddour Yes, it is possible to use your optical flow, but based on my experince, the optical flow does not help to improve AU detection performance too much. Please see related section of my paper, it only helps to improve the convergence speed of the training.

malkaddour commented 4 years ago

Thank you very much! Yes indeed, I saw that but I was just curious to see how the choice of optical flow might affect the performance. It's possible for me to store precomputed optical flow in a separate directory and just load those while training, right?

machanic commented 4 years ago

Yes, this is what I have done before.

machanic commented 4 years ago

@malkaddour If you have interesting experiment finding, please let me know. Thank you.

machanic commented 4 years ago

Closing now. Feel free to reopen in case you want further clarifications!

amin717 commented 4 years ago

I downloaded the pretrained two steam (optical flow & RGB) model file (.npz) but when I set default value for pretrained_model_flow:

parser.add_argument('--pretrained_model_flow',default='/pretrained_model/DISFA_3_fold_3_resnet101@rgb_flow@use_paper_num_label@roi_pooling@T#10_model.npz', help="path of optical flow pretrained model, can also use the same npz with rgb")

I get this error:

KeyError: 'head/fc/b is not a file in the archive'

When I opened the .npz file, it contained the files of both the RGB and OF models. For this reason, I just put the OF parameters in another .npz file so that the algorithm could copy the parameters, but I was still justified by such errors.

KeyError: 'fc6/b is not a file in the archive' ValueError: could not broadcast input array from shape (64,20,7,7) into shape (64,3,7,7)

machanic commented 4 years ago

@amin717 The "--pretrained_model_rgb" and "--pretrained_model_flow“ in two_stream_rgb_flow/train.py is not intended to let you pass in the already trained two stream model. Note that the charaters of pretrained , pre means the model is normal AU R-CNN model file, rather than two-stream model . Because the file of two_stream_rgb_flow/train.py's purpose is to train a new model . After training, the model saves to the "DISFA_3_fold_3_resnet101@rgb_flow@use_paper_num_label@roi_pooling@T#10_model.npz" So --pretrained_model_rgb or --pretrained_model_flow should both be passed in "resnet101" or the normal trained AU R-CNN file (not two-stream,but the normal AU R-CNN trained file.). (e.g., BP4D_dataset/resnet101/3_fold_1_resnet101_linear_model.npz). I recommend to use the imagenet pretrained model that is "resnet101"

The file DISFA_3_fold_3_resnet101@rgb_flow@use_paper_num_label@roi_pooling@T#10_model.npz that I uploaded is the already trained model file, not pretrained file.

You can use this file to pass in two_stream_rgb_flow/evaluator_F1_main.py "--model" to test F1 score.

I will revise the description of "--pretrained_model_flow" and "--pretrained_model_rgb" of two_stream_rgb_flow/train.py to make it clearer.

malkaddour commented 4 years ago

@malkaddour If you have interesting experiment finding, please let me know. Thank you.

I will definitely let you know if I find anything interesting once I get it working, and sounds good! I'll reopen it in case I run into any issues that I can't figure out the solution to, thanks again!