search

laura-wang / video-pace

code for our ECCV-2020 paper: Self-supervised Video Representation Learning by Pace Prediction
99 stars 12 forks source link

Cannot reproduce the supervised performance on UCF101 #9

Closed KT27-A closed 3 years ago

KT27-A commented 3 years ago

Thank you very much for your inspiring work. However, I encountered a problem when reproducing the performance. I followed your code to do the self-supervised learning. I got about 60-70% accuracy in pace prediction. However, when I freeze the Conv weights and only train the final FC layer for supervised learning, I just got 0.10 average accuracy on training. When training final FC, I used the same data augmentation method as self-supervised learning as your paper said. Could you please tell me more about the fine-tuning details?

AronCao49 commented 3 years ago

Initially, I have a similar question about the fine-tuning strategy during the implementation. However, I find that the performance can be reproduced if you unfreeze the Conv weight during the fine-tuning stage. According to the paper, it seems that while the Conv weights are retrieved from the pre-trained model, they are trained together with the FC layer during fine-tuning.

Regarding the action recognition task, during the fine-tuning stage, weights of convolutional layers are retained from the self-supervised learning networks while weights of fully-connected layers are randomly initialized. The whole network is then trained with cross-entropy loss.

Anyway, you could try to unfreeze all the weights and see how it goes. In my case, I can reproduce the results when using R(2+1) on UCF101.

KT27-A commented 3 years ago

Thank you very much for your response. I also reproduced the results when I fine-tuned the whole network. Now I have been trying to reproduce another paper "Video Representation Learning by Recognizing Temporal Transformations". They trained on a pretext task and got ~50% on UCF101, but I just got 20%.

AronCao49 commented 3 years ago

I haven't reproduced that work so far. Maybe you can double-check other settings (batch size per GPU for example) compared to the original paper. Anyway, wish you all the best with your work.

KT27-A commented 3 years ago

Thanks. You too.

Hussein-A-Hassan commented 3 years ago

Initially, I have a similar question about the fine-tuning strategy during the implementation. However, I find that the performance can be reproduced if you unfreeze the Conv weight during the fine-tuning stage. According to the paper, it seems that while the Conv weights are retrieved from the pre-trained model, they are trained together with the FC layer during fine-tuning.

Regarding the action recognition task, during the fine-tuning stage, weights of convolutional layers are retained from the self-supervised learning networks while weights of fully-connected layers are randomly initialized. The whole network is then trained with cross-entropy loss.

Anyway, you could try to unfreeze all the weights and see how it goes. In my case, I can reproduce the results when using R(2+1) on UCF101.

Hello @AronCao49 Could you please post the evaluation code for the action recognition that you used to reproduce the same results? What is the exact sampling method when sampling 10 clips from each video in the testing split of the UCF ?

Thanks