HHTseng / video-classification

Tutorial for video classification/ action recognition using 3D CNN/ CNN+RNN on UCF101
940 stars 216 forks source link

Issue in Conv3D/Conv3D_video_predictions.py #6

Closed raghavgarg97-zz closed 5 years ago

raghavgarg97-zz commented 5 years ago

I have a couple of concerns. 1) Please change your transforms in Conv3D. They are 3 channel transforms but model works on grayscale images.So please average the mean and sd.simillarly during the training phase.I hope this is correct.

2) Your repo mentions that you are using 29 frames but I think your code only curently works on 28 frames.Am i correct? " selected_frames = np.arange(begin_frame, end_frame, skip_frame).tolist() " only selects [1,2.............28]

I really like the simplicity of your code.Great work man!

HHTseng commented 5 years ago

Hi raghavgarg97, Thank you for addressing the two concerns. You are right on both:

  1. Due to the gray images used for Conv3d, the Torchvision transformation needs to be changed to: transforms.Normalize(mean=[0.5], std=[0.5])] [for Torchvision 0.2.2 or above] or transforms.Normalize(mean=[0.449, 0.449, 0.449], std=[0.226, 0.226, 0.226])] [before Torchvision 0.2.1]

I think you mainly pointed out I should not use the RGB convention for processing grayscale images, and this should fix the issue.

  1. Indeed, minimal 28 frames was what I meant to say, sorry for the confusion.

Your careful comments are now amended in the repo. Deeply appreciate your response and please continue to let me know if you spot anything strange!

Sincerely, HTseng

raghavgarg97-zz commented 5 years ago

While training the ResNetCRNN network on 4 GPUs ,I got this error when 1st Epoch was about to be completed (98~99%)" RuntimeError: set_storage is not allowed on Tensor created from .data or .detach () " I think this issue is probably not in PyTorch 0.4.1 but with pytorch 1.1.0.Could you suggest changes so that it works in new Torch version? Update: I traced down the error to the fact that You have called flattenparameters() in RNNEncoder which gives error in Torch 1.1.0.Thanks to https://github.com/pytorch/pytorch/issues/21108 Update2: Using older verion of pytorch this problem is removed.Training using batch size 40..my accuracy after 6 epochs is not going beyonf 3%.Is is normal?

HHTseng commented 5 years ago

Regarding to the problem: RuntimeError: set_storage is not allowed on Tensor created from .data or .detach () You're right, the error was caused by flattenparameters() in RNNEncoder with Pytorch 1.1.0, which did not appear before Pytorch 1.0.1 post that I used earlier. One quick way to get rid of it is simply removing the line flattenparameters(), with the cost of using more memory I guess. Thank you for pointing this pytorch/pytorch#21108 out.

For ResNetCRNN with batch size 40, it is normal that accuracy is usually low in the first 10 epochs, it should start to show performance after 20 epochs or so. RNN seems takes longer time to learn temporal behavior.