search

gsig / charades-algorithms

Activity Recognition Algorithms for the Charades Dataset
http://allenai.org/plato/charades/
201 stars 96 forks source link

Unusual Training Behaviour #3

Closed ahmed-nad closed 7 years ago

ahmed-nad commented 7 years ago

I followed the instructions and proceeded towards retraining the rgbnet with default parameters, and ran it for 6 epochs with other parameters unchanged, I checked the epoch#.txt files and found out that the the network is predicting the same values for each video, the values change between epochs but stay the same for each video inside the file.

As a debug step i also predicted some images using RESNET inside and it was also giving same probabilities for any picture i feed in it, I tried every architecture

Ihave attached screenshot of one epoch file and .txt files for all epochs

screen shot 2017-07-21 at 1 14 21 pm [Uploading epoch1.txt…]() [Uploading epoch2.txt…]() [Uploading epoch3.txt…]() [Uploading epoch4.txt…]() [Uploading epoch5.txt…]()

gsig commented 7 years ago

Strange. Is is definitely loading in the pretrained imagenet network? Is it finding all the data?

Can you attach your full log file? (At least up to the first few epochs)

ahmed-nad commented 7 years ago

I solved the issue, i think it was a problem with caffe loader module which i compiled myself, it was linking to an old version of protobuf thus causing problems, i reinstalled everything and the problem disappeared