Values are not matching the reported values

[avijit9]

Hi,

I'm trying to reproduce the results reported in the paper using this repo. But the value for action classification on Thumos14 dataset is somewhat not matching the reported result (I am getting 71.67 mAP instead of 86.9 reported in the paper). I have use all the default parameters. Is there something I am missing?

Other values are somewhat close to the values reported.

I am waiting for your reply. Thanks in advance.

[naraysa]

Hi,

We obtained the results (for both classification and detection) using this exact same code of the repo and the I3D features provided in W-TALC repo. If you are running with default parameters, then I would suspect it is the features, which are causing this issue.

[avijit9]

Hi,

Thanks for your reply. I downloaded the features from the link mentioned. There are two files related to Thumos14 with I3D namely Thumos14 and Thumos14reduced. Which one did you use? I used the former one.

[avijit9]

Hi,

I have run the experiments again. The performance is still the same. Can you please check and suggest something?

Thanks in advance.

[naraysa]

We used the Thumos14 dataset with 101 classes.

I have re-run again (partially), using the default command for Thumos14. I am getting the classification accuracy around 87% from 1000 iterations itself. Log attached for reference. Thumos143cnet-results.log

[avijit9]

Are you using Pytorch 1.0 or 0.4.0? I am using 1.0. Could this be a problem?

[avijit9]

I tested with the Pytorch 0.4 too. The problem remains same. Is it possible to share the data files you are using? Though, I downloaded from the W-TALC repo, just wanted to make sure that everything else is fine. The following is the testing result after 1.5K iterations.

Testing test data point 0 of 1574
Testing test data point 100 of 1574
Testing test data point 200 of 1574
Testing test data point 300 of 1574
Testing test data point 400 of 1574
Testing test data point 500 of 1574
Testing test data point 600 of 1574
Testing test data point 700 of 1574
Testing test data point 800 of 1574
Testing test data point 900 of 1574
Testing test data point 1000 of 1574
Testing test data point 1100 of 1574
Testing test data point 1200 of 1574
Testing test data point 1300 of 1574
Testing test data point 1400 of 1574
Testing test data point 1500 of 1574
71.21894972183733
Testing for IoU 0.100000
Testing for IoU 0.200000
Testing for IoU 0.300000
Testing for IoU 0.400000
Testing for IoU 0.500000
Testing for IoU 0.700000
Classification map 71.218950
Detection map @ 0.100000 = 53.271604
Detection map @ 0.200000 = 46.245738
Detection map @ 0.300000 = 36.436472
Detection map @ 0.400000 = 28.423280
Detection map @ 0.500000 = 21.451247
Detection map @ 0.700000 = 6.593520
Mean Detection map = 32.070310

As you can see, detection mAPs are close to what given in your log file. But the classification one is not.

Thanks.

[naraysa]

We used Pytorch 0.4.1. We haven't tested on 1.0. Please see attached file for the environment we used. Though, not all dependencies listed in the file are required. 3cnet_env.txt

[avijit9]

Hi,

What do you suggest then? For ActivityNet I am getting the comparable results though. Is there any subtle difference in the parameters that we might have missed?

[naraysa]

The parameters are the default ones. I have shared the original features i had downloaded from WTALC repo. You can try with these features. https://drive.google.com/open?id=1APhxRLhiVsmh8blMm_KNGpKwuT7OMRky

[avijit9]

Hi, Thanks.

I downloaded the features file you provided above, recloned the repo and ran the code again. But the performance is still the same even after 34K iterations. :|

Testing test data point 0 of 1574
Testing test data point 100 of 1574
Testing test data point 200 of 1574
Testing test data point 300 of 1574
Testing test data point 400 of 1574
Testing test data point 500 of 1574
Testing test data point 600 of 1574
Testing test data point 700 of 1574
Testing test data point 800 of 1574
Testing test data point 900 of 1574
Testing test data point 1000 of 1574
Testing test data point 1100 of 1574
Testing test data point 1200 of 1574
Testing test data point 1300 of 1574
Testing test data point 1400 of 1574
Testing test data point 1500 of 1574
70.7731877814146
Testing for IoU 0.100000
Testing for IoU 0.200000
Testing for IoU 0.300000
Testing for IoU 0.400000
Testing for IoU 0.500000
Testing for IoU 0.700000
Classification map 70.773188
Detection map @ 0.100000 = 59.492297
Detection map @ 0.200000 = 54.128837
Detection map @ 0.300000 = 44.390475
Detection map @ 0.400000 = 33.828924
Detection map @ 0.500000 = 26.243971
Detection map @ 0.700000 = 8.161848
Mean Detection map = 37.707725

Is it possible to get a zipped version of your code from which you generated the log file above?

[naraysa]

Not sure what the issue is, since it is not reproducible for me. And it is outputting correct detection values.

I ran it with the latest commit of the repo. I have shared it here anyway. 3cnet-zip.zip

[avijit9]

Thanks. I'll try it out. Btw, which GPU do you use for this experiment?

[avijit9]

Hi,

I have found out the problem. Actually, I kept the features files and annotation files in SSD and the code on HDD. Now, when running the code I was giving the full path of the data as --dataset-name. As a result everything was fine except this block of code was not running - https://github.com/naraysa/3c-net/blob/master/test.py#L47

Thanks a lot for helping me out :)