Cannot reproduce performance reported in the paper, especially for UCF-101-24

[kinux98]

First of all, thanks for sharing this great work!

I have tried to reproduce the performance of UCF-101-24 using your code, but it failed.

(100 epoch, vmap@0.2 : 90.02 vs 95.1 (in the paper) / fmap@0.2 : 84.62 vs 89.1 (in the paper)

In addition, the current version of the code is different from what you described in your README.md, so I'm having trouble reproducing the experiment.

It would be grateful if you could tell me how to run the experiments of UCF-101-24 based on the current version of the code.

[AKASH2907]

Explicitly what differences between README and code? Can you please point out the lines so I can look into it? What is the performance at 0.5 metrics?

[kinux98]

1. differences between README and code :

In the README.md, there are no explicit training commands for UCF-101-24 or Jhmdb. ( only with running the code respectively for variance and gradient maps, not for each dataset).

I hope you to provide more direct and detail instruction for training UCF-101-24 / JHMDB (for example, python3 main_ucf101.py --args.. ).

2. Performance at 0.5 for UCF-101-24

training with following command with default arguments in the code : python main_ucf101.py --bv --gv --cyclic --exp_id=ucf-101-24_test --bs=8 --epochs=100 --wt_loc=1 --wt_cls=0.3 --wt_cons=0.7 --thresh_epoch=11 --n_frames=5

-- results --

vmap@0.2 : 90.02 (reproduced) | 95.1 (in the paper) fmap@0.2 : 84.62 (reproduced) | 89.1 (in the paper)

vmap@0.5 : 50.22 (reproduced) | 71.8 (in the paper) fmap@0.5 : 52.05 (reproduced) | 69.8 (in the paper)

Thanks for your help!

[AKASH2907]

I see you are running both variance and gradient at the same time. Can you please run once this command?

python main_ucf101.py --epochs 100 --bs 8 --loc_loss dice --lr 1e-4 --pkl_file_label train_annots_20_labeled.pkl --pkl_file_unlabel train_annots_80_unlabeled.pkl --wt_loc 1 --wt_cls 1 --wt_cons 0.1 --const_loss l2 --bv --n_frames 5 --thresh_epoch 11 --exp_id cyclic_variance_maps

[kinux98]

Hi, i got the following results :

vmap@0.2/0.5 : JHMDB : 95.94 (94.8) / 65.39 (62.8) UCF101-24 : 93.18 (95.2) / 56.58 (71.8) fmap@0.2/0.5 : JHMDB : 86.93 (89.2) / 63.14 (63.6) UCF101-24 : 88.24 (89.6) / 56.20 (69.8)

(.) indicates result from the paper.

Note that, @0.5 shows seriously low performance than that of @0.2 in the case of UCF101-24..

[AKASH2907]

I'm not sure why you're getting that score. I just evaluated and I'm getting these scores: IoU f-mAP: 0.0 1.0 0.05 0.9320641867415654 0.1 0.9240938990295099 0.15 0.9135093983188565 0.2 0.9010715984895877 (89.6) 0.25 0.8837437625673311 0.3 0.8613493699592949 0.35 0.8317839511186788 0.4 0.7944087199101718 0.45 0.7501351896891274 0.5 0.6994532221040055 (69.8) 0.55 0.6412284970386476 0.6 0.5695245172312181 0.65 0.48511241030942376 0.7 0.38497744167524556 0.75 0.2721435638659735 0.8 0.1571815056086843 0.85 0.06581932237497974 0.9 0.01282576140636419 0.95 0.0008423108219138969

IoU v-mAP: 0.0 1.0 0.05 0.9834135228872071 0.1 0.9744290685080159 0.15 0.9701037641827116 0.2 0.9581593274292763 (95.2) 0.25 0.9295676588970306 0.3 0.8987566061299131 0.35 0.87217743714003 0.4 0.8289668671679272 0.45 0.785389444484375 0.5 0.7235224452591954 (71.8) 0.55 0.6369963845209002 0.6 0.5477197691609584 0.65 0.4278471287055454 0.7 0.27486963952644616 0.75 0.140720797587886 0.8 0.05307216223377892 0.85 0.012169446051024997 0.9 0.0 0.95 0.0

I verified the loader and model file with my local files.

[kinux98]

Ok, I will try to run the experiment again and reopen this issue when i encounter the problems. Thank you for your assistance.

[AKASH2907]

Yeah, sorry not able to figure out the issue exactly. You can print all maps once and check. Replace these lines at line 188 in evaluate_ucf101.py file to get scores at all maps.

        #config.write_output('IoU f-mAP:\n')
        for i in range(20):
            print(iou_threshs[i], fmAP[i])
        print(fAP[:, 10])
        print('IoU v-mAP:')
        for i in range(20):
            print(iou_threshs[i], vmAP[i])
        print(vAP[:, 10])