Poor performance of checkpoint models

[vikrant-at-honda]

Hi @alexandrosstergiou,

Thank you so much for this amazing work.

However, when tried doing inference using the pre-trained checkpoints provided, the top-1 or top-5 is very less. Could you please help me in understanding if I am missing something out here?

Thank you so much in advance.

Dataset used is UCF101. Label processing done using scripts in labels folder.

Here is the log of inference for 90% video visibility: 2024-03-05 12:20:51 linux root[24581] INFO VideoIter:: iterator initialized (phase: 'val', num: 914)

2024-03-05 12:21:05 linux root[24581] WARNING Initialiser:: The following keys were missing: [] 2024-03-05 12:21:05 linux root[24581] WARNING Initialiser:: The following keys were not expected: [] 2024-03-05 12:21:05 linux root[24581] INFO Initialiser:: Only model state resumed from: /home/work/model/early_action_prediction/progressive_action_prediction/UCF/Tempr_h_movinet_ada_obs_09.pth' 2024-03-05 12:21:05 linux root[24581] INFO Running inference 2024-03-05 12:22:30 linux root[24581] INFO Inference: average top-1 acc: 0.00000 average top-5 acc: 0.02037 average loss 21.21562 2024-03-05 12:22:30 linux root[24581] INFO Inference: >> Sampler 0 average top-1 acc: 0.00000 average top-5 acc: 0.00399 average loss 16.58495 2024-03-05 12:22:30 linux root[24581] INFO Inference: >> Sampler 1 average top-1 acc: 0.03367 average top-5 acc: 0.09589 average loss 26.51506 2024-03-05 12:22:30 linux root[24581] INFO Inference: >> Sampler 2 average top-1 acc: 0.00000 average top-5 acc: 0.00347 average loss 24.16939 2024-03-05 12:22:30 linux root[24581] INFO Inference: >> Sampler 3 average top-1 acc: 0.00308 average top-5 acc: 0.03084 average loss 21.43339 2024-03-05 12:22:30 linux root[24581] INFO Inference: Label:Basketballaverage accuracy: 0.00000 num:36 2024-03-05 12:22:30 linux root[24581] INFO Inference: Label:BasketballDunkaverage accuracy: 0.00000 num:37 2024-03-05 12:22:30 linux root[24581] INFO Inference: Label:Bikingaverage accuracy: 0.00000 num:38 2024-03-05 12:22:30 linux root[24581] INFO Inference: Label:CliffDivingaverage accuracy: 0.00000 num:39 2024-03-05 12:22:30 linux root[24581] INFO Inference: Label:CricketBowlingaverage accuracy: 0.00000 num:36 2024-03-05 12:22:30 linux root[24581] INFO Inference: Label:Divingaverage accuracy: 0.00000 num:45 2024-03-05 12:22:30 linux root[24581] INFO Inference: Label:Fencingaverage accuracy: 0.00000 num:34 2024-03-05 12:22:30 linux root[24581] INFO Inference: Label:FloorGymnasticsaverage accuracy: 0.00000 num:36 2024-03-05 12:22:30 linux root[24581] INFO Inference: Label:GolfSwingaverage accuracy: 0.00000 num:39 2024-03-05 12:22:30 linux root[24581] INFO Inference: Label:HorseRidingaverage accuracy: 0.00000 num:49 2024-03-05 12:22:30 linux root[24581] INFO Inference: Label:IceDancingaverage accuracy: 0.00000 num:46 2024-03-05 12:22:30 linux root[24581] INFO Inference: Label:LongJumpaverage accuracy: 0.00000 num:39 2024-03-05 12:22:30 linux root[24581] INFO Inference: Label:PoleVaultaverage accuracy: 0.00000 num:40 2024-03-05 12:22:30 linux root[24581] INFO Inference: Label:RopeClimbingaverage accuracy: 0.00000 num:34 2024-03-05 12:22:30 linux root[24581] INFO Inference: Label:SalsaSpinaverage accuracy: 0.00000 num:43 2024-03-05 12:22:30 linux root[24581] INFO Inference: Label:SkateBoardingaverage accuracy: 0.00000 num:32 2024-03-05 12:22:30 linux root[24581] INFO Inference: Label:Skiingaverage accuracy: 0.00000 num:40 2024-03-05 12:22:30 linux root[24581] INFO Inference: Label:Skijetaverage accuracy: 0.00000 num:28 2024-03-05 12:22:30 linux root[24581] INFO Inference: Label:SoccerJugglingaverage accuracy: 0.00000 num:39 2024-03-05 12:22:30 linux root[24581] INFO Inference: Label:Surfingaverage accuracy: 0.00000 num:33 2024-03-05 12:22:30 linux root[24581] INFO Inference: Label:TennisSwingaverage accuracy: 0.00000 num:49 2024-03-05 12:22:30 linux root[24581] INFO Inference: Label:TrampolineJumpingaverage accuracy: 0.00000 num:32 2024-03-05 12:22:30 linux root[24581] INFO Inference: Label:VolleyballSpikingaverage accuracy: 0.00000 num:35 2024-03-05 12:22:30 linux root[24581] INFO Inference: Label:WalkingWithDogaverage accuracy: 0.00000 num:35 2024-03-05 12:22:30 linux root[24581] INFO > Avg class accuracy: 0.00000 2024-03-05 12:22:30 linux root[24581] INFO >> Avg class accuracy for sampler0: 0.00000 2024-03-05 12:22:30 linux root[24581] INFO >> Avg class accuracy for sampler1: 0.01273 2024-03-05 12:22:30 linux root[24581] INFO >> Avg class accuracy for sampler2: 0.00000 2024-03-05 12:22:30 linux root[24581] INFO >> Avg class accuracy for sampler3`: 0.00199 2024-03-05 12:22:30 linux root[24581] INFO --- Finished ---

[alexandrosstergiou]

Hi @vikrant-at-honda ,

Thanks for pointing this out. I've updated inference.py in the latest commit (#27074d9). There were some errors due to outdated code snippets for the video iterator and model loading. This should work now with the right parameters:

python inference.py --frame_size 224 --num_samplers 4 --gpus 0 --dataset UCF-101 --data_dir <some_dir>/ucf101/data --label_dir <some_dir>/ucf101/labels/split1 --workers 8 --backbone x3d --chkp TemPr_weights/UCF-101/Tempr_h_movinet_ada_obs_09.pth --pool ada --precision mixed --video_per 0.9

[vikrant-at-honda]

Hi @alexandrosstergiou,

Thank you so much for the update. I am afraid that I couldn't reproduce results again. I have one particular query about the data preprocessing part where you are trying to create csv files. I think that part of the code is maybe outdated as well. Please confirm and let me know if that is indeed correct.

[vikrant-at-honda]

I have one more request about release of pretrained models used for training. Please let me know from where to get those models.