Cannot replicate Kinetics-400 Results

[ilkarman]

Thank you very much for posting the codebase! However, I'm having difficulty replicating the 71.0% acc mentioned in the paper for 'bLVNet-TAM-8×2', I only get around 55% when validating each epoch. Training accuracy is 58%. If I do 10 crops per video for validation this rises to around 57%

Would it be possible to share your training log file? Or know your final training accuracy to see if something is wrong with my validation/eval script?

Wanted to check if any of the below was wrong:

• TAM Backbone is initialised from 'ImageNet-bLResNet-50-a2-b4.pth.tar'
• For bLVNet-model:

  {'depth': 50, 'alpha': 2, 'beta': 4, 'groups': 16, 'num_classes': 400, 'dropout': 0.5, 'blending_frames': 3, 'input_channels': 3, 'pretrained': None, 'dataset': 'kinetics400', 'imagenet_blnet_pretrained': True}

• For the training: 0.01 LR with a total batch of 64 (8 * 8 per GPU); cosine-annealing LR-schedule trained for 50 epochs

I thought perhaps this should be trained for 100 epochs; similar to TSM, not 50?

[chunfuchen]

As you pointed out in #4, the pretrained ImageNet model might not be loaded correctly, do you get this results after fixed the issue?

[ilkarman]

Thanks very much for your reply! Unfortunately this is still the case for me despite loading pretrained weights. I was wondering whether you could share your training/validation accuracy for a few epochs (not the multi-crop, multi-clip validation) to help debug?

[ilkarman]

I thought it could be more helpful to leave this log below (r50_a2_b4_f8x2). Initiating from pre-trained weights and then training 50 epochs with cosine-annealing from LR of 0.01 for a batch of 64 samples:

Epoch 0 Validation Acc: 0.09
Epoch 10 Validation Acc: 0.29
Epoch 20 Validation Acc: 0.35
Epoch 30 Validation Acc: 0.41
Epoch 40 Validation Acc: 0.47
Epoch 49 Validation Acc: 0.53

Using LR=0.001 I get:

Epoch 0 Validation Acc: 0.04
Epoch 10 Validation Acc: 0.36
Epoch 20 Validation Acc: 0.43
Epoch 30 Validation Acc: 0.46
Epoch 40 Validation Acc: 0.52
Epoch 49 Validation Acc: 0.54

Using LR=0.005 I get:

Epoch 0 Validation Acc: 0.11
Epoch 10 Validation Acc: 0.29
Epoch 20 Validation Acc: 0.35
Epoch 30 Validation Acc: 0.41
Epoch 40 Validation Acc: 0.47
Epoch 49 Validation Acc: 0.53

Training - GroupMultiScaleCrop(224), RandomHorizontalFlip(0.5) Validation - ResizeShortestSide(256), CentreCrop(224)

So validation is always around 0.53% and training-accuracy is around 52%. I guess to reach 71.0% acc with multi-clip, multi-crop the regular validation should be 65% + So I'm definitely behind a lot.

[chunfuchen]

Hi, sorry for the late reply.

1. We trained the Kinetics400 with 100 epochs instead of 50 epochs, sorry for the confusion.

2. You might want to check the number of videos you had in the training and validation set of the Kinetics400 dataset (https://github.com/facebookresearch/video-nonlocal-net/issues/67)

3. here is the log I retrain the model (by using the codes in this repo) with total batch size 72 under 6 GPUs setting:

   
   Namespace(alpha=2, batch_size=72, beta=4, blending_frames=3, dataset='kinetics400', dense_sampling=False, depth=50, disable_scaleup=True, dropout=0.5, epochs=100, evaluate=False, frames_per_group=1, gpu=None, groups=16, imagenet_blnet_pretrained=True, input_channels=3, input_shape=224, logdir='./', lr=0.01, lr_scheduler='cosine', lr_steps=[15, 30, 45], modality='rgb', momentum=0.9, num_classes=400, num_clips=1, num_crops=1, pretrained=False, print_freq=500, random_sampling=False, resume=None, show_model=False, start_epoch=0, weight_decay=0.0005, workers=64)

100 epochs, single clip, and single crop acc

Val : [001/100] Loss: 3.7331 Top@1: 19.4875 Top@5: 45.4624 Speed: 967.55 ms/batch Val : [010/100] Loss: 2.3200 Top@1: 45.5692 Top@5: 73.5218 Speed: 1037.19 ms/batch Val : [020/100] Loss: 2.1634 Top@1: 49.6924 Top@5: 75.4639 Speed: 951.47 ms/batch Val : [030/100] Loss: 2.0265 Top@1: 51.9752 Top@5: 78.5500 Speed: 950.88 ms/batch Val : [040/100] Loss: 1.8245 Top@1: 55.8544 Top@5: 81.2802 Speed: 953.56 ms/batch Val : [050/100] Loss: 1.8113 Top@1: 57.8118 Top@5: 81.9360 Speed: 1041.54 ms/batch Val : [060/100] Loss: 1.7302 Top@1: 59.8759 Top@5: 82.9783 Speed: 1095.85 ms/batch Val : [070/100] Loss: 1.5482 Top@1: 63.2874 Top@5: 85.2509 Speed: 1160.07 ms/batch Val : [080/100] Loss: 1.4294 Top@1: 66.3582 Top@5: 87.0304 Speed: 993.56 ms/batch Val : [090/100] Loss: 1.3322 Top@1: 69.1952 Top@5: 88.1946 Speed: 893.33 ms/batch Val : [100/100] Loss: 1.3105 Top@1: 69.8968 Top@5: 88.4641 Speed: 968.87 ms/batch

The above model with 3 crops and 3 clip testing

Val@224(224) (# crops = 3, # clips = 3): Top@1: 71.2543 Top@5: 89.3386

I also retrained one with 50 epochs, it results in about 2% lower than the one trained with 100 epochs.