Configs on Charades

[lhyfst]

Dear author, Could you please release the configs for training and testing on Charades? Even an example is helpful. I can train models on Charades, but the map is very low like 0.057181. I have not figured out the problem. If you could provide the configs on Charades, that will be very helpful. Thanks!

[ecr23xx]

Same question here. I try to load the video-long-feature-bank's pertained model on Charades, but the validation mAP is still 0.057181. It seems something is wrong with the evaluation metrics?

[lhyfst]

@ecr23xx you need to set ENSEMBLE_METHOD as max instead of sum. However, I still cannot get a reasonable result like what the paper shows after using max as ENSEMBLE_METHOD. My results are around 0.1 but the results in the paper are around 0.4.

[haooooooqi]

Hi, Thanks for playing with PySF! We will release the config soon.

[ecr23xx]

Hi @takatosp1

The download link for frame lists of SSv2 and Charades is broken. Can you fix it?

[jiaming-zhou]

I got the same result(map=0.05*) on Charades, can you release the configs on Charades?

[GoodMan0]

@qianhanxuan @ecr23xx Hi, do you have the frame list for SSv2 and Charades, can you share with me? Thanks!

[ecr23xx]

@qianhanxuan You need to freeze all BN parameters in training. And in inference, you need to set ENSEMBLE_METHOD as max instead of sum. I've successfully reproduced the result on Charades with I3D.

[ecr23xx]

@GoodMan0 You can download Charades' frame lists HERE. I do not have frame lists for SSv2 but it should be easy to generate using the official train/test file.

[GoodMan0]

@ecr23xx Thanks!!!

[jiaming-zhou]

@qianhanxuan You need to freeze all BN parameters in training. And in inference, you need to set ENSEMBLE_METHOD as max instead of sum. I've successfully reproduced the result on Charades with I3D.

@ecr23xx Thanks! I have sloved this problem. But I got map=30.*% on Charades when I using the fine-tuned charades_R101-I3D-NL_baseline_model released in Long-term feature banks Repo, which should be 40.4% however. Have you tried this before? Anyway, thank you very much!

[JunweiLiang]

@ecr23xx Did you reproduce Charades for the LFB paper or Slowfast paper?

[ecr23xx]

@JunweiLiang My results on Charades:

Slow only: mAP = 36.2 I3D: mAP = 35.46 SlowFast: mAP = 39.31

Results in the paper is higher than that. Possible reasons could be 1) sampling rate and number of frames; 2) I freeze all BN layers, but it seems the paper does not. Maybe we should wait for @takatosp1 to release official configs.

[JunweiLiang]

@ecr23xx Thanks. I only got ~0.36 with SlowFast with 16x8 frames and LFB's training schedule (24k iterations), and ~0.38 with 32x4 frames (I also freeze all BN parameters). What training schedule and frames did you use to get 0.39?

[ecr23xx]

@JunweiLiang You can refer to this commit

[GoodMan0]

@ecr23xx Did you get the results by using this commit you said before? I follow this configs only get the results: slowfast-101(k600): 39.66, slowfast-50(k400): 38.94, results in the paper is higher than mine. So I want to know your other training configs, like Base_LR , batch_size and NUM_GPUs, and I set Base_LR=0.0375 , batch_size=8 and NUM_GPUs=1, can you share your modified configs? Thanks!

[ecr23xx]

@GoodMan0 You should linearly scale the BASE_LR if you scale NUM_GPUs and BATCH_SIZE. Related configs in BN should also modified according to the BATCH_SIZE you set. I completely follow the configs in that commit (NUM_GPUS = 8, BATCH_SIZE = 16, BASE_LR = 0.0375).

[Kewenjing1020]

Why the released model(38.9%) doesn't achieve the accuracy reported in the paper(42.1%)? With the released config, I get 39.6%, but still 2.5% lower than the accuracy in the paper. Does anyone know why's the gap?

[ZilinGao]

Why the released model(38.9%) doesn't achieve the accuracy reported in the paper(42.1%)? With the released config, I get 39.6%, but still 2.5% lower than the accuracy in the paper. Does anyone know why's the gap?

@Kewenjing1020 hello~ You said you tried to reproduce the performance of Slowfast R-101 on Charades, obtaining mAP with 39.6%. I also want to reproduce this result, but it seems that the pre-trained model on K-400 is not avaliable now(the link in MODEL_ZOO.md is broken). So how did you reproduce the experiment? Is there other pre-trained model source?

Thanks a lot!

[RishiDesai]

@JunweiLiang My results on Charades:

Slow only: mAP = 36.2 I3D: mAP = 35.46 SlowFast: mAP = 39.31

Results in the paper is higher than that. Possible reasons could be 1) sampling rate and number of frames; 2) I freeze all BN layers, but it seems the paper does not. Maybe we should wait for @takatosp1 to release official configs.

@ecr23xx I did the Charades experiment (Standard SlowFast 16x8 with Kinetics-400) using the default config file from the repo with these changes: BatchSize=32, NumGPUs=8. I got mAP=36, whereas the ModelZoo shows mAP=38.9. I'm wondering why Facebook's numbers are 2.9 above mine. If you don't mind, could you please share your config file with me? Thanks!

[ecr23xx]

@RishiDesai I use the default config file from the repo (16x8), except for that I freeze all BN layers instead of using Sync BN. I notice that you increase the batch size to 32 (the default is 16). Maybe this modification causes the performance drop.