Resprodcution AVAv2.2

[gurkirt]

If you feel we have help you, give us a STAR! :satisfied: Done!

Notice

There are several common situations in the reimplementation issues as below

I am interested in issue 1.

1. Reimplement a model in the model zoo using the provided configs

First of all, thank you for the amazing code base.

I am interested in Spatiotemporal detection models on AVA dataset. I followed your dataset instructions and used the provided and was able to get a test time mAP of 26.27 with the given model. I used AVAv2.2 model, specifically, slowfast_temporal_max_kinetics_pretrained_r50_8x8x1_cosine_10e_ava22_rgb. Where reported is 26.4 seems like the best mAP after epoch 8 rather than at the end of complete training. I am not sure which epoch checkpoint you released in the model zoo. Nevertheless, I think it is reasonable could because of the environment and Pytorch version differences.

However when I train, the same model, I get 25.73 as the best mAP after the 8th epoch. I don't know what to attribute that. Only difference from existing was I used 4 titanX with 24GB each so I could set 12 examples per GPU while train and training it on 4 GPU without changing anything else.

Here is my environment:

  2021-11-12 22:08:59,195 - mmaction - INFO - Environment info:
  ------------------------------------------------------------
  sys.platform: linux
  Python: 3.8.5 (default, Oct  6 2020, 10:04:29) [GCC 6.3.0]
  CUDA available: True
  GPU 0: GeForce RTX 2080 Ti
  CUDA_HOME: /cluster/apps/gcc-6.3.0/cuda-11.0.3-qdlibd2luz2fy7izfefao4c5yitxwjus
  NVCC: Build cuda_11.0_bu.TC445_37.28845127_0
  GCC: gcc (GCC) 6.3.0
  PyTorch: 1.7.1+cu110
  PyTorch compiling details: PyTorch built with:
    - GCC 7.3
    - C++ Version: 201402
    - Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
    - Intel(R) MKL-DNN v1.6.0 (Git Hash 5ef631a030a6f73131c77892041042805a06064f)
    - OpenMP 201511 (a.k.a. OpenMP 4.5)
    - NNPACK is enabled
    - CPU capability usage: AVX2
    - CUDA Runtime 11.0
    - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80
    - CuDNN 8.0.5
    - Magma 2.5.2
    - Build settings: BLAS=MKL, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DUSE_VULKAN_WRAPPER -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, 

  TorchVision: 0.8.2+cu110
  OpenCV: 4.5.1
  MMCV: 1.3.16
  MMCV Compiler: GCC 7.3
  MMCV CUDA Compiler: 11.0
  MMAction2: 0.19.0+ca01f4e
  ------------------------------------------------------------

Accidentally, I deleted the training log, but I could evaluation the log for both my model and your released one.

[kennymckormick]

We provide the best checkpoint with the best results and report the mAP of it, we are going to double-check and test this checkpoint. Besides, I'm not sure what's the reason for inferior precision in your experiments, maybe you can try to use batch size 6 with half learning rate.

[gurkirt]

Thanks for the suggestion. I think I am going to switch back to pyslowfast because of another issue about evaluation comparison.

[hukkai]

I also got a lower results (reported: 27.8 mAP, what I got: 27.1 mAP) for this config: https://github.com/open-mmlab/mmaction2/blob/master/configs/detection/acrn/slowfast_acrn_kinetics_pretrained_r50_8x8x1_cosine_10e_ava22_rgb.py

Does Pytorch version matter to the performance? Since I saw a warning:

UserWarning: The default behavior for interpolate/upsample with float scale_factor changed in 1.6.0 to align with other frameworks/libraries, and now uses scale_factor directly, instead of relying on the computed output size. If you wish to restore the old behavior, please set recompute_scale_factor=True. See the documentation of nn.Upsample for details.

[gurkirt]

To be honest, I did not try what @kennymckormick mentioned but I doubt that will actual results in anything better. I switched back to pyslowfast because of another reason, https://github.com/open-mmlab/mmaction2/issues/1268. I get better numbers in the untrimmed setting using pyslowfast. Untrimmed setting what is being shown in papers. So trimmed setting which is being used here is not useful for my case.