search

open-mmlab / mmrotate

OpenMMLab Rotated Object Detection Toolbox and Benchmark
https://mmrotate.readthedocs.io/en/latest/
Apache License 2.0
1.89k stars 560 forks source link

[Reimplementation] The FCOS detector cannot be reproduced. The mAP is much lower than the model zoo. #589

Open dmhz opened 2 years ago

dmhz commented 2 years ago

Prerequisite

Task

I'm using the official example scripts/configs for the officially supported tasks/models/datasets.

Branch

master branch https://github.com/open-mmlab/mmrotate

Environment

sys.platform: linux Python: 3.8.13 (default, Mar 28 2022, 11:38:47) [GCC 7.5.0] CUDA available: True GPU 0: NVIDIA GeForce RTX 3090 CUDA_HOME: /opt/cuda NVCC: Cuda compilation tools, release 11.8, V11.8.89 GCC: gcc (GCC) 12.2.0 PyTorch: 1.8.2+cu111 PyTorch compiling details: PyTorch built with:

TorchVision: 0.9.2+cu111 OpenCV: 4.6.0 MMCV: 1.6.0 MMCV Compiler: GCC 7.3 MMCV CUDA Compiler: 11.1 MMRotate: 0.3.3+4bbef14

Reproduces the problem - code sample

python tools/train.py configs/rotated_fcos/rotated_fcos_r50_fpn_1x_dota_le90.py python tools/train.py configs/rotated_faster_rcnn/rotated_faster_rcnn_r50_fpn_1x_dota_le90.py

Reproduces the problem - command or script

python tools/test.py configs/rotated_fcos/rotated_fcos_r50_fpn_1x_dota_le90.py work_dirs/rotated_fcos_r50_fpn_1x_dota_le90/latest.pth --format-only --eval-options submission_dir=task1

python tools/test.py configs/rotated_faster_rcnn/rotated_faster_rcnn_r50_fpn_1x_dota_le90.py work_dirs/rotated_faster_rcnn_r50_fpn_1x_dota_le90/latest.pth --format-only --eval-options submission_dir=task1

Reproduces the problem - error message

The detection accuracy of FCOS on the DOTA dataset is much lower than the model provided by model zoo.

Additional information

Thank the authors for providing such an excellent rotating target detection framework.

I downloaded the latest code of mmrotate and replicated the FCOS algorithm on the DOTA dataset with RTX3090. I used the same data split method, consistent configuration files, and failed to get a good mAP after running it many times.

The following are the results submitted on the dota test server twice: AP50: 0.7032243822038402 AP75: 0.38472094328866135 mAP: 0.3986036947128665 AP50: 0.7047541260814543 AP75: 0.3857469785305509 mAP: 0.39624439296748515

I also verified the algorithm in v0.3.1, and also got a low map: AP50: 0.7016260775289945 AP75: 0.3813169279633645 mAP: 0.39712732623330294

In addition, I run the configuration file rotated faster rcnn r50 fpn 1x dota_ Le90.py can obtain the same detection accuracy as the officially published model: AP50: 0.7362327956605763 AP75: 0.40734393918484924 mAP: 0.41351331849739337

yangxue0827 commented 2 years ago

For DOTA dataset, an accuracy gap of about 0.5% seems normal. It is recommended that you follow the results you have obtained. Our results are only for your reference.

dmhz commented 2 years ago

Thanks for the reply of the author. In my experiments, the average accuracy of FCOS is 0.8-1 (70.2-70.4) lower than Model Zoo (71.28). Is there any other additional tricks in the model provided by Model Zoo?

I want to know whether the precision of mmrotate is related to the version of mmcv or mmdet?

yangxue0827 commented 2 years ago

I will give you feedback after I train again.

dmhz commented 2 years ago

okay, thank you.

yangxue0827 commented 2 years ago

Any suggestions? @liuyanyi

liuyanyi commented 2 years ago

The result of FCOS model is unstable, i still no idea why it behaive like this, maybe norm_on_bbox and center_sampling affect the result, but that need more experiment, looking for your feedback too.

In our 1.x refactor(#543), i got a lower result too. Reported ap50 is result in 0.x. The result is not stable and may have 1% ap difference.

model Reported ap50 map ap50 ap75
Rotated FCOS 71.28 41.10 70.87 41.21
Rotated FCOS kld 71.89 40.37 71.55 38.92
Rotated FCOS hbb csl 71.76 40.22 70.79 39.26
dmhz commented 2 years ago

Thanks for your reply. I'll test dev1.x and give you feedback.

If the model provided in the initial version (v0.3.1) is the better one after many trainings? Can you tell me the FCOS (v0.3.x) approximate range of mAP during many experiments for reference.

@liuyanyi

liuyanyi commented 2 years ago

The logic in the fcos of the initial version and newer version should be the same and no major changes have been made. So I think version wont affect training.

While develop fcos in 0.3, i remember 71.28 is the best result i got.

Here is my exps during refactor 1.x, there was a 1% difference in the ap of the three experiments.

rotated_fcos run ap50 ap75 map
1 71.63 42.34 41.85
2 70.72 41.06 40.95
3 71.32 41.26 41.25

rotated_fcos_csl

run ap50 ap75 map  w
1 70.84 39.51 40.15 w0.2
2 71.15 39.64 40.38 w0.2
3 71.26 40.03 40.39 w0.2
4 70.94 40.51 40.78 w0.6
yangxue0827 commented 2 years ago

My reimplementation: 

This is your evaluation result for task 1 (VOC metrics):

mAP: 0.7121946689619715
ap of each class: plane:0.8906049957433385, baseball-diamond:0.7352599576125792, bridge:0.4853550551625228, ground-track-field:0.6272555673055425, small-vehicle:0.7847658521031989, large-vehicle:0.7335348612561778, ship:0.8618551298188438, tennis-court:0.9090909090909093, basketball-court:0.8132608032925924, storage-tank:0.8407974408264521, soccer-ball-field:0.5744272576514078, roundabout:0.614308353447493, harbor:0.6327183850671182, swimming-pool:0.6676415731071267, helicopter:0.5120438929442703
COCO style result:

AP50: 0.7121946689619715
AP75: 0.38454486600750865
mAP: 0.3960396939468972
The submitted information is :

Description: rotated_fcos_r50_fpn_1x_dota_le90

Environment:

2022-11-01 18:43:07,213 - mmrotate - INFO - Environment info:
------------------------------------------------------------
sys.platform: linux
Python: 3.7.11 (default, Jul 27 2021, 14:32:16) [GCC 7.5.0]
CUDA available: True
GPU 0: NVIDIA GeForce RTX 3090
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 11.3, V11.3.109
GCC: gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
PyTorch: 1.9.0+cu111
PyTorch compiling details: PyTorch built with:
  - GCC 7.3
  - C++ Version: 201402
  - Intel(R) oneAPI Math Kernel Library Version 2021.3-Product Build 20210617 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v2.1.2 (Git Hash 98be7e8afa711dc9b66c8ff3504129cb82013cdb)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 11.1
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86
  - CuDNN 8.0.5
  - Magma 2.5.2
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.1, CUDNN_VERSION=8.0.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.9.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, 

TorchVision: 0.10.0+cu111
OpenCV: 4.5.5
MMCV: 1.6.0
MMCV Compiler: GCC 7.5
MMCV CUDA Compiler: 11.3
MMRotate: 0.2.0+
------------------------------------------------------------

Log: 20221101_184300.log

yangxue0827 commented 2 years ago

My codebase: https://github.com/yangxue0827/h2rbox-mmrotate

dmhz commented 2 years ago

The logic in the fcos of the initial version and newer version should be the same and no major changes have been made. So I think version wont affect training.

While develop fcos in 0.3, i remember 71.28 is the best result i got.

Here is my exps during refactor 1.x, there was a 1% difference in the ap of the three experiments.

rotated_fcos

run ap50 ap75 map 1 71.63 42.34 41.85 2 70.72 41.06 40.95 Here is my exps during refactor 1.x, there was a 1% difference in the ap of the three experiments.

rotated_fcos

run ap50 ap75 map 1 71.63 42.34 41.85 2 70.72 41.06 40.95 3 71.32 41.26 41.25 rotated_fcos_cs 3 71.32 41.26 41.25 rotated_fcos_csl

run ap50 ap75 map  w 1 70.84 39.51 40.15 w0.2 2 71.15 39.64 40.38 w0.2 3 71.26 40.03 40.39 w0.2 4 70.94 40.51 40.78 w0.6

I tested the mmrotate development version, which can achieve higher results. The following are the test results I ran twice: AP50: 0.7150217002058156 AP75: 0.42193368883257576 mAP: 0.41591037131929853

AP50: 0.7120718620827703 AP75: 0.4098099929434117 mAP: 0.4078548301075865

The software version I tested is as follows: mmcv==2.0.0rc2 mmdet==3.0.0rc2 mmengine==0.2.0 mmrotate==1.0.0rc0

However, the accuracy of mmrotate (v0.3.x) is not high, which may be due to a specific software version or other reasons. I have no ability to find the differences.

dmhz commented 2 years ago

https://github.com/yangxue0827/h2rbox-mmrotate

rotated_fcos_r50_fpn_1x_dota_le90.py

mmcv-full==1.6.0 mmdet==2.25.3 mmrotate==0.2.0

This is your evaluation result for task 1 (VOC metrics):

mAP: 0.7080817910895376 ap of each class: plane:0.891049340837604, baseball-diamond:0.7265259086435176, bridge:0.48298530261650086, ground-track-field:0.5933720824738311, small-vehicle:0.790293514900079, large-vehicle:0.7293038272089639, ship:0.8593364303461166, tennis-court:0.9089545113551116, basketball-court:0.8169677011977725, storage-tank:0.842900847286389, soccer-ball-field:0.5647016098941234, roundabout:0.6549169070790201, harbor:0.6268073214237, swimming-pool:0.6857934950110997, helicopter:0.4473180660692365 COCO style result:

AP50: 0.7080817910895376 AP75: 0.3762085109574612 mAP: 0.39150821002761227

Seems to be a reasonable result.

20221103_003210.log

hu-wei-carrzy commented 2 years ago

Excuse me, may I ask, i am novice.I want to reproduce oriented RCNN，i used ss_trainval.json,ss_test.json cut picture，and used oriented_rcnn_r50_fpn_1x_dota_le90-6d2b2ce0.pth，but i can get the good result，if you have time to see my question ,i will tank you very much，i also ask at https://github.com/open-mmlab/mmrotate/issues/594，my E-mail is 2780222819@qq.com,

dmhz commented 2 years ago

I have never seen such low accuracy in my test, which may be caused by some settings or other reasons. What is your system environment and software version. You can try to use the official configuration file without modification for training. email: dmhz@live.cn @hu-wei-carrzy

zhangleigood commented 1 year ago

My codebase: https://github.com/yangxue0827/h2rbox-mmrotate

My reimplementation:

This is your evaluation result for task 1 (VOC metrics):

mAP: 0.7121946689619715
ap of each class: plane:0.8906049957433385, baseball-diamond:0.7352599576125792, bridge:0.4853550551625228, ground-track-field:0.6272555673055425, small-vehicle:0.7847658521031989, large-vehicle:0.7335348612561778, ship:0.8618551298188438, tennis-court:0.9090909090909093, basketball-court:0.8132608032925924, storage-tank:0.8407974408264521, soccer-ball-field:0.5744272576514078, roundabout:0.614308353447493, harbor:0.6327183850671182, swimming-pool:0.6676415731071267, helicopter:0.5120438929442703
COCO style result:

AP50: 0.7121946689619715
AP75: 0.38454486600750865
mAP: 0.3960396939468972
The submitted information is :

Description: rotated_fcos_r50_fpn_1x_dota_le90

Environment:

2022-11-01 18:43:07,213 - mmrotate - INFO - Environment info:
------------------------------------------------------------
sys.platform: linux
Python: 3.7.11 (default, Jul 27 2021, 14:32:16) [GCC 7.5.0]
CUDA available: True
GPU 0: NVIDIA GeForce RTX 3090
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 11.3, V11.3.109
GCC: gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
PyTorch: 1.9.0+cu111
PyTorch compiling details: PyTorch built with:
  - GCC 7.3
  - C++ Version: 201402
  - Intel(R) oneAPI Math Kernel Library Version 2021.3-Product Build 20210617 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v2.1.2 (Git Hash 98be7e8afa711dc9b66c8ff3504129cb82013cdb)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 11.1
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86
  - CuDNN 8.0.5
  - Magma 2.5.2
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.1, CUDNN_VERSION=8.0.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.9.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, 

TorchVision: 0.10.0+cu111
OpenCV: 4.5.5
MMCV: 1.6.0
MMCV Compiler: GCC 7.5
MMCV CUDA Compiler: 11.3
MMRotate: 0.2.0+
------------------------------------------------------------

Log: 20221101_184300.log

请问您复现的时候是使用ms_trainval还是ss_trainval裁剪DOTA数据集的，里面的参数是怎么设置的？感谢您的工作