The test cannot calculate ap on its own dataset

yangzishy commented 1 year ago

Prerequisite

[X] I have searched Issues and Discussions but cannot get the expected help.
[X] I have read the FAQ documentation but cannot get the expected help.
[X] The bug has not been fixed in the latest version (master) or latest version (1.x).

Task

I'm using the official example scripts/configs for the officially supported tasks/models/datasets.

Branch

master branch https://github.com/open-mmlab/mmrotate

Environment

sys.platform: win32 Python: 3.8.13 (default, Mar 28 2022, 06:59:08) [MSC v.1916 64 bit (AMD64)] CUDA available: True GPU 0: NVIDIA GeForce GTX 1050 Ti CUDA_HOME: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2 NVCC: Cuda compilation tools, release 10.2, V10.2.8 MSVC: 用于 x64 的 Microsoft (R) C/C++ 优化编译器 19.29.30146 版 GCC: n/a PyTorch: 1.8.0 PyTorch compiling details: PyTorch built with:

C++ Version: 199711
MSVC 192829337
Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191125 for Intel(R) 64 architecture applications
Intel(R) MKL-DNN v1.7.0 (Git Hash 7aed236906b1f7a05c0917e5257a1af05e9ff683)
OpenMP 2019
CPU capability usage: AVX2
CUDA Runtime 10.2
NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_37,code=compute_37
CuDNN 7.6.5
Magma 2.5.4
Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=10.2, CUDNN_VERSION=7.6.5, CXX_COMPILER=C:/cb/pytorch_1000000000000/work/tmp_bin/sccache-cl.exe, CXX_FLAGS=/DWIN32 /D_WINDOWS /GR /EHsc /w /bigobj -DUSE_PTHREADPOOL -openmp:experimental -DNDEBUG -DUSE_FBGEMM -DUSE_XNNPACK, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.8.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=OFF, USE_OPENMP=ON,

TorchVision: 0.9.0 OpenCV: 4.6.0 MMCV: 1.6.0 MMCV Compiler: MSVC 192930137 MMCV CUDA Compiler: 10.2 MMRotate: 0.3.2+unknown

Reproduces the problem - code sample

(mmrotate) PS F:\postgraduatework\mmrotate-main\mmrotate-main> python tools/test.py --config run_model/roi_trans_1/roi_trans_r50_fpn_1x_dota_le90.py --checkpoint run_model/roi_trans_1/latest.pth --eval mAP --show-dir work_dirs/roi_trans_1/result F:\SoftWare\Anaconda\envs\mmrotate\lib\site-packages\mmdet\models\dense_heads\anchor_head.py:116: UserWarning: DeprecationWarning: num_anchors is deprecated, for consistency or also use num_base_priors instead warnings.warn('DeprecationWarning: num_anchors is deprecated, ' load checkpoint from local path: run_model/roi_trans_1/latest.pth [ ] 0/30, elapsed: 0s, ETA:F:\SoftWare\Anaconda\envs\mmrotate\lib\site-packages\mmdet\models\dense_heads\anchor_head.py:123: UserWarning: DeprecationWarning: anchor_generator is deprecated, please use "prior_generator" instead warnings.warn('DeprecationWarning: anchor_generator is deprecated, ' [>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 30/30, 0.9 task/s, elapsed: 33s, ETA: 0sTraceback (most recent call last): File "tools/test.py", line 257, in main() File "tools/test.py", line 249, in main metric = dataset.evaluate(outputs, **eval_kwargs) File "f:\postgraduatework\mmrotate-main\mmrotate-main\mmrotate\datasets\dota.py", line 205, in evaluate meanap, = eval_rbbox_map( File "f:\postgraduatework\mmrotate-main\mmrotate-main\mmrotate\core\evaluation\eval_map.py", line 177, in eval_rbbox_map cls_dets, cls_gts, cls_gts_ignore = get_cls_results( File "f:\postgraduatework\mmrotate-main\mmrotate-main\mmrotate\core\evaluation\eval_map.py", line 114, in get_cls_results cls_gts.append(ann['bboxes'][gt_inds, :]) TypeError: list indices must be integers or slices, not tuple

Reproduces the problem - command or script

python tools/test.py --config run_model/roi_trans_1/roi_trans_r50_fpn_1x_dota_le90.py --checkpoint run_model/roi_trans_1/latest.pth --eval mAP --show-dir work_dirs/roi_trans_1/result

Reproduces the problem - error message

TypeError: list indices must be integers or slices, not tuple

Additional information

Test and calculate the ap on your own dota data set.

Make your own dataset and make it into Dota format.

zytx121 commented 1 year ago

Hi @Yangzishy! This error is caused by the loading error of your dataset. Please check whether your dataset annotation format is consistent with the DOTA.

yangzishy commented 1 year ago

Hi @Yangzishy! This error is caused by the loading error of your dataset. Please check whether your dataset annotation format is consistent with the DOTA Hi @zytx121！ The format of my test data set is in Dota format, and it can detect the results of the test set, but it cannot calculate ap

yangzishy commented 1 year ago

Hi @Yangzishy! This error is caused by the loading error of your dataset. Please check whether your dataset annotation format is consistent with the DOTA.

Hi @zytx121 ! Please take a look at my reply. I need your help. Thank you

zytx121 commented 1 year ago

Hi @Yangzishy @jf0606, the maintenance of the old version has stopped. Welcome to the new version: https://github.com/open-mmlab/mmrotate/tree/1.x

jf0606 commented 1 year ago

Hi @Yangzishy! This error is caused by the loading error of your dataset. Please check whether your dataset annotation format is consistent with the DOTA.

Hi @zytx121 ! Please take a look at my reply. I need your help. Thank you

you can check you code in dotav1.py. I found it worked when i modify test=dict{ann_file=your path} test=dict( type=dataset_type, ann_file=data_root + 'test/annfiles/', img_prefix=data_root + 'test/images/', pipeline=test_pipeline))

yangzishy commented 1 year ago

Hi @Yangzishy! This error is caused by the loading error of your dataset. Please check whether your dataset annotation format is consistent with the DOTA.

Hi @zytx121 ! Please take a look at my reply. I need your help. Thank you

you can check you code in dotav1.py. I found it worked when i modify test=dict{ann_file=your path} test=dict( type=dataset_type, ann_file=data_root + 'test/annfiles/', img_prefix=data_root + 'test/images/', pipeline=test_pipeline)) Hi @jf0606 ！Have you solved it？

yangzishy commented 1 year ago

Hi @Yangzishy @jf0606, the maintenance of the old version has stopped. Welcome to the new version: https://github.com/open-mmlab/mmrotate/tree/1.x

Hi @zytx121 ！Thank you for your answer. I'll try the new version

yangzishy commented 1 year ago

Hi @Yangzishy! This error is caused by the loading error of your dataset. Please check whether your dataset annotation format is consistent with the DOTA.

Hi @zytx121 ! Please take a look at my reply. I need your help. Thank you

you can check you code in dotav1.py. I found it worked when i modify test=dict{ann_file=your path} test=dict( type=dataset_type, ann_file=data_root + 'test/annfiles/', img_prefix=data_root + 'test/images/', pipeline=test_pipeline))

Hi @jf0606 ！I have the same problem you mentioned, but it doesn't solve the problem. The same error still occurs

Manjuphoenix commented 1 year ago

Same issue, when i try to find mAP on custom dataset but with the config file same as DOTA (with just classes being changed)

as @jf0606 pointed:

Hi @Yangzishy! This error is caused by the loading error of your dataset. Please check whether your dataset annotation format is consistent with the DOTA.

Hi @zytx121 ! Please take a look at my reply. I need your help. Thank you

you can check you code in dotav1.py. I found it worked when i modify test=dict{ann_file=your path} test=dict( type=dataset_type, ann_file=data_root + 'test/annfiles/', img_prefix=data_root + 'test/images/', pipeline=test_pipeline))

I tried changing the images to annfiles directory but this gives a new error as below:

warnings.warn('DeprecationWarning: anchor_generator is deprecated, ' [>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 66/66, 2.6 task/s, elapsed: 25s, ETA: 0sTraceback (most recent call last): File "/home/user/manjunath/mmrotate/tools/test.py", line 257, in main() File "/home/user/manjunat/mmrotate/tools/test.py", line 249, in main metric = dataset.evaluate(outputs, **eval_kwargs) File "/home/user/manjunath/mmrotate/mmrotate/datasets/dota.py", line 205, in evaluate meanap, = eval_rbbox_map( File "/home/user/manjunath/mmrotate/mmrotate/core/evaluation/eval_map.py", line 177, in eval_rbbox_map cls_dets, cls_gts, cls_gts_ignore = get_cls_results( File "/home/user/manjunath/mmrotate/mmrotate/core/evaluation/eval_map.py", line 114, in get_cls_results cls_gts.append(ann['bboxes'][gt_inds, :]) TypeError: list indices must be integers or slices, not tuple

yangzishy commented 1 year ago

Same issue, when i try to find mAP on custom dataset but with the config file same as DOTA (with just classes being changed)

as @jf0606 pointed:

Hi @Yangzishy! This error is caused by the loading error of your dataset. Please check whether your dataset annotation format is consistent with the DOTA.

Hi @zytx121 ! Please take a look at my reply. I need your help. Thank you

you can check you code in dotav1.py. I found it worked when i modify test=dict{ann_file=your path} test=dict( type=dataset_type, ann_file=data_root + 'test/annfiles/', img_prefix=data_root + 'test/images/', pipeline=test_pipeline))

I tried changing the images to annfiles directory but this gives a new error as below:

warnings.warn('DeprecationWarning: anchor_generator is deprecated, ' [>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 66/66, 2.6 task/s, elapsed: 25s, ETA: 0sTraceback (most recent call last): File "/home/user/manjunath/mmrotate/tools/test.py", line 257, in main() File "/home/user/manjunat/mmrotate/tools/test.py", line 249, in main metric = dataset.evaluate(outputs, **eval_kwargs) File "/home/user/manjunath/mmrotate/mmrotate/datasets/dota.py", line 205, in evaluate meanap, = eval_rbbox_map( File "/home/user/manjunath/mmrotate/mmrotate/core/evaluation/eval_map.py", line 177, in eval_rbbox_map cls_dets, cls_gts, cls_gts_ignore = get_cls_results( File "/home/user/manjunath/mmrotate/mmrotate/core/evaluation/eval_map.py", line 114, in get_cls_results cls_gts.append(ann['bboxes'][gt_inds, :]) TypeError: list indices must be integers or slices, not tuple

Hi @jf0606 !Thank you for your response. I use mmrotate-1. x to test my dataset model with a map value

thetashall commented 1 year ago

其实就是路径写错了，源代码dotav1.py“ test=dict( type=dataset_type, ann_file=data_root + 'test/images/', img_prefix=data_root + 'test/images/', pipeline=test_pipeline))” 把 ann_file=data_root + 'test/images/',改为 ann_file=data_root + 'test/annfiles/'

ZJM-2000 commented 11 months ago

DOTA's test set has only images, no tags, that is, no annfiles, so eval map cannot be used on the test set. According to the document please submit assessment https://github.com/open-mmlab/mmrotate/blob/main/docs/en/get_started.md to generate compressed files online

open-mmlab / mmrotate