[Reimplementation] KeyError when running evaluation

[chrimidt]

Prerequisite

• [X] I have searched Issues and Discussions but cannot get the expected help.
• [X] I have read the FAQ documentation but cannot get the expected help.
• [X] The bug has not been fixed in the latest version (master) or latest version (3.x).

💬 Describe the reimplementation questions

Hi, I'm trying to evaluate the model but I'm running into a KeyErrror with "start_coord" when running

python ./tools/test.py "configs\cfinet\faster_rcnn_r50_fpn_cfinet_1x.py" "work_dirs\epoch_12.pth" --eval bbox

using the pre-trained weights.

This is the output when running:

(my_env) C:\Users\min_s\cvproject\CFINet>python ./tools/test.py "C:\Users\min_s\cvproject\CFINet\configs\cfinet\faster_rcnn_r50_fpn_cfinet_1x.py" "C:\Users\min_s\cvproject\CFINet\work_dirs\epoch_12.pth" --eval bbox c:\users\min_s\cvproject\cfinet\mmdet\utils\setup_env.py:38: UserWarning: Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. warnings.warn( c:\users\min_s\cvproject\cfinet\mmdet\utils\setup_env.py:48: UserWarning: Setting MKL_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. warnings.warn( loading annotations into memory... Done (t=0.45s) creating index... index created! loading annotations into memory... Done (t=0.28s) creating index... index created! c:\users\min_s\cvproject\cfinet\mmdet\models\losses\iou_loss.py:266: UserWarning: DeprecationWarning: Setting "linear=True" in IOULoss is deprecated, please use "mode=linear" instead. warnings.warn('DeprecationWarning: Setting "linear=True" in ' c:\users\min_s\cvproject\cfinet\mmdet\models\dense_heads\anchor_head.py:116: UserWarning: DeprecationWarning: num_anchors is deprecated, for consistency or also use num_base_priors instead warnings.warn('DeprecationWarning: num_anchors is deprecated, ' c:\users\min_s\cvproject\cfinet\mmdet\models\dense_heads\anchor_head.py:123: UserWarning: DeprecationWarning: anchor_generator is deprecated, please use "prior_generator" instead warnings.warn('DeprecationWarning: anchor_generator is deprecated, ' load checkpoint from local path: C:\Users\min_s\cvproject\CFINet\work_dirs\epoch_12.pth The model and loaded state dict do not match exactly

unexpected key in source state_dict: rpn_head.rpn_conv.weight, rpn_head.rpn_conv.bias, rpn_head.rpn_cls.weight, rpn_head.rpn_cls.bias, rpn_head.rpn_reg.weight, rpn_head.rpn_reg.bias

missing keys in source state_dict: rpn_head.stages.0.rpn_conv.conv.weight, rpn_head.stages.0.rpn_conv.conv.bias, rpn_head.stages.0.rpn_reg.weight, rpn_head.stages.0.rpn_reg.bias, rpn_head.stages.1.rpn_conv.conv.weight, rpn_head.stages.1.rpn_cls.weight, rpn_head.stages.1.rpn_cls.bias, rpn_head.stages.1.rpn_reg.weight, rpn_head.stages.1.rpn_reg.bias

[ ] 0/7428, elapsed: 0s, ETA:C:\Users\min_s\cvproject\my_env\lib\site-packages\torch\functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ..\aten\src\ATen\native\TensorShape.cpp:2157.) return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined] [>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 7428/7428, 10.6 task/s, elapsed: 699s, ETA: 0s

Merge detected results of patch for whole image evaluating... Traceback (most recent call last): File "./tools/test.py", line 287, in main() File "./tools/test.py", line 279, in main metric = dataset.evaluate(outputs, **eval_kwargs) File "c:\users\min_s\cvproject\cfinet\mmdet\datasets\sodad.py", line 506, in evaluate merged_results = self.merge_dets( File "c:\users\min_s\cvproject\cfinet\mmdet\datasets\sodad.py", line 426, in merge_dets x_start, y_start = data_info['start_coord'] KeyError: 'start_coord'

Any clue to what I'm missing here? Thank you!

Environment

Same as described in the repository.

sys.platform: win32 Python: 3.8.10 (tags/v3.8.10:3d8993a, May 3 2021, 11:48:03) [MSC v.1928 64 bit (AMD64)] CUDA available: True GPU 0: NVIDIA GeForce RTX 3070 Ti Laptop GPU CUDA_HOME: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.3 NVCC: Cuda compilation tools, release 11.3, V11.3.58 MSVC: Microsoft (R) C/C++ Optimizing Compiler Version 19.40.33812 for x64 GCC: n/a PyTorch: 1.10.0+cu113 PyTorch compiling details: PyTorch built with:

• C++ Version: 199711
• MSVC 192829337
• Intel(R) Math Kernel Library Version 2020.0.2 Product Build 20200624 for Intel(R) 64 architecture applications
• Intel(R) MKL-DNN v2.2.3 (Git Hash 7336ca9f055cf1bfa13efb658fe15dc9b41f0740)
• OpenMP 2019
• LAPACK is enabled (usually provided by MKL)
• CPU capability usage: AVX2
• CUDA Runtime 11.3
• NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=compute_37
• CuDNN 8.2
• Magma 2.5.4
• Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.3, CUDNN_VERSION=8.2.0, CXX_COMPILER=C:/w/b/windows/tmp_bin/sccache-cl.exe, CXX_FLAGS=/DWIN32 /D_WINDOWS /GR /EHsc /w /bigobj -DUSE_PTHREADPOOL -openmp:experimental -IC:/w/b/windows/mkl/include -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_FBGEMM -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.10.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=OFF, USE_OPENMP=ON,

TorchVision: 0.11.0+cu113 OpenCV: 4.10.0 MMCV: 1.5.0 MMCV Compiler: MSVC 192930140 MMCV CUDA Compiler: 11.3 MMDetection: 2.26.0+2167eeb

Expected results

No response

Additional information

In the configs/base/datasts/sodad.py

data = dict( samples_per_gpu=2, workers_per_gpu=2, train=dict( type=dataset_type, ann_file=data_root + 'divData/Annotations/train.json', img_prefix=data_root + 'divData/Images/', pipeline=train_pipeline, ori_ann_file=data_root + 'rawData/Annotations/train.json' ), val=dict( type=dataset_type, ann_file=data_root + 'rawData/Annotations/val.json', img_prefix=data_root + 'rawData/Images/', pipeline=test_pipeline, ori_ann_file=data_root + 'rawData/Annotations/val_wo_ignore.json' ), test=dict( type=dataset_type, ann_file=data_root + 'rawData/Annotations/test.json', img_prefix=data_root + 'rawData/Images/', pipeline=test_pipeline, ori_ann_file=data_root + 'rawData/Annotations/test_wo_ignore.json' ))

[shaunyuan22]

the ann_file and img_prefix in val and test part should be the divided data instead of raw data

[chrimidt]

Thank you for your reply, that worked! I had mixed up the data somehow. However, I'm suspecting I'm not evaluating on the correct data, as when the evaluation is done, these are the results:

(my_env) C:\Users\min_s\cvproject\CFINet>python ./tools/test.py "C:\Users\min_s\cvproject\CFINet\configs\cfinet\faster_rcnn_r50_fpn_cfinet_1x.py" "C:\Users\min_s\cvproject\CFINet\configs\cfinet\faster_rcnn_r50_fpn_fi_roi_head_0822\epoch_12.pth" --work-dir C:\Users\min_s\cvproject\CFINet\work_dirs\Test --eval bbox c:\users\min_s\cvproject\cfinet\mmdet\utils\setup_env.py:38: UserWarning: Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. warnings.warn( c:\users\min_s\cvproject\cfinet\mmdet\utils\setup_env.py:48: UserWarning: Setting MKL_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. warnings.warn( loading annotations into memory... Done (t=1.72s) creating index... index created! loading annotations into memory... Done (t=0.44s) creating index... index created! c:\users\min_s\cvproject\cfinet\mmdet\models\losses\iou_loss.py:266: UserWarning: DeprecationWarning: Setting "linear=True" in IOULoss is deprecated, please use "mode=linear" instead. warnings.warn('DeprecationWarning: Setting "linear=True" in ' c:\users\min_s\cvproject\cfinet\mmdet\models\dense_heads\anchor_head.py:116: UserWarning: DeprecationWarning: num_anchors is deprecated, for consistency or also use num_base_priors instead warnings.warn('DeprecationWarning: num_anchors is deprecated, ' c:\users\min_s\cvproject\cfinet\mmdet\models\dense_heads\anchor_head.py:123: UserWarning: DeprecationWarning: anchor_generator is deprecated, please use "prior_generator" instead warnings.warn('DeprecationWarning: anchor_generator is deprecated, ' load checkpoint from local path: C:\Users\min_s\cvproject\CFINet\configs\cfinet\faster_rcnn_r50_fpn_fi_roi_head_0822\epoch_12.pth The model and loaded state dict do not match exactly

unexpected key in source state_dict: rpn_head.rpn_conv.weight, rpn_head.rpn_conv.bias, rpn_head.rpn_cls.weight, rpn_head.rpn_cls.bias, rpn_head.rpn_reg.weight, rpn_head.rpn_reg.bias

missing keys in source state_dict: rpn_head.stages.0.rpn_conv.conv.weight, rpn_head.stages.0.rpn_conv.conv.bias, rpn_head.stages.0.rpn_reg.weight, rpn_head.stages.0.rpn_reg.bias, rpn_head.stages.1.rpn_conv.conv.weight, rpn_head.stages.1.rpn_cls.weight, rpn_head.stages.1.rpn_cls.bias, rpn_head.stages.1.rpn_reg.weight, rpn_head.stages.1.rpn_reg.bias

[ ] 0/161579, elapsed: 0s, ETA:C:\Users\min_s\cvproject\my_env\lib\site-packages\torch\functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ..\aten\src\ATen\native\TensorShape.cpp:2157.) return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined] [>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 161579/161579, 10.2 task/s, elapsed: 15886s, ETA: 0s

Merge detected results of patch for whole image evaluating... Merge results completed, it costs 21.4 seconds.

Evaluating bbox... Loading and preparing results... DONE (t=0.01s) creating index... index created! Running per image evaluation... Evaluate annotation type bbox DONE (t=3.94s). Accumulating evaluation results... DONE (t=0.66s). Average Precision (AP) @[ IoU=0.50:0.95 | area= Small | maxDets=100 ] = 0.001 Average Precision (AP) @[ IoU=0.50 | area= Small | maxDets=100 ] = 0.003 Average Precision (AP) @[ IoU=0.75 | area= Small | maxDets=100 ] = 0.001 Average Precision (AP) @[ IoU=0.50:0.95 | area= eS | maxDets=100 ] = 0.001 Average Precision (AP) @[ IoU=0.50:0.95 | area= rS | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= gS | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area=Normal | maxDets=100 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= Small | maxDets=100 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= Small | maxDets=300 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= Small | maxDets=1000 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= eS | maxDets=1000 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= rS | maxDets=1000 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= gS | maxDets=1000 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area=Normal | maxDets=1000 ] = 0.000

+---------------+-------+----------------+-------+--------------+-------+ | category | AP | category | AP | category | AP | +---------------+-------+----------------+-------+--------------+-------+ | people | 0.000 | rider | 0.000 | bicycle | 0.000 | | motor | 0.000 | vehicle | 0.003 | traffic-sign | 0.001 | | traffic-light | 0.002 | traffic-camera | 0.000 | warning-cone | 0.000 | +---------------+-------+----------------+-------+--------------+-------+ {'bbox_AP': 0.001, 'bbox_AP_50': 0.003, 'bbox_AP_75': 0.001, 'bbox_AP_eS': 0.001, 'bbox_AP_rS': 0.0, 'bbox_AP_gS': 0.0, 'bbox_AP_Normal': 0.0, 'bbox_mAP_copypaste': '0.001 0.003 0.001 0.001 0.000 0.000 0.000 '}

The config file:

data = dict( samples_per_gpu=2, workers_per_gpu=2, train=dict( type=dataset_type, ann_file=data_root + 'divData/Annotations/train.json', img_prefix=data_root + 'divData/Images/train', pipeline=train_pipeline, ori_ann_file=data_root + 'rawData/Annotations/train.json' ), val=dict( type=dataset_type, ann_file=data_root + 'divData/Annotations/val.json', img_prefix=data_root + 'divData/Images/val', pipeline=test_pipeline, ori_ann_file=data_root + 'rawData/Annotations/val_wo_ignore.json' ), test=dict( type=dataset_type, ann_file=data_root + 'divData/Annotations/test.json', img_prefix=data_root + 'divData/Images/test', pipeline=test_pipeline, ori_ann_file=data_root + 'rawData/Annotations/test_wo_ignore.json' ))

Do you have any idea what could be wrong? I don't suspect the results are meant to be 0...

I'm hoping for your reply.

[shaunyuan22]

it seems that the trianing fails. what are the batch size and learning rate? the training log could be helpful to figure out this problem