After one round of training, during the Running per image evaluation..., a MemoryError error occurs.

Prerequisite

[X] I have searched Issues and Discussions but cannot get the expected help.
[X] I have read the FAQ documentation but cannot get the expected help.
[X] The bug has not been fixed in the latest version (master) or latest version (1.x).

Task

I'm using the official example scripts/configs for the officially supported tasks/models/datasets.

Branch

master branch https://github.com/open-mmlab/mmrotate

Environment

sys.platform: win32 Python: 3.8.17 (default, Jul 5 2023, 20:44:21) [MSC v.1916 64 bit (AMD64)] CUDA available: True GPU 0: NVIDIA GeForce RTX 2060 SUPER CUDA_HOME: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.3 NVCC: Cuda compilation tools, release 11.3, V11.3.58 MSVC: 用于 x64 的 Microsoft (R) C/C++ 优化编译器 19.16.27049 版 GCC: n/a PyTorch: 1.12.1 PyTorch compiling details: PyTorch built with:

C++ Version: 199711
MSVC 192829337
Intel(R) Math Kernel Library Version 2020.0.2 Product Build 20200624 for Intel(R) 64 architecture applications
Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815)
OpenMP 2019
LAPACK is enabled (usually provided by MKL)
CPU capability usage: AVX2
CUDA Runtime 11.3
NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm70;-gencode;arch=compute 75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=compute_37
CuDNN 8.3.2 (built against CUDA 11.5)
Magma 2.5.4
Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.3, CUDNN_VERSION=8.3.2, CXX_COMPILER=C:/cb/pytorch_1000000000000/work/tmp_bin/sccache-cl.exe, CXX_FLAGS=/DWIN32 /D_WINDOWS /GR /EHsc /w /bigobj -DUSE_PTHREADPOO L -openmp:experimental -IC:/cb/pytorch_1000000000000/work/mkl/include -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_FBGEMM -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO, LAPACK_INFO=mkl, PERF_WITH_AVX =1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.12.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=OFF, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=OFF, USE_OPENMP=ON, USE _ROCM=OFF,

TorchVision: 0.13.1 OpenCV: 4.8.0 MMCV: 1.5.3 MMCV Compiler: MSVC 191627049 MMCV CUDA Compiler: 11.3 MMRotate: 0.3.3+04da23d

Reproduces the problem - code sample

python tools/train.py configs/sodaa-benchmarks/rotated_faster_rcnn_r50_fpn_1x.py

Reproduces the problem - command or script

python tools/train.py configs/sodaa-benchmarks/rotated_faster_rcnn_r50_fpn_1x.py

Reproduces the problem - error message

Running per image evaluation... Traceback (most recent call last): File "tools/train.py", line 192, in main() File "tools/train.py", line 181, in main train_detector( File "f:\codes\soda-mmrotate\mmrotate\apis\train.py", line 141, in train_detector runner.run(data_loaders, cfg.workflow) File "F:\VirtualEnvironment\SODA_A\lib\site-packages\mmcv\runner\epoch_based_runner.py", line 130, in run epoch_runner(data_loaders[i], **kwargs) File "F:\VirtualEnvironment\SODA_A\lib\site-packages\mmcv\runner\epoch_based_runner.py", line 56, in train self.call_hook('after_train_epoch') File "F:\VirtualEnvironment\SODA_A\lib\site-packages\mmcv\runner\base_runner.py", line 309, in call_hook getattr(hook, fn_name)(self) File "F:\VirtualEnvironment\SODA_A\lib\site-packages\mmcv\runner\hooks\evaluation.py", line 267, in after_train_epoch self._do_evaluate(runner) File "F:\VirtualEnvironment\SODA_A\lib\site-packages\mmdet\core\evaluation\eval_hooks.py", line 63, in _do_evaluate key_score = self.evaluate(runner, results) File "F:\VirtualEnvironment\SODA_A\lib\site-packages\mmcv\runner\hooks\evaluation.py", line 363, in evaluate eval_res = self.dataloader.dataset.evaluate( File "f:\codes\soda-mmrotate\mmrotate\datasets\sodaa.py", line 447, in evaluate SODAAEval.evaluate() File "f:\codes\soda-mmrotate\mmrotate\datasets\sodaa_eval\sodaa_eval.py", line 199, in evaluate contents = pool.map(evaluateImgFunc, File "F:\VirtualEnvironment\SODA_A\lib\multiprocessing\pool.py", line 364, in map return self._map_async(func, iterable, mapstar, chunksize).get() File "F:\VirtualEnvironment\SODA_A\lib\multiprocessing\pool.py", line 771, in get raise self._value File "F:\VirtualEnvironment\SODA_A\lib\multiprocessing\pool.py", line 537, in _handle_tasks put(task) File "F:\VirtualEnvironment\SODA_A\lib\multiprocessing\connection.py", line 206, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "F:\VirtualEnvironment\SODA_A\lib\multiprocessing\reduction.py", line 51, in dumps cls(buf, protocol).dump(obj) MemoryError

Additional information

No response

shaunyuan22 / SODA-mmrotate