[Bug]

[pphgood]

Prerequisite

• [X] I have searched Issues and Discussions but cannot get the expected help.
• [X] I have read the FAQ documentation but cannot get the expected help.
• [X] The bug has not been fixed in the latest version (master) or latest version (1.x).

Task

I'm using the official example scripts/configs for the officially supported tasks/models/datasets.

Branch

master branch https://github.com/open-mmlab/mmrotate

Environment

Install according to the official documents

Reproduces the problem - code sample

def min_area_polygons(pointsets: torch.Tensor) -> torch.Tensor: """Find the smallest polygons that surrounds all points in the point sets.

Args:
    pointsets (Tensor): point sets with shape  (N, 18).

Returns:
    torch.Tensor: Return the smallest polygons with shape (N, 8).
"""
polygons = pointsets.new_zeros((pointsets.size(0), 8))
ext_module.min_area_polygons(pointsets, polygons)
return polygons

Reproduces the problem - command or script

python tools/test.py or.py work_dirs/or1/epoch_12.pth --format-only --eval-options submission_dir=work_dirs/Task1_results

Reproduces the problem - error message

[>>>>>>>>>>>>>>>>>>>>>>>>> ] 5579/10833, 9.4 task/s, elapsed: 592s, ETA: 557sException ignored in: <function _MultiProcessingDataLoaderIter.del at 0x00000187DA88D4C0> Traceback (most recent call last): File "C:\Users\pph.conda\envs\pytorch\lib\site-packages\torch\utils\data\dataloader.py", line 1481, in del self._shutdown_workers() File "C:\Users\pph.conda\envs\pytorch\lib\site-packages\torch\utils\data\dataloader.py", line 1469, in _shutdown_workers w.terminate() File "C:\Users\pph.conda\envs\pytorch\lib\multiprocessing\process.py", line 133, in terminate self._popen.terminate() File "C:\Users\pph.conda\envs\pytorch\lib\multiprocessing\popen_spawn_win32.py", line 123, in terminate _winapi.TerminateProcess(int(self._handle), TERMINATE) PermissionError: [WinError 5] 拒绝访问。 Traceback (most recent call last): File "tools/test.py", line 257, in main() File "tools/test.py", line 222, in main outputs = single_gpu_test(model, data_loader, args.show, args.show_dir, File "C:\Users\pph.conda\envs\pytorch\lib\site-packages\mmdet\apis\test.py", line 29, in single_gpu_test result = model(return_loss=False, rescale=True, data) File "C:\Users\pph.conda\envs\pytorch\lib\site-packages\mmdet\models\detectors\base.py", line 147, in forward_test return self.simple_test(imgs[0], img_metas[0], kwargs) File "d:\mmrotate\mmrotate\mmrotate\models\detectors\single_stage.py", line 101, in simple_test bbox_list = self.bbox_head.get_bboxes( File "C:\Users\pph.conda\envs\pytorch\lib\site-packages\mmcv\runner\fp16_utils.py", line 208, in new_func return old_func(*args, **kwargs) File "d:\mmrotate\mmrotate\mmrotate\models\dense_heads\oriented_reppoints_head.py", line 1125, in get_bboxes results = self._get_bboxes_single(cls_score_list, point_pred_list, File "d:\mmrotate\mmrotate\mmrotate\models\dense_heads\oriented_reppoints_head.py", line 1216, in _get_bboxes_single polys = min_area_polygons(pts) File "C:\Users\pph.conda\envs\pytorch\lib\site-packages\mmcv\ops\min_area_polygons.py", line 19, in min_area_polygons ext_module.min_area_polygons(pointsets, polygons) RuntimeError: CUDA error: an illegal memory access was encountered Error in atexit._run_exitfuncs: Traceback (most recent call last): File "C:\Users\pph.conda\envs\pytorch\lib\multiprocessing\popen_spawn_win32.py", line 123, in terminate _winapi.TerminateProcess(int(self._handle), TERMINATE) PermissionError: [WinError 5] 拒绝访问。

Additional information

It's fine when I'm training, but when I'm using the test it happens。The same error happened when I was running on my server。 But when I tested the validation set, this error did not occur。

[zytx121]

Hi @pphgood, this may be because the min_area_polygons operator crashed because no target was detected on a picture in the test set.

• You can add try, except in _get_bboxes_single to skip it when it crashes.
• Try some potentially useful solutions mentioned in https://github.com/open-mmlab/mmrotate/issues/405.

BTW the maintenance of the old version has stopped. Welcome to the new version: https://github.com/open-mmlab/mmrotate/tree/1.x

[pphgood]

Hi @pphgood, this may be because the min_area_polygons operator crashed because no target was detected on a picture in the test set.

• You can add try, except in _get_bboxes_single to skip it when it crashes.
• Try some potentially useful solutions mentioned in CUDA error: an illegal memory access was encountered #405.

BTW the maintenance of the old version has stopped. Welcome to the new version: https://github.com/open-mmlab/mmrotate/tree/1.x Thank you for your answer, but I have another problem during the experiment. I changed the loss function. Finally, the experimental effect increased somewhat in the training set and verification set, but it was worse in the test set. May I ask if this is the cause of overfitting?