search

open-mmlab / mmrotate

OpenMMLab Rotated Object Detection Toolbox and Benchmark
https://mmrotate.readthedocs.io/en/latest/
Apache License 2.0
1.84k stars 541 forks source link

[Bug] Can't train model with the existing configs and DOTA v1.0 in the Docker #1031

Closed sjhong310 closed 4 months ago

sjhong310 commented 4 months ago

Prerequisite

Task

I'm using the official example scripts/configs for the officially supported tasks/models/datasets.

Branch

master branch https://github.com/open-mmlab/mmrotate

Environment

I am using a docker image provided at here https://github.com/open-mmlab/mmrotate/blob/main/docker/Dockerfile Didn't change anything in the file.

root@0e0a77f05000:/mmrotate# python mmrotate/utils/collect_env.py
/opt/conda/lib/python3.7/site-packages/mmcv/__init__.py:21: UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details.
  'On January 1, 2023, MMCV will release v2.0.0, in which it will remove '
sys.platform: linux
Python: 3.7.7 (default, May  7 2020, 21:25:33) [GCC 7.3.0]
CUDA available: True
GPU 0,1,2,3: Quadro RTX 6000
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 10.1, V10.1.24
GCC: gcc (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
PyTorch: 1.6.0
PyTorch compiling details: PyTorch built with:
  - GCC 7.3
  - C++ Version: 201402
  - Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v1.5.0 (Git Hash e2ac1fac44c5078ca927cb9b90e1b3066a0b2ed0)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 10.1
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_37,code=compute_37
  - CuDNN 7.6.3
  - Magma 2.5.2
  - Build settings: BLAS=MKL, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DUSE_VULKAN_WRAPPER -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_STATIC_DISPATCH=OFF,

TorchVision: 0.7.0
OpenCV: 4.9.0
MMCV: 1.7.2
MMCV Compiler: GCC 7.3
MMCV CUDA Compiler: 10.1
MMRotate: 0.3.4+9ea1aee

Reproduces the problem - code sample

2024-05-09 07:50:35,627 - mmrotate - INFO - Environment info:

sys.platform: linux Python: 3.7.7 (default, May 7 2020, 21:25:33) [GCC 7.3.0] CUDA available: True GPU 0,1,2,3: Quadro RTX 6000 CUDA_HOME: /usr/local/cuda NVCC: Cuda compilation tools, release 10.1, V10.1.24 GCC: gcc (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0 PyTorch: 1.6.0 PyTorch compiling details: PyTorch built with:

TorchVision: 0.7.0 OpenCV: 4.9.0 MMCV: 1.7.2 MMCV Compiler: GCC 7.3 MMCV CUDA Compiler: 10.1 MMRotate: 0.3.4+

2024-05-09 07:50:35,892 - mmrotate - INFO - Distributed training: False 2024-05-09 07:50:36,206 - mmrotate - INFO - Config: dataset_type = 'DOTADataset' data_root = '/mmrotate/dataset/dota/' img_norm_cfg = dict( mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True) train_pipeline = [ dict(type='LoadImageFromFile'), dict(type='LoadAnnotations', with_bbox=True), dict(type='RResize', img_scale=(1024, 1024)), dict( type='RRandomFlip', flip_ratio=[0.25, 0.25, 0.25], direction=['horizontal', 'vertical', 'diagonal'], version='le90'), dict( type='Normalize', mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True), dict(type='Pad', size_divisor=32), dict(type='DefaultFormatBundle'), dict(type='Collect', keys=['img', 'gt_bboxes', 'gt_labels']) ] test_pipeline = [ dict(type='LoadImageFromFile'), dict( type='MultiScaleFlipAug', img_scale=(1024, 1024), flip=False, transforms=[ dict(type='RResize'), dict( type='Normalize', mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True), dict(type='Pad', size_divisor=32), dict(type='DefaultFormatBundle'), dict(type='Collect', keys=['img']) ]) ] data = dict( samples_per_gpu=2, workers_per_gpu=2, train=dict( type='DOTADataset', ann_file='/mmrotate/dataset/dota/train/annotations_dota/', img_prefix='/mmrotate/dataset/dota/train/images/', pipeline=[ dict(type='LoadImageFromFile'), dict(type='LoadAnnotations', with_bbox=True), dict(type='RResize', img_scale=(1024, 1024)), dict( type='RRandomFlip', flip_ratio=[0.25, 0.25, 0.25], direction=['horizontal', 'vertical', 'diagonal'], version='le90'), dict( type='Normalize', mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True), dict(type='Pad', size_divisor=32), dict(type='DefaultFormatBundle'), dict(type='Collect', keys=['img', 'gt_bboxes', 'gt_labels']) ], version='le90'), val=dict( type='DOTADataset', ann_file='/mmrotate/dataset/dota/val/annotations_dota/', img_prefix='/mmrotate/dataset/dota/val/images/', pipeline=[ dict(type='LoadImageFromFile'), dict( type='MultiScaleFlipAug', img_scale=(1024, 1024), flip=False, transforms=[ dict(type='RResize'), dict( type='Normalize', mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True), dict(type='Pad', size_divisor=32), dict(type='DefaultFormatBundle'), dict(type='Collect', keys=['img']) ]) ], version='le90'), test=dict( type='DOTADataset', ann_file='/mmrotate/dataset/dota/test/images/', img_prefix='/mmrotate/dataset/dota/test/images/', pipeline=[ dict(type='LoadImageFromFile'), dict( type='MultiScaleFlipAug', img_scale=(1024, 1024), flip=False, transforms=[ dict(type='RResize'), dict( type='Normalize', mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True), dict(type='Pad', size_divisor=32), dict(type='DefaultFormatBundle'), dict(type='Collect', keys=['img']) ]) ], version='le90')) evaluation = dict(interval=1, metric='mAP') optimizer = dict(type='SGD', lr=0.0025, momentum=0.9, weight_decay=0.0001) optimizer_config = dict(grad_clip=dict(max_norm=35, norm_type=2)) lr_config = dict( policy='step', warmup='linear', warmup_iters=500, warmup_ratio=0.3333333333333333, step=[8, 11]) runner = dict(type='EpochBasedRunner', max_epochs=12) checkpoint_config = dict(interval=1) log_config = dict( interval=50, hooks=[dict(type='TextLoggerHook'), dict(type='TensorboardLoggerHook')]) dist_params = dict(backend='nccl') log_level = 'INFO' load_from = None resume_from = None workflow = [('train', 1)] opencv_num_threads = 0 mp_start_method = 'fork' angle_version = 'le90' model = dict( type='RotatedRetinaNet', backbone=dict( type='ResNet', depth=50, num_stages=4, out_indices=(0, 1, 2, 3), frozen_stages=1, zero_init_residual=False, norm_cfg=dict(type='BN', requires_grad=True), norm_eval=True, style='pytorch', init_cfg=dict(type='Pretrained', checkpoint='torchvision://resnet50')), neck=dict( type='FPN', in_channels=[256, 512, 1024, 2048], out_channels=256, start_level=1, add_extra_convs='on_input', num_outs=5), bbox_head=dict( type='RotatedRetinaHead', num_classes=15, in_channels=256, stacked_convs=4, feat_channels=256, assign_by_circumhbbox=None, anchor_generator=dict( type='RotatedAnchorGenerator', octave_base_scale=4, scales_per_octave=3, ratios=[1.0, 0.5, 2.0], strides=[8, 16, 32, 64, 128]), bbox_coder=dict( type='DeltaXYWHAOBBoxCoder', angle_range='le90', norm_factor=None, edge_swap=True, proj_xy=True, target_means=(0.0, 0.0, 0.0, 0.0, 0.0), target_stds=(1.0, 1.0, 1.0, 1.0, 1.0)), loss_cls=dict( type='FocalLoss', use_sigmoid=True, gamma=2.0, alpha=0.25, loss_weight=1.0), loss_bbox=dict(type='L1Loss', loss_weight=1.0)), train_cfg=dict( assigner=dict( type='MaxIoUAssigner', pos_iou_thr=0.5, neg_iou_thr=0.4, min_pos_iou=0, ignore_iof_thr=-1, iou_calculator=dict(type='RBboxOverlaps2D')), allowed_border=-1, pos_weight=-1, debug=False), test_cfg=dict( nms_pre=2000, min_bbox_size=0, score_thr=0.05, nms=dict(iou_thr=0.1), max_per_img=2000)) work_dir = 'work_dirs/retina_test_dota' auto_resume = False gpu_ids = [1]

2024-05-09 07:50:36,206 - mmrotate - INFO - Set random seed to 1074697350, deterministic: False 2024-05-09 07:50:40,843 - mmrotate - INFO - Start running, host: root@17830fabe0da, work_dir: /mmrotate/work_dirs/retina_test_dota 2024-05-09 07:50:40,844 - mmrotate - INFO - Hooks will be executed in the following order: before_run: (VERY_HIGH ) StepLrUpdaterHook
(NORMAL ) CheckpointHook
(LOW ) EvalHook
(VERY_LOW ) TextLoggerHook
(VERY_LOW ) TensorboardLoggerHook

before_train_epoch: (VERY_HIGH ) StepLrUpdaterHook
(LOW ) IterTimerHook
(LOW ) EvalHook
(VERY_LOW ) TextLoggerHook
(VERY_LOW ) TensorboardLoggerHook

before_train_iter: (VERY_HIGH ) StepLrUpdaterHook
(LOW ) IterTimerHook
(LOW ) EvalHook

after_train_iter: (ABOVE_NORMAL) OptimizerHook
(NORMAL ) CheckpointHook
(LOW ) IterTimerHook
(LOW ) EvalHook
(VERY_LOW ) TextLoggerHook
(VERY_LOW ) TensorboardLoggerHook

after_train_epoch: (NORMAL ) CheckpointHook
(LOW ) EvalHook
(VERY_LOW ) TextLoggerHook
(VERY_LOW ) TensorboardLoggerHook

before_val_epoch: (LOW ) IterTimerHook
(VERY_LOW ) TextLoggerHook
(VERY_LOW ) TensorboardLoggerHook

before_val_iter: (LOW ) IterTimerHook

after_val_iter: (LOW ) IterTimerHook

after_val_epoch: (VERY_LOW ) TextLoggerHook
(VERY_LOW ) TensorboardLoggerHook

after_run: (VERY_LOW ) TextLoggerHook
(VERY_LOW ) TensorboardLoggerHook

2024-05-09 07:50:40,844 - mmrotate - INFO - workflow: [('train', 1)], max: 12 epochs 2024-05-09 07:50:40,844 - mmrotate - INFO - Checkpoints will be saved to /mmrotate/work_dirs/retina_test_dota by HardDiskBackend. 2024-05-09 07:50:57,841 - mmrotate - INFO - Epoch [1][50/705] lr: 9.967e-04, eta: 0:47:29, time: 0.339, data_time: 0.057, memory: 3499, loss_cls: 1.1545, loss_bbox: 1.6136, loss: 2.7681, grad_norm: 2.3187 2024-05-09 07:51:12,825 - mmrotate - INFO - Epoch [1][100/705] lr: 1.163e-03, eta: 0:44:28, time: 0.300, data_time: 0.008, memory: 3499, loss_cls: 1.1424, loss_bbox: 1.6118, loss: 2.7541, grad_norm: 2.5894 2024-05-09 07:51:28,852 - mmrotate - INFO - Epoch [1][150/705] lr: 1.330e-03, eta: 0:44:16, time: 0.321, data_time: 0.008, memory: 3509, loss_cls: 1.1445, loss_bbox: 1.4542, loss: 2.5986, grad_norm: 5.9852 2024-05-09 07:51:44,991 - mmrotate - INFO - Epoch [1][200/705] lr: 1.497e-03, eta: 0:44:06, time: 0.323, data_time: 0.010, memory: 3509, loss_cls: 1.1405, loss_bbox: 2.5820, loss: 3.7225, grad_norm: 5.6293 2024-05-09 07:52:00,438 - mmrotate - INFO - Epoch [1][250/705] lr: 1.663e-03, eta: 0:43:31, time: 0.309, data_time: 0.007, memory: 3509, loss_cls: 1.1216, loss_bbox: 1.4662, loss: 2.5878, grad_norm: 3.5802 2024-05-09 07:52:16,492 - mmrotate - INFO - Epoch [1][300/705] lr: 1.830e-03, eta: 0:43:20, time: 0.321, data_time: 0.009, memory: 3509, loss_cls: 1.0774, loss_bbox: 1.4967, loss: 2.5741, grad_norm: 4.6339 2024-05-09 07:52:31,333 - mmrotate - INFO - Epoch [1][350/705] lr: 1.997e-03, eta: 0:42:38, time: 0.297, data_time: 0.010, memory: 3509, loss_cls: 1.0364, loss_bbox: 1.5021, loss: 2.5385, grad_norm: 3.2541 2024-05-09 07:52:47,354 - mmrotate - INFO - Epoch [1][400/705] lr: 2.163e-03, eta: 0:42:27, time: 0.320, data_time: 0.027, memory: 3509, loss_cls: 1.1033, loss_bbox: 1.4071, loss: 2.5104, grad_norm: 5.1294 2024-05-09 07:53:03,048 - mmrotate - INFO - Epoch [1][450/705] lr: 2.330e-03, eta: 0:42:10, time: 0.314, data_time: 0.007, memory: 3509, loss_cls: 1.1053, loss_bbox: 1.4499, loss: 2.5553, grad_norm: 4.2854 2024-05-09 07:53:19,067 - mmrotate - INFO - Epoch [1][500/705] lr: 2.497e-03, eta: 0:41:57, time: 0.320, data_time: 0.023, memory: 3509, loss_cls: 1.0981, loss_bbox: 1.4408, loss: 2.5390, grad_norm: 3.5911 2024-05-09 07:53:33,850 - mmrotate - INFO - Epoch [1][550/705] lr: 2.500e-03, eta: 0:41:27, time: 0.296, data_time: 0.010, memory: 3509, loss_cls: 1.0690, loss_bbox: 1.4869, loss: 2.5559, grad_norm: 3.3604 2024-05-09 07:53:49,977 - mmrotate - INFO - Epoch [1][600/705] lr: 2.500e-03, eta: 0:41:16, time: 0.323, data_time: 0.007, memory: 3509, loss_cls: 1.0452, loss_bbox: 1.3941, loss: 2.4394, grad_norm: 4.7154 2024-05-09 07:54:05,941 - mmrotate - INFO - Epoch [1][650/705] lr: 2.500e-03, eta: 0:41:03, time: 0.319, data_time: 0.010, memory: 3509, loss_cls: 1.0714, loss_bbox: 1.4318, loss: 2.5032, grad_norm: 2.4818 2024-05-09 07:54:21,467 - mmrotate - INFO - Epoch [1][700/705] lr: 2.500e-03, eta: 0:40:45, time: 0.311, data_time: 0.007, memory: 4198, loss_cls: 1.0048, loss_bbox: 1.5255, loss: 2.5302, grad_norm: 6.5543 2024-05-09 07:54:23,414 - mmrotate - INFO - Saving checkpoint at 1 epochs 2024-05-09 07:55:18,124 - mmrotate - INFO - +--------------------+------+------+--------+-------+ | class | gts | dets | recall | ap | +--------------------+------+------+--------+-------+ | plane | 2531 | 0 | 0.000 | 0.000 | | baseball-diamond | 214 | 0 | 0.000 | 0.000 | | bridge | 463 | 0 | 0.000 | 0.000 | | ground-track-field | 144 | 0 | 0.000 | 0.000 | | small-vehicle | 5438 | 0 | 0.000 | 0.000 | | large-vehicle | 4387 | 0 | 0.000 | 0.000 | | ship | 8960 | 0 | 0.000 | 0.000 | | tennis-court | 760 | 0 | 0.000 | 0.000 | | basketball-court | 132 | 0 | 0.000 | 0.000 | | storage-tank | 2888 | 0 | 0.000 | 0.000 | | soccer-ball-field | 153 | 0 | 0.000 | 0.000 | | roundabout | 179 | 0 | 0.000 | 0.000 | | harbor | 2090 | 0 | 0.000 | 0.000 | | swimming-pool | 440 | 0 | 0.000 | 0.000 | | helicopter | 73 | 0 | 0.000 | 0.000 | +--------------------+------+------+--------+-------+ | mAP | | | | 0.000 | +--------------------+------+------+--------+-------+ 2024-05-09 07:55:18,125 - mmrotate - INFO - Exp name: retina_test.py 2024-05-09 07:55:18,125 - mmrotate - INFO - Epoch(val) [1][458] mAP: 0.0000 2024-05-09 07:55:35,154 - mmrotate - INFO - Epoch [2][50/705] lr: 2.500e-03, eta: 0:40:24, time: 0.340, data_time: 0.061, memory: 4198, loss_cls: 1.1283, loss_bbox: 1.4365, loss: 2.5647, grad_norm: 2.8443 2024-05-09 07:55:51,192 - mmrotate - INFO - Epoch [2][100/705] lr: 2.500e-03, eta: 0:40:11, time: 0.321, data_time: 0.008, memory: 4253, loss_cls: 1.1564, loss_bbox: 1.5060, loss: 2.6624, grad_norm: 2.0132 2024-05-09 07:56:06,830 - mmrotate - INFO - Epoch [2][150/705] lr: 2.500e-03, eta: 0:39:54, time: 0.313, data_time: 0.014, memory: 4253, loss_cls: 1.0941, loss_bbox: 1.4003, loss: 2.4944, grad_norm: 1.5078 2024-05-09 07:56:22,090 - mmrotate - INFO - Epoch [2][200/705] lr: 2.500e-03, eta: 0:39:35, time: 0.305, data_time: 0.009, memory: 4253, loss_cls: 0.9978, loss_bbox: 1.3620, loss: 2.3598, grad_norm: 3.3643 2024-05-09 07:56:36,904 - mmrotate - INFO - Epoch [2][250/705] lr: 2.500e-03, eta: 0:39:12, time: 0.296, data_time: 0.008, memory: 4253, loss_cls: 1.0691, loss_bbox: 1.4037, loss: 2.4728, grad_norm: 4.1828 2024-05-09 07:56:51,801 - mmrotate - INFO - Epoch [2][300/705] lr: 2.500e-03, eta: 0:38:50, time: 0.298, data_time: 0.009, memory: 4253, loss_cls: 1.0717, loss_bbox: 2.3950, loss: 3.4667, grad_norm: 1.5941 2024-05-09 07:57:07,278 - mmrotate - INFO - Epoch [2][350/705] lr: 2.500e-03, eta: 0:38:34, time: 0.310, data_time: 0.008, memory: 4253, loss_cls: 1.0812, loss_bbox: 1.4011, loss: 2.4823, grad_norm: 3.2867 2024-05-09 07:57:22,306 - mmrotate - INFO - Epoch [2][400/705] lr: 2.500e-03, eta: 0:38:14, time: 0.301, data_time: 0.008, memory: 4253, loss_cls: 1.0413, loss_bbox: 1.3514, loss: 2.3927, grad_norm: 2.6018 2024-05-09 07:57:37,790 - mmrotate - INFO - Epoch [2][450/705] lr: 2.500e-03, eta: 0:37:58, time: 0.310, data_time: 0.008, memory: 4253, loss_cls: 1.0255, loss_bbox: 1.5244, loss: 2.5499, grad_norm: 2.2819 2024-05-09 07:57:53,342 - mmrotate - INFO - Epoch [2][500/705] lr: 2.500e-03, eta: 0:37:42, time: 0.311, data_time: 0.009, memory: 4253, loss_cls: 1.0342, loss_bbox: 1.3847, loss: 2.4188, grad_norm: 2.7886 2024-05-09 07:58:08,874 - mmrotate - INFO - Epoch [2][550/705] lr: 2.500e-03, eta: 0:37:26, time: 0.311, data_time: 0.009, memory: 4253, loss_cls: 1.0763, loss_bbox: 1.4496, loss: 2.5260, grad_norm: 1.6289 2024-05-09 07:58:24,346 - mmrotate - INFO - Epoch [2][600/705] lr: 2.500e-03, eta: 0:37:10, time: 0.309, data_time: 0.009, memory: 4253, loss_cls: 1.0347, loss_bbox: 1.4348, loss: 2.4695, grad_norm: 6.3407 2024-05-09 07:58:40,338 - mmrotate - INFO - Epoch [2][650/705] lr: 2.500e-03, eta: 0:36:56, time: 0.320, data_time: 0.011, memory: 4253, loss_cls: 0.9904, loss_bbox: 1.3666, loss: 2.3571, grad_norm: 1.4740 2024-05-09 07:58:54,697 - mmrotate - INFO - Epoch [2][700/705] lr: 2.500e-03, eta: 0:36:34, time: 0.287, data_time: 0.007, memory: 4253, loss_cls: 1.0184, loss_bbox: 1.4759, loss: 2.4943, grad_norm: 2.3987 2024-05-09 07:58:56,344 - mmrotate - INFO - Saving checkpoint at 2 epochs 2024-05-09 07:59:51,031 - mmrotate - INFO - +--------------------+------+------+--------+-------+ | class | gts | dets | recall | ap | +--------------------+------+------+--------+-------+ | plane | 2531 | 0 | 0.000 | 0.000 | | baseball-diamond | 214 | 0 | 0.000 | 0.000 | | bridge | 463 | 0 | 0.000 | 0.000 | | ground-track-field | 144 | 0 | 0.000 | 0.000 | | small-vehicle | 5438 | 0 | 0.000 | 0.000 | | large-vehicle | 4387 | 0 | 0.000 | 0.000 | | ship | 8960 | 0 | 0.000 | 0.000 | | tennis-court | 760 | 0 | 0.000 | 0.000 | | basketball-court | 132 | 0 | 0.000 | 0.000 | | storage-tank | 2888 | 0 | 0.000 | 0.000 | | soccer-ball-field | 153 | 0 | 0.000 | 0.000 | | roundabout | 179 | 0 | 0.000 | 0.000 | | harbor | 2090 | 0 | 0.000 | 0.000 | | swimming-pool | 440 | 0 | 0.000 | 0.000 | | helicopter | 73 | 0 | 0.000 | 0.000 | +--------------------+------+------+--------+-------+ | mAP | | | | 0.000 | +--------------------+------+------+--------+-------+ 2024-05-09 07:59:51,032 - mmrotate - INFO - Exp name: retina_test.py 2024-05-09 07:59:51,033 - mmrotate - INFO - Epoch(val) [2][458] mAP: 0.0000 2024-05-09 08:00:07,564 - mmrotate - INFO - Epoch [3][50/705] lr: 2.500e-03, eta: 0:36:14, time: 0.330, data_time: 0.059, memory: 4253, loss_cls: 0.9695, loss_bbox: 1.4802, loss: 2.4497, grad_norm: 1.9543 2024-05-09 08:00:22,741 - mmrotate - INFO - Epoch [3][100/705] lr: 2.500e-03, eta: 0:35:57, time: 0.304, data_time: 0.008, memory: 4253, loss_cls: 0.9811, loss_bbox: 1.4069, loss: 2.3880, grad_norm: 1.7206 2024-05-09 08:00:38,394 - mmrotate - INFO - Epoch [3][150/705] lr: 2.500e-03, eta: 0:35:42, time: 0.313, data_time: 0.016, memory: 4253, loss_cls: 0.9904, loss_bbox: 1.4214, loss: 2.4117, grad_norm: 1.6557 2024-05-09 08:00:52,759 - mmrotate - INFO - Epoch [3][200/705] lr: 2.500e-03, eta: 0:35:22, time: 0.287, data_time: 0.007, memory: 4253, loss_cls: 1.0375, loss_bbox: 1.4690, loss: 2.5065, grad_norm: 2.2356 2024-05-09 08:01:07,978 - mmrotate - INFO - Epoch [3][250/705] lr: 2.500e-03, eta: 0:35:05, time: 0.304, data_time: 0.008, memory: 4253, loss_cls: 0.9892, loss_bbox: 1.4960, loss: 2.4851, grad_norm: 3.4251 2024-05-09 08:01:23,083 - mmrotate - INFO - Epoch [3][300/705] lr: 2.500e-03, eta: 0:34:48, time: 0.302, data_time: 0.012, memory: 4253, loss_cls: 1.0163, loss_bbox: 1.3274, loss: 2.3437, grad_norm: 1.9915 2024-05-09 08:01:38,822 - mmrotate - INFO - Epoch [3][350/705] lr: 2.500e-03, eta: 0:34:34, time: 0.315, data_time: 0.013, memory: 4253, loss_cls: 0.9956, loss_bbox: 1.3539, loss: 2.3495, grad_norm: 2.0242 2024-05-09 08:01:53,751 - mmrotate - INFO - Epoch [3][400/705] lr: 2.500e-03, eta: 0:34:16, time: 0.299, data_time: 0.009, memory: 4253, loss_cls: 0.9521, loss_bbox: 1.4236, loss: 2.3757, grad_norm: 2.4350 2024-05-09 08:02:08,785 - mmrotate - INFO - Epoch [3][450/705] lr: 2.500e-03, eta: 0:33:59, time: 0.301, data_time: 0.024, memory: 4253, loss_cls: 1.1541, loss_bbox: 1.4368, loss: 2.5909, grad_norm: 2.8515 2024-05-09 08:02:24,599 - mmrotate - INFO - Epoch [3][500/705] lr: 2.500e-03, eta: 0:33:45, time: 0.316, data_time: 0.009, memory: 4253, loss_cls: 0.9641, loss_bbox: 1.4287, loss: 2.3927, grad_norm: 4.5084 2024-05-09 08:02:40,063 - mmrotate - INFO - Epoch [3][550/705] lr: 2.500e-03, eta: 0:33:30, time: 0.309, data_time: 0.013, memory: 4253, loss_cls: 0.9163, loss_bbox: 2.3040, loss: 3.2203, grad_norm: 3.4800 2024-05-09 08:02:55,640 - mmrotate - INFO - Epoch [3][600/705] lr: 2.500e-03, eta: 0:33:15, time: 0.311, data_time: 0.008, memory: 4253, loss_cls: 0.9931, loss_bbox: 1.2020, loss: 2.1951, grad_norm: 4.7726 2024-05-09 08:03:10,926 - mmrotate - INFO - Epoch [3][650/705] lr: 2.500e-03, eta: 0:32:59, time: 0.306, data_time: 0.013, memory: 4253, loss_cls: 1.0030, loss_bbox: 1.4620, loss: 2.4649, grad_norm: 3.8628 2024-05-09 08:03:26,581 - mmrotate - INFO - Epoch [3][700/705] lr: 2.500e-03, eta: 0:32:44, time: 0.313, data_time: 0.035, memory: 4253, loss_cls: 0.9505, loss_bbox: 1.3111, loss: 2.2616, grad_norm: 2.7668 2024-05-09 08:03:28,177 - mmrotate - INFO - Saving checkpoint at 3 epochs 2024-05-09 08:04:26,502 - mmrotate - INFO - +--------------------+------+-------+--------+-------+ | class | gts | dets | recall | ap | +--------------------+------+-------+--------+-------+ | plane | 2531 | 16238 | 0.025 | 0.003 | | baseball-diamond | 214 | 0 | 0.000 | 0.000 | | bridge | 463 | 0 | 0.000 | 0.000 | | ground-track-field | 144 | 0 | 0.000 | 0.000 | | small-vehicle | 5438 | 30988 | 0.028 | 0.004 | | large-vehicle | 4387 | 809 | 0.005 | 0.003 | | ship | 8960 | 8497 | 0.002 | 0.000 | | tennis-court | 760 | 900 | 0.107 | 0.100 | | basketball-court | 132 | 0 | 0.000 | 0.000 | | storage-tank | 2888 | 499 | 0.000 | 0.000 | | soccer-ball-field | 153 | 0 | 0.000 | 0.000 | | roundabout | 179 | 0 | 0.000 | 0.000 | | harbor | 2090 | 2347 | 0.000 | 0.000 | | swimming-pool | 440 | 0 | 0.000 | 0.000 | | helicopter | 73 | 0 | 0.000 | 0.000 | +--------------------+------+-------+--------+-------+ | mAP | | | | 0.007 | +--------------------+------+-------+--------+-------+ 2024-05-09 08:04:26,504 - mmrotate - INFO - Exp name: retina_test.py 2024-05-09 08:04:26,504 - mmrotate - INFO - Epoch(val) [3][458] mAP: 0.0074 2024-05-09 08:04:43,433 - mmrotate - INFO - Epoch [4][50/705] lr: 2.500e-03, eta: 0:32:26, time: 0.338, data_time: 0.061, memory: 4253, loss_cls: 0.8863, loss_bbox: 1.3444, loss: 2.2306, grad_norm: 3.2133 2024-05-09 08:05:00,264 - mmrotate - INFO - Epoch [4][100/705] lr: 2.500e-03, eta: 0:32:15, time: 0.337, data_time: 0.010, memory: 4253, loss_cls: 0.9591, loss_bbox: 1.2910, loss: 2.2501, grad_norm: 9.2290 2024-05-09 08:05:15,149 - mmrotate - INFO - Epoch [4][150/705] lr: 2.500e-03, eta: 0:31:58, time: 0.298, data_time: 0.009, memory: 4253, loss_cls: 0.9331, loss_bbox: 1.2575, loss: 2.1906, grad_norm: 3.0507 2024-05-09 08:05:30,501 - mmrotate - INFO - Epoch [4][200/705] lr: 2.500e-03, eta: 0:31:42, time: 0.307, data_time: 0.012, memory: 4253, loss_cls: 0.9232, loss_bbox: 1.2560, loss: 2.1791, grad_norm: 3.3126 2024-05-09 08:05:45,556 - mmrotate - INFO - Epoch [4][250/705] lr: 2.500e-03, eta: 0:31:25, time: 0.301, data_time: 0.009, memory: 4253, loss_cls: 0.9158, loss_bbox: 1.2029, loss: 2.1187, grad_norm: 3.2310 2024-05-09 08:06:01,298 - mmrotate - INFO - Epoch [4][300/705] lr: 2.500e-03, eta: 0:31:10, time: 0.315, data_time: 0.009, memory: 4253, loss_cls: 0.9183, loss_bbox: 1.4229, loss: 2.3412, grad_norm: 3.4405 2024-05-09 08:06:16,972 - mmrotate - INFO - Epoch [4][350/705] lr: 2.500e-03, eta: 0:30:55, time: 0.313, data_time: 0.007, memory: 4253, loss_cls: 0.8338, loss_bbox: 1.3688, loss: 2.2027, grad_norm: 4.6154 2024-05-09 08:06:31,617 - mmrotate - INFO - Epoch [4][400/705] lr: 2.500e-03, eta: 0:30:38, time: 0.293, data_time: 0.009, memory: 4253, loss_cls: 0.9425, loss_bbox: 1.3761, loss: 2.3186, grad_norm: 4.7292 2024-05-09 08:06:45,984 - mmrotate - INFO - Epoch [4][450/705] lr: 2.500e-03, eta: 0:30:20, time: 0.287, data_time: 0.007, memory: 4253, loss_cls: 1.0112, loss_bbox: 1.3542, loss: 2.3654, grad_norm: 3.4535 2024-05-09 08:07:02,151 - mmrotate - INFO - Epoch [4][500/705] lr: 2.500e-03, eta: 0:30:06, time: 0.323, data_time: 0.027, memory: 4253, loss_cls: 0.9537, loss_bbox: 1.2940, loss: 2.2477, grad_norm: 3.3590 2024-05-09 08:07:17,922 - mmrotate - INFO - Epoch [4][550/705] lr: 2.500e-03, eta: 0:29:51, time: 0.315, data_time: 0.017, memory: 4253, loss_cls: 0.8327, loss_bbox: 2.2570, loss: 3.0898, grad_norm: 3.5167 2024-05-09 08:07:33,040 - mmrotate - INFO - Epoch [4][600/705] lr: 2.500e-03, eta: 0:29:35, time: 0.302, data_time: 0.016, memory: 4253, loss_cls: 0.8491, loss_bbox: 1.3057, loss: 2.1548, grad_norm: 4.2350 2024-05-09 08:07:48,643 - mmrotate - INFO - Epoch [4][650/705] lr: 2.500e-03, eta: 0:29:20, time: 0.312, data_time: 0.009, memory: 4253, loss_cls: 0.7653, loss_bbox: 1.2757, loss: 2.0410, grad_norm: 3.8604 2024-05-09 08:08:03,853 - mmrotate - INFO - Epoch [4][700/705] lr: 2.500e-03, eta: 0:29:04, time: 0.304, data_time: 0.023, memory: 4253, loss_cls: 0.8892, loss_bbox: 1.3199, loss: 2.2091, grad_norm: 4.7549 2024-05-09 08:08:05,818 - mmrotate - INFO - Saving checkpoint at 4 epochs 2024-05-09 08:10:04,712 - mmrotate - INFO - +--------------------+------+-------+--------+-------+ | class | gts | dets | recall | ap | +--------------------+------+-------+--------+-------+ | plane | 2531 | 18036 | 0.096 | 0.035 | | baseball-diamond | 214 | 1456 | 0.005 | 0.000 | | bridge | 463 | 12954 | 0.000 | 0.000 | | ground-track-field | 144 | 336 | 0.000 | 0.000 | | small-vehicle | 5438 | 24324 | 0.018 | 0.002 | | large-vehicle | 4387 | 17781 | 0.019 | 0.002 | | ship | 8960 | 21287 | 0.002 | 0.001 | | tennis-court | 760 | 14292 | 0.303 | 0.019 | | basketball-court | 132 | 235 | 0.000 | 0.000 | | storage-tank | 2888 | 15492 | 0.008 | 0.000 | | soccer-ball-field | 153 | 297 | 0.000 | 0.000 | | roundabout | 179 | 7897 | 0.011 | 0.000 | | harbor | 2090 | 16818 | 0.023 | 0.001 | | swimming-pool | 440 | 12735 | 0.018 | 0.000 | | helicopter | 73 | 0 | 0.000 | 0.000 | +--------------------+------+-------+--------+-------+ | mAP | | | | 0.004 | +--------------------+------+-------+--------+-------+ 2024-05-09 08:10:04,713 - mmrotate - INFO - Exp name: retina_test.py 2024-05-09 08:10:04,713 - mmrotate - INFO - Epoch(val) [4][458] mAP: 0.0041 2024-05-09 08:10:20,937 - mmrotate - INFO - Epoch [5][50/705] lr: 2.500e-03, eta: 0:28:46, time: 0.324, data_time: 0.053, memory: 4253, loss_cls: 0.9601, loss_bbox: 1.2593, loss: 2.2194, grad_norm: 5.4767 2024-05-09 08:10:36,231 - mmrotate - INFO - Epoch [5][100/705] lr: 2.500e-03, eta: 0:28:30, time: 0.306, data_time: 0.007, memory: 4253, loss_cls: 0.8857, loss_bbox: 2.1185, loss: 3.0042, grad_norm: 2.8694 2024-05-09 08:10:52,221 - mmrotate - INFO - Epoch [5][150/705] lr: 2.500e-03, eta: 0:28:15, time: 0.320, data_time: 0.008, memory: 4253, loss_cls: 0.7913, loss_bbox: 1.2813, loss: 2.0725, grad_norm: 4.0616 2024-05-09 08:11:07,899 - mmrotate - INFO - Epoch [5][200/705] lr: 2.500e-03, eta: 0:28:00, time: 0.314, data_time: 0.010, memory: 4253, loss_cls: 0.7851, loss_bbox: 1.1966, loss: 1.9818, grad_norm: 3.8314 2024-05-09 08:11:24,578 - mmrotate - INFO - Epoch [5][250/705] lr: 2.500e-03, eta: 0:27:47, time: 0.334, data_time: 0.014, memory: 4253, loss_cls: 0.7784, loss_bbox: 1.2707, loss: 2.0490, grad_norm: 4.1703 2024-05-09 08:11:41,035 - mmrotate - INFO - Epoch [5][300/705] lr: 2.500e-03, eta: 0:27:33, time: 0.329, data_time: 0.016, memory: 4253, loss_cls: 0.7030, loss_bbox: 1.2389, loss: 1.9419, grad_norm: 4.2816 2024-05-09 08:11:56,472 - mmrotate - INFO - Epoch [5][350/705] lr: 2.500e-03, eta: 0:27:18, time: 0.309, data_time: 0.006, memory: 4253, loss_cls: 0.8082, loss_bbox: 1.3018, loss: 2.1099, grad_norm: 4.4020 2024-05-09 08:12:11,229 - mmrotate - INFO - Epoch [5][400/705] lr: 2.500e-03, eta: 0:27:01, time: 0.295, data_time: 0.007, memory: 4253, loss_cls: 0.7397, loss_bbox: 1.2048, loss: 1.9445, grad_norm: 3.6967 2024-05-09 08:12:26,641 - mmrotate - INFO - Epoch [5][450/705] lr: 2.500e-03, eta: 0:26:46, time: 0.308, data_time: 0.007, memory: 4253, loss_cls: 0.7426, loss_bbox: 1.2170, loss: 1.9596, grad_norm: 3.9004 2024-05-09 08:12:41,366 - mmrotate - INFO - Epoch [5][500/705] lr: 2.500e-03, eta: 0:26:29, time: 0.294, data_time: 0.011, memory: 4253, loss_cls: 0.7468, loss_bbox: 1.2114, loss: 1.9582, grad_norm: 4.0265 2024-05-09 08:12:57,406 - mmrotate - INFO - Epoch [5][550/705] lr: 2.500e-03, eta: 0:26:14, time: 0.321, data_time: 0.007, memory: 4253, loss_cls: 0.7845, loss_bbox: 1.2832, loss: 2.0677, grad_norm: 5.0221 2024-05-09 08:13:12,906 - mmrotate - INFO - Epoch [5][600/705] lr: 2.500e-03, eta: 0:25:59, time: 0.310, data_time: 0.009, memory: 4253, loss_cls: 0.7416, loss_bbox: 1.2803, loss: 2.0220, grad_norm: 4.1792 2024-05-09 08:13:27,391 - mmrotate - INFO - Epoch [5][650/705] lr: 2.500e-03, eta: 0:25:42, time: 0.290, data_time: 0.008, memory: 4253, loss_cls: 0.8260, loss_bbox: 1.2781, loss: 2.1041, grad_norm: 5.8878 2024-05-09 08:13:42,041 - mmrotate - INFO - Epoch [5][700/705] lr: 2.500e-03, eta: 0:25:25, time: 0.293, data_time: 0.014, memory: 4253, loss_cls: 0.7453, loss_bbox: 1.1510, loss: 1.8963, grad_norm: 3.5803 2024-05-09 08:13:43,378 - mmrotate - INFO - Saving checkpoint at 5 epochs 2024-05-09 08:15:42,038 - mmrotate - INFO - +--------------------+------+-------+--------+-------+ | class | gts | dets | recall | ap | +--------------------+------+-------+--------+-------+ | plane | 2531 | 58999 | 0.235 | 0.060 | | baseball-diamond | 214 | 1984 | 0.000 | 0.000 | | bridge | 463 | 0 | 0.000 | 0.000 | | ground-track-field | 144 | 1336 | 0.056 | 0.001 | | small-vehicle | 5438 | 87715 | 0.112 | 0.007 | | large-vehicle | 4387 | 50531 | 0.041 | 0.001 | | ship | 8960 | 76062 | 0.023 | 0.002 | | tennis-court | 760 | 20700 | 0.500 | 0.384 | | basketball-court | 132 | 1168 | 0.000 | 0.000 | | storage-tank | 2888 | 27669 | 0.042 | 0.001 | | soccer-ball-field | 153 | 779 | 0.000 | 0.000 | | roundabout | 179 | 1017 | 0.000 | 0.000 | | harbor | 2090 | 43787 | 0.036 | 0.000 | | swimming-pool | 440 | 16731 | 0.061 | 0.000 | | helicopter | 73 | 0 | 0.000 | 0.000 | +--------------------+------+-------+--------+-------+ | mAP | | | | 0.030 | +--------------------+------+-------+--------+-------+ 2024-05-09 08:15:42,039 - mmrotate - INFO - Exp name: retina_test.py 2024-05-09 08:15:42,039 - mmrotate - INFO - Epoch(val) [5][458] mAP: 0.0304 2024-05-09 08:15:58,070 - mmrotate - INFO - Epoch [6][50/705] lr: 2.500e-03, eta: 0:25:07, time: 0.320, data_time: 0.056, memory: 4253, loss_cls: 0.6653, loss_bbox: 1.0592, loss: 1.7246, grad_norm: 3.4496 2024-05-09 08:16:13,674 - mmrotate - INFO - Epoch [6][100/705] lr: 2.500e-03, eta: 0:24:52, time: 0.312, data_time: 0.014, memory: 4253, loss_cls: 0.7031, loss_bbox: 1.1863, loss: 1.8893, grad_norm: 4.0695 2024-05-09 08:16:28,878 - mmrotate - INFO - Epoch [6][150/705] lr: 2.500e-03, eta: 0:24:36, time: 0.304, data_time: 0.012, memory: 4253, loss_cls: 0.7351, loss_bbox: 1.1198, loss: 1.8550, grad_norm: 5.1554 2024-05-09 08:16:43,177 - mmrotate - INFO - Epoch [6][200/705] lr: 2.500e-03, eta: 0:24:19, time: 0.286, data_time: 0.012, memory: 4253, loss_cls: 0.9693, loss_bbox: 2.1695, loss: 3.1388, grad_norm: 4.2160 2024-05-09 08:16:58,550 - mmrotate - INFO - Epoch [6][250/705] lr: 2.500e-03, eta: 0:24:04, time: 0.307, data_time: 0.008, memory: 4253, loss_cls: 0.7535, loss_bbox: 1.1503, loss: 1.9038, grad_norm: 3.4019 2024-05-09 08:17:13,793 - mmrotate - INFO - Epoch [6][300/705] lr: 2.500e-03, eta: 0:23:48, time: 0.305, data_time: 0.008, memory: 4253, loss_cls: 0.7707, loss_bbox: 1.1734, loss: 1.9441, grad_norm: 5.3797 2024-05-09 08:17:29,110 - mmrotate - INFO - Epoch [6][350/705] lr: 2.500e-03, eta: 0:23:33, time: 0.306, data_time: 0.008, memory: 4253, loss_cls: 0.7215, loss_bbox: 1.1151, loss: 1.8366, grad_norm: 3.8404 2024-05-09 08:17:45,107 - mmrotate - INFO - Epoch [6][400/705] lr: 2.500e-03, eta: 0:23:18, time: 0.320, data_time: 0.030, memory: 4253, loss_cls: 0.8041, loss_bbox: 1.1411, loss: 1.9453, grad_norm: 5.0260 2024-05-09 08:18:00,344 - mmrotate - INFO - Epoch [6][450/705] lr: 2.500e-03, eta: 0:23:02, time: 0.305, data_time: 0.014, memory: 4253, loss_cls: 0.7417, loss_bbox: 1.1763, loss: 1.9179, grad_norm: 4.0390 2024-05-09 08:18:15,968 - mmrotate - INFO - Epoch [6][500/705] lr: 2.500e-03, eta: 0:22:47, time: 0.312, data_time: 0.011, memory: 4253, loss_cls: 0.7193, loss_bbox: 1.1883, loss: 1.9076, grad_norm: 4.9056 2024-05-09 08:18:31,175 - mmrotate - INFO - Epoch [6][550/705] lr: 2.500e-03, eta: 0:22:32, time: 0.304, data_time: 0.007, memory: 4253, loss_cls: 0.7330, loss_bbox: 1.1305, loss: 1.8634, grad_norm: 4.3187 2024-05-09 08:18:46,340 - mmrotate - INFO - Epoch [6][600/705] lr: 2.500e-03, eta: 0:22:16, time: 0.303, data_time: 0.014, memory: 4253, loss_cls: 0.8148, loss_bbox: 1.2134, loss: 2.0282, grad_norm: 4.9653 2024-05-09 08:19:02,050 - mmrotate - INFO - Epoch [6][650/705] lr: 2.500e-03, eta: 0:22:01, time: 0.314, data_time: 0.008, memory: 4253, loss_cls: 0.7310, loss_bbox: 1.1977, loss: 1.9286, grad_norm: 4.7498 2024-05-09 08:19:16,903 - mmrotate - INFO - Epoch [6][700/705] lr: 2.500e-03, eta: 0:21:45, time: 0.297, data_time: 0.007, memory: 4253, loss_cls: 0.6573, loss_bbox: 1.1095, loss: 1.7669, grad_norm: 4.6386 2024-05-09 08:19:18,303 - mmrotate - INFO - Saving checkpoint at 6 epochs 2024-05-09 08:20:46,536 - mmrotate - INFO - +--------------------+------+-------+--------+-------+ | class | gts | dets | recall | ap | +--------------------+------+-------+--------+-------+ | plane | 2531 | 52343 | 0.203 | 0.097 | | baseball-diamond | 214 | 4825 | 0.178 | 0.007 | | bridge | 463 | 0 | 0.000 | 0.000 | | ground-track-field | 144 | 2306 | 0.139 | 0.003 | | small-vehicle | 5438 | 58102 | 0.158 | 0.026 | | large-vehicle | 4387 | 41343 | 0.044 | 0.001 | | ship | 8960 | 56468 | 0.065 | 0.006 | | tennis-court | 760 | 12689 | 0.504 | 0.392 | | basketball-court | 132 | 2255 | 0.114 | 0.001 | | storage-tank | 2888 | 34791 | 0.065 | 0.001 | | soccer-ball-field | 153 | 1276 | 0.072 | 0.001 | | roundabout | 179 | 3505 | 0.011 | 0.000 | | harbor | 2090 | 37205 | 0.065 | 0.000 | | swimming-pool | 440 | 15501 | 0.039 | 0.000 | | helicopter | 73 | 0 | 0.000 | 0.000 | +--------------------+------+-------+--------+-------+ | mAP | | | | 0.036 | +--------------------+------+-------+--------+-------+ 2024-05-09 08:20:46,537 - mmrotate - INFO - Exp name: retina_test.py 2024-05-09 08:20:46,537 - mmrotate - INFO - Epoch(val) [6][458] mAP: 0.0357 2024-05-09 08:21:03,646 - mmrotate - INFO - Epoch [7][50/705] lr: 2.500e-03, eta: 0:21:28, time: 0.342, data_time: 0.055, memory: 4253, loss_cls: 0.7280, loss_bbox: 2.0557, loss: 2.7838, grad_norm: 4.9893 2024-05-09 08:21:18,609 - mmrotate - INFO - Epoch [7][100/705] lr: 2.500e-03, eta: 0:21:12, time: 0.299, data_time: 0.006, memory: 4253, loss_cls: 0.7216, loss_bbox: 1.1466, loss: 1.8683, grad_norm: 5.0034 2024-05-09 08:21:34,228 - mmrotate - INFO - Epoch [7][150/705] lr: 2.500e-03, eta: 0:20:57, time: 0.312, data_time: 0.007, memory: 4253, loss_cls: 0.6745, loss_bbox: 1.1132, loss: 1.7877, grad_norm: 4.4693 2024-05-09 08:21:50,101 - mmrotate - INFO - Epoch [7][200/705] lr: 2.500e-03, eta: 0:20:42, time: 0.317, data_time: 0.008, memory: 4253, loss_cls: 0.6664, loss_bbox: 1.0562, loss: 1.7226, grad_norm: 4.3926 2024-05-09 08:22:05,789 - mmrotate - INFO - Epoch [7][250/705] lr: 2.500e-03, eta: 0:20:27, time: 0.314, data_time: 0.010, memory: 4253, loss_cls: 0.6541, loss_bbox: 1.1188, loss: 1.7729, grad_norm: 4.5773 2024-05-09 08:22:21,662 - mmrotate - INFO - Epoch [7][300/705] lr: 2.500e-03, eta: 0:20:12, time: 0.317, data_time: 0.008, memory: 4253, loss_cls: 0.5705, loss_bbox: 1.0052, loss: 1.5757, grad_norm: 4.3992 2024-05-09 08:22:36,690 - mmrotate - INFO - Epoch [7][350/705] lr: 2.500e-03, eta: 0:19:56, time: 0.301, data_time: 0.008, memory: 4253, loss_cls: 0.6497, loss_bbox: 1.0912, loss: 1.7409, grad_norm: 4.7876 2024-05-09 08:22:53,790 - mmrotate - INFO - Epoch [7][400/705] lr: 2.500e-03, eta: 0:19:42, time: 0.342, data_time: 0.039, memory: 4253, loss_cls: 0.7356, loss_bbox: 1.1662, loss: 1.9019, grad_norm: 5.6723 2024-05-09 08:23:09,298 - mmrotate - INFO - Epoch [7][450/705] lr: 2.500e-03, eta: 0:19:27, time: 0.310, data_time: 0.008, memory: 4253, loss_cls: 0.6153, loss_bbox: 1.0432, loss: 1.6585, grad_norm: 4.3573 2024-05-09 08:23:23,945 - mmrotate - INFO - Epoch [7][500/705] lr: 2.500e-03, eta: 0:19:10, time: 0.293, data_time: 0.009, memory: 4253, loss_cls: 0.5882, loss_bbox: 0.9774, loss: 1.5657, grad_norm: 4.3929 2024-05-09 08:23:38,953 - mmrotate - INFO - Epoch [7][550/705] lr: 2.500e-03, eta: 0:18:55, time: 0.300, data_time: 0.017, memory: 4253, loss_cls: 0.6575, loss_bbox: 1.1640, loss: 1.8215, grad_norm: 5.1873 2024-05-09 08:23:53,583 - mmrotate - INFO - Epoch [7][600/705] lr: 2.500e-03, eta: 0:18:39, time: 0.293, data_time: 0.010, memory: 4253, loss_cls: 0.6210, loss_bbox: 1.0176, loss: 1.6385, grad_norm: 4.9087 2024-05-09 08:24:09,076 - mmrotate - INFO - Epoch [7][650/705] lr: 2.500e-03, eta: 0:18:23, time: 0.310, data_time: 0.023, memory: 4253, loss_cls: 0.6502, loss_bbox: 1.0906, loss: 1.7408, grad_norm: 4.9144 2024-05-09 08:24:24,415 - mmrotate - INFO - Epoch [7][700/705] lr: 2.500e-03, eta: 0:18:08, time: 0.307, data_time: 0.021, memory: 4253, loss_cls: 0.6441, loss_bbox: 1.0001, loss: 1.6443, grad_norm: 4.7330 2024-05-09 08:24:27,038 - mmrotate - INFO - Saving checkpoint at 7 epochs 2024-05-09 08:26:06,193 - mmrotate - INFO - +--------------------+------+--------+--------+-------+ | class | gts | dets | recall | ap | +--------------------+------+--------+--------+-------+ | plane | 2531 | 61193 | 0.339 | 0.265 | | baseball-diamond | 214 | 4147 | 0.215 | 0.006 | | bridge | 463 | 0 | 0.000 | 0.000 | | ground-track-field | 144 | 2906 | 0.160 | 0.002 | | small-vehicle | 5438 | 259215 | 0.124 | 0.032 | | large-vehicle | 4387 | 37082 | 0.078 | 0.006 | | ship | 8960 | 51592 | 0.027 | 0.002 | | tennis-court | 760 | 6343 | 0.734 | 0.295 | | basketball-court | 132 | 3011 | 0.189 | 0.002 | | storage-tank | 2888 | 101663 | 0.081 | 0.002 | | soccer-ball-field | 153 | 2904 | 0.203 | 0.004 | | roundabout | 179 | 6488 | 0.112 | 0.001 | | harbor | 2090 | 43537 | 0.074 | 0.003 | | swimming-pool | 440 | 15146 | 0.068 | 0.001 | | helicopter | 73 | 380 | 0.014 | 0.000 | +--------------------+------+--------+--------+-------+ | mAP | | | | 0.041 | +--------------------+------+--------+--------+-------+

Reproduces the problem - command or script

After change dataset path in the config file, I ran command below.

python tools/train.py configs/rotated_retinanet/rotated_retinanet_obb_r50_fpn_1x_dota_le90.py --work-dir work_dirs/retina_test_dota

Reproduces the problem - error message

Code works without any error, but mAP is too low. Loss value is not decreased well.

I used existing code without any change, I used existing docker environment, I used existing DOTA v1.0 datsaet

But I can't train my model.

What factors should I consider to fix this issue?

P.S. When you see this log https://download.openmmlab.com/mmrotate/v0.1.0/rotated_retinanet/rotated_retinanet_obb_r50_fpn_1x_dota_le90/rotated_retinanet_obb_r50_fpn_1x_dota_le90_20220128_130740.log.json They seem like they also used DOTA-v1.0 and batch size=2, (which is the same as mine) But the iter per epoch of them is 6400 my iter per epoch is 705

I think my dataset is also different to theirs, but my model's mAP should go upto 50~70% easily even though the size of dataset is small, I guess.