open-mmlab / mmyolo

OpenMMLab YOLO series toolbox and benchmark. Implemented RTMDet, RTMDet-Rotated,YOLOv5, YOLOv6, YOLOv7, YOLOv8,YOLOX, PPYOLOE, etc.
https://mmyolo.readthedocs.io/zh_CN/dev/
GNU General Public License v3.0
2.93k stars 532 forks source link

The testing results of the whole dataset is empty training YOLOv8 #845

Open qybing opened 1 year ago

qybing commented 1 year ago

Prerequisite

🐞 Describe the bug

yolov8_s_syncbn_fast_8xb16-500e_coco.py:

08/08 02:27:06 - mmengine - INFO - Config: default_scope = 'mmyolo' default_hooks = dict( timer=dict(type='IterTimerHook'), logger=dict(type='LoggerHook', interval=50), param_scheduler=dict( type='YOLOv5ParamSchedulerHook', scheduler_type='linear', lr_factor=0.01, max_epochs=40), checkpoint=dict( type='CheckpointHook', interval=10, save_best='auto', max_keep_ckpts=2), sampler_seed=dict(type='DistSamplerSeedHook'), visualization=dict(type='mmdet.DetVisualizationHook')) env_cfg = dict( cudnn_benchmark=True, mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0), dist_cfg=dict(backend='nccl')) vis_backends = [ dict(type='LocalVisBackend'), ] visualizer = dict( type='mmdet.DetLocalVisualizer', vis_backends=[ dict(type='LocalVisBackend'), ], name='visualizer') log_processor = dict(type='LogProcessor', window_size=50, by_epoch=True) log_level = 'INFO' load_from = 'https://download.openmmlab.com/mmyolo/v0/yolov8/yolov8_s_syncbn_fast_8xb16-500e_coco/yolov8_s_syncbn_fast_8xb16-500e_coco_20230117_180101-5aa5f0f1.pth' resume = False file_client_args = dict(backend='disk') _file_client_args = dict(backend='disk') tta_model = dict( type='mmdet.DetTTAModel', tta_cfg=dict(nms=dict(type='nms', iou_threshold=0.65), max_per_img=300)) img_scales = [ ( 640, 640, ), ( 320, 320, ), ( 960, 960, ), ] _multiscale_resize_transforms = [ dict( type='Compose', transforms=[ dict(type='YOLOv5KeepRatioResize', scale=( 640, 640, )), dict( type='LetterResize', scale=( 640, 640, ), allow_scale_up=False, pad_val=dict(img=114)), ]), dict( type='Compose', transforms=[ dict(type='YOLOv5KeepRatioResize', scale=( 320, 320, )), dict( type='LetterResize', scale=( 320, 320, ), allow_scale_up=False, pad_val=dict(img=114)), ]), dict( type='Compose', transforms=[ dict(type='YOLOv5KeepRatioResize', scale=( 960, 960, )), dict( type='LetterResize', scale=( 960, 960, ), allow_scale_up=False, pad_val=dict(img=114)), ]), ] tta_pipeline = [ dict(type='LoadImageFromFile', file_client_args=dict(backend='disk')), dict( type='TestTimeAug', transforms=[ [ dict( type='Compose', transforms=[ dict(type='YOLOv5KeepRatioResize', scale=( 640, 640, )), dict( type='LetterResize', scale=( 640, 640, ), allow_scale_up=False, pad_val=dict(img=114)), ]), dict( type='Compose', transforms=[ dict(type='YOLOv5KeepRatioResize', scale=( 320, 320, )), dict( type='LetterResize', scale=( 320, 320, ), allow_scale_up=False, pad_val=dict(img=114)), ]), dict( type='Compose', transforms=[ dict(type='YOLOv5KeepRatioResize', scale=( 960, 960, )), dict( type='LetterResize', scale=( 960, 960, ), allow_scale_up=False, pad_val=dict(img=114)), ]), ], [ dict(type='mmdet.RandomFlip', prob=1.0), dict(type='mmdet.RandomFlip', prob=0.0), ], [ dict(type='mmdet.LoadAnnotations', with_bbox=True), ], [ dict( type='mmdet.PackDetInputs', meta_keys=( 'img_id', 'img_path', 'ori_shape', 'img_shape', 'scale_factor', 'pad_param', 'flip', 'flip_direction', )), ], ]), ] data_root = '/home/jovan/code/detection/mmyolo/data/ocr_detect/' train_ann_file = 'voc_train.json' train_data_prefix = 'JPEGImages/' val_ann_file = 'voc_val.json' val_data_prefix = 'JPEGImages/' class_name = ( 'project_name', 'project_name_all', 'project_number', 'project_number_all', 'project_manager', 'project_manager_all', 'build_unit', 'build_unit_all', 'construction_unit', 'construction_unit_all', 'confidential_person', 'confidential_person_all', 'disclosure_time', 'disclosure_time_all', 'participants_presentation', 'participants_presentation_all', 'content', ) metainfo = dict( classes=( 'project_name', 'project_name_all', 'project_number', 'project_number_all', 'project_manager', 'project_manager_all', 'build_unit', 'build_unit_all', 'construction_unit', 'construction_unit_all', 'confidential_person', 'confidential_person_all', 'disclosure_time', 'disclosure_time_all', 'participants_presentation', 'participants_presentation_all', 'content', )) num_classes = 17 train_batch_size_per_gpu = 32 train_num_workers = 8 persistent_workers = True base_lr = 0.01 max_epochs = 40 close_mosaic_epochs = 5 model_test_cfg = dict( multi_label=True, nms_pre=30000, score_thr=0.001, nms=dict(type='nms', iou_threshold=0.7), max_per_img=300) img_scale = ( 640, 640, ) dataset_type = 'YOLOv5CocoDataset' val_batch_size_per_gpu = 1 val_num_workers = 2 batch_shapes_cfg = None deepen_factor = 0.33 widen_factor = 0.5 strides = [ 8, 16, 32, ] last_stage_out_channels = 1024 num_det_layers = 3 norm_cfg = dict(type='BN', momentum=0.03, eps=0.001) affine_scale = 0.5 max_aspect_ratio = 100 tal_topk = 10 tal_alpha = 0.5 tal_beta = 6.0 loss_cls_weight = 0.5 loss_bbox_weight = 7.5 loss_dfl_weight = 0.375 lr_factor = 0.01 weight_decay = 0.0005 save_epoch_intervals = 10 val_interval_stage2 = 1 max_keep_ckpts = 2 model = dict( type='YOLODetector', data_preprocessor=dict( type='YOLOv5DetDataPreprocessor', mean=[ 0.0, 0.0, 0.0, ], std=[ 255.0, 255.0, 255.0, ], bgr_to_rgb=True), backbone=dict( type='YOLOv8CSPDarknet', arch='P5', frozen_stages=4, last_stage_out_channels=1024, deepen_factor=0.33, widen_factor=0.5, norm_cfg=dict(type='BN', momentum=0.03, eps=0.001), act_cfg=dict(type='SiLU', inplace=True)), neck=dict( type='YOLOv8PAFPN', deepen_factor=0.33, widen_factor=0.5, in_channels=[ 256, 512, 1024, ], out_channels=[ 256, 512, 1024, ], num_csp_blocks=3, norm_cfg=dict(type='BN', momentum=0.03, eps=0.001), act_cfg=dict(type='SiLU', inplace=True)), bbox_head=dict( type='YOLOv8Head', head_module=dict( type='YOLOv8HeadModule', num_classes=17, in_channels=[ 256, 512, 1024, ], widen_factor=0.5, reg_max=16, norm_cfg=dict(type='BN', momentum=0.03, eps=0.001), act_cfg=dict(type='SiLU', inplace=True), featmap_strides=[ 8, 16, 32, ]), prior_generator=dict( type='mmdet.MlvlPointGenerator', offset=0.5, strides=[ 8, 16, 32, ]), bbox_coder=dict(type='DistancePointBBoxCoder'), loss_cls=dict( type='mmdet.CrossEntropyLoss', use_sigmoid=True, reduction='none', loss_weight=0.5), loss_bbox=dict( type='IoULoss', iou_mode='ciou', bbox_format='xyxy', reduction='sum', loss_weight=7.5, return_iou=False), loss_dfl=dict( type='mmdet.DistributionFocalLoss', reduction='mean', loss_weight=0.375)), train_cfg=dict( assigner=dict( type='BatchTaskAlignedAssigner', num_classes=17, use_ciou=True, topk=10, alpha=0.5, beta=6.0, eps=1e-09)), test_cfg=dict( multi_label=True, nms_pre=30000, score_thr=0.001, nms=dict(type='nms', iou_threshold=0.7), max_per_img=300)) albu_train_transforms = [ dict(type='Blur', p=0.01), dict(type='MedianBlur', p=0.01), dict(type='ToGray', p=0.01), dict(type='CLAHE', p=0.01), ] pre_transform = [ dict(type='LoadImageFromFile', file_client_args=dict(backend='disk')), dict(type='LoadAnnotations', with_bbox=True), ] last_transform = [ dict( type='mmdet.Albu', transforms=[ dict(type='Blur', p=0.01), dict(type='MedianBlur', p=0.01), dict(type='ToGray', p=0.01), dict(type='CLAHE', p=0.01), ], bbox_params=dict( type='BboxParams', format='pascal_voc', label_fields=[ 'gt_bboxes_labels', 'gt_ignore_flags', ]), keymap=dict(img='image', gt_bboxes='bboxes')), dict(type='YOLOv5HSVRandomAug'), dict(type='mmdet.RandomFlip', prob=0.5), dict( type='mmdet.PackDetInputs', meta_keys=( 'img_id', 'img_path', 'ori_shape', 'img_shape', 'flip', 'flip_direction', )), ] train_pipeline = [ dict(type='LoadImageFromFile', file_client_args=dict(backend='disk')), dict(type='LoadAnnotations', with_bbox=True), dict( type='Mosaic', img_scale=( 640, 640, ), pad_val=114.0, pre_transform=[ dict( type='LoadImageFromFile', file_client_args=dict(backend='disk')), dict(type='LoadAnnotations', with_bbox=True), ]), dict( type='YOLOv5RandomAffine', max_rotate_degree=0.0, max_shear_degree=0.0, scaling_ratio_range=( 0.5, 1.5, ), max_aspect_ratio=100, border=( -320, -320, ), border_val=( 114, 114, 114, )), dict( type='mmdet.Albu', transforms=[ dict(type='Blur', p=0.01), dict(type='MedianBlur', p=0.01), dict(type='ToGray', p=0.01), dict(type='CLAHE', p=0.01), ], bbox_params=dict( type='BboxParams', format='pascal_voc', label_fields=[ 'gt_bboxes_labels', 'gt_ignore_flags', ]), keymap=dict(img='image', gt_bboxes='bboxes')), dict(type='YOLOv5HSVRandomAug'), dict(type='mmdet.RandomFlip', prob=0.5), dict( type='mmdet.PackDetInputs', meta_keys=( 'img_id', 'img_path', 'ori_shape', 'img_shape', 'flip', 'flip_direction', )), ] train_pipeline_stage2 = [ dict(type='LoadImageFromFile', file_client_args=dict(backend='disk')), dict(type='LoadAnnotations', with_bbox=True), dict(type='YOLOv5KeepRatioResize', scale=( 640, 640, )), dict( type='LetterResize', scale=( 640, 640, ), allow_scale_up=True, pad_val=dict(img=114.0)), dict( type='YOLOv5RandomAffine', max_rotate_degree=0.0, max_shear_degree=0.0, scaling_ratio_range=( 0.5, 1.5, ), max_aspect_ratio=100, border_val=( 114, 114, 114, )), dict( type='mmdet.Albu', transforms=[ dict(type='Blur', p=0.01), dict(type='MedianBlur', p=0.01), dict(type='ToGray', p=0.01), dict(type='CLAHE', p=0.01), ], bbox_params=dict( type='BboxParams', format='pascal_voc', label_fields=[ 'gt_bboxes_labels', 'gt_ignore_flags', ]), keymap=dict(img='image', gt_bboxes='bboxes')), dict(type='YOLOv5HSVRandomAug'), dict(type='mmdet.RandomFlip', prob=0.5), dict( type='mmdet.PackDetInputs', meta_keys=( 'img_id', 'img_path', 'ori_shape', 'img_shape', 'flip', 'flip_direction', )), ] train_dataloader = dict( batch_size=32, num_workers=8, persistent_workers=True, pin_memory=True, sampler=dict(type='DefaultSampler', shuffle=True), collate_fn=dict(type='yolov5_collate'), dataset=dict( type='YOLOv5CocoDataset', data_root='/home/jovan/code/detection/mmyolo/data/ocr_detect/', ann_file='voc_train.json', data_prefix=dict(img='JPEGImages/'), filter_cfg=dict(filter_empty_gt=False, min_size=32), pipeline=[ dict( type='LoadImageFromFile', file_client_args=dict(backend='disk')), dict(type='LoadAnnotations', with_bbox=True), dict( type='Mosaic', img_scale=( 640, 640, ), pad_val=114.0, pre_transform=[ dict( type='LoadImageFromFile', file_client_args=dict(backend='disk')), dict(type='LoadAnnotations', with_bbox=True), ]), dict( type='YOLOv5RandomAffine', max_rotate_degree=0.0, max_shear_degree=0.0, scaling_ratio_range=( 0.5, 1.5, ), max_aspect_ratio=100, border=( -320, -320, ), border_val=( 114, 114, 114, )), dict( type='mmdet.Albu', transforms=[ dict(type='Blur', p=0.01), dict(type='MedianBlur', p=0.01), dict(type='ToGray', p=0.01), dict(type='CLAHE', p=0.01), ], bbox_params=dict( type='BboxParams', format='pascal_voc', label_fields=[ 'gt_bboxes_labels', 'gt_ignore_flags', ]), keymap=dict(img='image', gt_bboxes='bboxes')), dict(type='YOLOv5HSVRandomAug'), dict(type='mmdet.RandomFlip', prob=0.5), dict( type='mmdet.PackDetInputs', meta_keys=( 'img_id', 'img_path', 'ori_shape', 'img_shape', 'flip', 'flip_direction', )), ])) test_pipeline = [ dict(type='LoadImageFromFile', file_client_args=dict(backend='disk')), dict(type='YOLOv5KeepRatioResize', scale=( 640, 640, )), dict( type='LetterResize', scale=( 640, 640, ), allow_scale_up=False, pad_val=dict(img=114)), dict(type='LoadAnnotations', with_bbox=True, scope='mmdet'), dict( type='mmdet.PackDetInputs', meta_keys=( 'img_id', 'img_path', 'ori_shape', 'img_shape', 'scale_factor', 'pad_param', )), ] val_dataloader = dict( batch_size=1, num_workers=2, persistent_workers=True, pin_memory=True, drop_last=False, sampler=dict(type='DefaultSampler', shuffle=False), dataset=dict( type='YOLOv5CocoDataset', data_root='/home/jovan/code/detection/mmyolo/data/ocr_detect/', test_mode=True, data_prefix=dict(img='JPEGImages/'), ann_file='voc_val.json', pipeline=[ dict( type='LoadImageFromFile', file_client_args=dict(backend='disk')), dict(type='YOLOv5KeepRatioResize', scale=( 640, 640, )), dict( type='LetterResize', scale=( 640, 640, ), allow_scale_up=False, pad_val=dict(img=114)), dict(type='LoadAnnotations', with_bbox=True, scope='mmdet'), dict( type='mmdet.PackDetInputs', meta_keys=( 'img_id', 'img_path', 'ori_shape', 'img_shape', 'scale_factor', 'pad_param', )), ], batch_shapes_cfg=None)) test_dataloader = dict( batch_size=1, num_workers=2, persistent_workers=True, pin_memory=True, drop_last=False, sampler=dict(type='DefaultSampler', shuffle=False), dataset=dict( type='YOLOv5CocoDataset', data_root='/home/jovan/code/detection/mmyolo/data/ocr_detect/', test_mode=True, data_prefix=dict(img='JPEGImages/'), ann_file='voc_val.json', pipeline=[ dict( type='LoadImageFromFile', file_client_args=dict(backend='disk')), dict(type='YOLOv5KeepRatioResize', scale=( 640, 640, )), dict( type='LetterResize', scale=( 640, 640, ), allow_scale_up=False, pad_val=dict(img=114)), dict(type='LoadAnnotations', with_bbox=True, scope='mmdet'), dict( type='mmdet.PackDetInputs', meta_keys=( 'img_id', 'img_path', 'ori_shape', 'img_shape', 'scale_factor', 'pad_param', )), ], batch_shapes_cfg=None)) param_scheduler = None optim_wrapper = dict( type='AmpOptimWrapper', clip_grad=dict(max_norm=10.0), optimizer=dict( type='SGD', lr=0.01, momentum=0.937, weight_decay=0.0005, nesterov=True, batch_size_per_gpu=32), constructor='YOLOv5OptimizerConstructor', loss_scale='dynamic') custom_hooks = [ dict( type='EMAHook', ema_type='ExpMomentumEMA', momentum=0.0001, update_buffers=True, strict_load=False, priority=49), dict( type='mmdet.PipelineSwitchHook', switch_epoch=35, switch_pipeline=[ dict( type='LoadImageFromFile', file_client_args=dict(backend='disk')), dict(type='LoadAnnotations', with_bbox=True), dict(type='YOLOv5KeepRatioResize', scale=( 640, 640, )), dict( type='LetterResize', scale=( 640, 640, ), allow_scale_up=True, pad_val=dict(img=114.0)), dict( type='YOLOv5RandomAffine', max_rotate_degree=0.0, max_shear_degree=0.0, scaling_ratio_range=( 0.5, 1.5, ), max_aspect_ratio=100, border_val=( 114, 114, 114, )), dict( type='mmdet.Albu', transforms=[ dict(type='Blur', p=0.01), dict(type='MedianBlur', p=0.01), dict(type='ToGray', p=0.01), dict(type='CLAHE', p=0.01), ], bbox_params=dict( type='BboxParams', format='pascal_voc', label_fields=[ 'gt_bboxes_labels', 'gt_ignore_flags', ]), keymap=dict(img='image', gt_bboxes='bboxes')), dict(type='YOLOv5HSVRandomAug'), dict(type='mmdet.RandomFlip', prob=0.5), dict( type='mmdet.PackDetInputs', meta_keys=( 'img_id', 'img_path', 'ori_shape', 'img_shape', 'flip', 'flip_direction', )), ]), ] val_evaluator = dict( type='mmdet.CocoMetric', proposal_nums=( 100, 1, 10, ), ann_file='/home/jovan/code/detection/mmyolo/data/ocr_detect/voc_val.json', metric='bbox') test_evaluator = dict( type='mmdet.CocoMetric', proposal_nums=( 100, 1, 10, ), ann_file='/home/jovan/code/detection/mmyolo/data/ocr_detect/voc_val.json', metric='bbox') train_cfg = dict( type='EpochBasedTrainLoop', max_epochs=40, val_interval=10, dynamic_intervals=[ ( 35, 1, ), ]) val_cfg = dict(type='ValLoop') test_cfg = dict(type='TestLoop') launcher = 'none' work_dir = './work_dirs/yolov8_s_syncbn_fast_8xb16-500e_coco'

But the log keeps prompting:

08/08 02:38:19 - mmengine - INFO - Epoch(val) [30][26/26] data_time: 0.0777 time: 0.1007 08/08 02:38:38 - mmengine - INFO - Exp name: yolov8_s_syncbn_fast_8xb16-500e_coco_20230808_022704 08/08 02:38:38 - mmengine - INFO - Epoch(train) [31][8/8] base_lr: 1.0000e-02 lr: 6.3603e-04 eta: 0:03:04 time: 2.2634 data_time: 2.0136 memory: 2550 grad_norm: 0.2847 loss: 0.0003 loss_cls: 0.0003 loss_bbox: 0.0000 loss_dfl: 0.0000 08/08 02:38:58 - mmengine - INFO - Exp name: yolov8_s_syncbn_fast_8xb16-500e_coco_20230808_022704 08/08 02:38:58 - mmengine - INFO - Epoch(train) [32][8/8] base_lr: 1.0000e-02 lr: 5.9351e-04 eta: 0:02:43 time: 2.2987 data_time: 2.0491 memory: 2550 grad_norm: 0.2669 loss: 0.0003 loss_cls: 0.0003 loss_bbox: 0.0000 loss_dfl: 0.0000 08/08 02:39:17 - mmengine - INFO - Exp name: yolov8_s_syncbn_fast_8xb16-500e_coco_20230808_022704 08/08 02:39:17 - mmengine - INFO - Epoch(train) [33][8/8] base_lr: 1.0000e-02 lr: 5.4704e-04 eta: 0:02:22 time: 2.2766 data_time: 2.0267 memory: 2550 grad_norm: 0.2498 loss: 0.0002 loss_cls: 0.0002 loss_bbox: 0.0000 loss_dfl: 0.0000 08/08 02:39:34 - mmengine - INFO - Exp name: yolov8_s_syncbn_fast_8xb16-500e_coco_20230808_022704 08/08 02:39:34 - mmengine - INFO - Epoch(train) [34][8/8] base_lr: 1.0000e-02 lr: 4.9661e-04 eta: 0:02:01 time: 2.2384 data_time: 1.9891 memory: 2550 grad_norm: 0.2415 loss: 0.0002 loss_cls: 0.0002 loss_bbox: 0.0000 loss_dfl: 0.0000 08/08 02:39:37 - mmengine - INFO - Evaluating bbox... Loading and preparing results... 08/08 02:39:37 - mmengine - ERROR - /opt/conda/lib/python3.7/site-packages/mmdet/evaluation/metrics/coco_metric.py - compute_metrics - 462 - The testing results of the whole dataset is empty.

Environment

sys.platform: linux Python: 3.7.10 (default, Feb 26 2021, 18:47:35) [GCC 7.3.0] CUDA available: True numpy_random_seed: 2147483648 GPU 0,1: Tesla T4 CUDA_HOME: /usr/local/cuda NVCC: Cuda compilation tools, release 11.1, V11.1.105 GCC: gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0 PyTorch: 1.9.0 PyTorch compiling details: PyTorch built with:

TorchVision: 0.10.0 OpenCV: 4.8.0 MMEngine: 0.8.2 MMCV: 2.0.1 MMDetection: 3.0.0 MMYOLO: 0.5.0+

Additional information

May I know which parameters I need to modify to solve this problem

lkh2022 commented 8 months ago

I also encountered this problem, and during the training process loss_bbox: 0.0000 loss_dfl: 0.0000 both losses are 0

slantingsun commented 6 months ago

me too