search

open-mmlab / mmdetection

OpenMMLab Detection Toolbox and Benchmark
https://mmdetection.readthedocs.io
Apache License 2.0
29.14k stars 9.39k forks source link

Customized data training, AR AP keep 0 -1, Mask RCNN #4494

Closed KevinZhoutianyi closed 3 years ago

KevinZhoutianyi commented 3 years ago

  1. I prepared the customized coco-like data set

    "info": {"description": "Example Dataset", "url": "https://github.com/waspinator/pycococreator", "version": "0.1.0", "year": 2018, "contributor": "waspinator", "date_created": "2021-01-20 10:14:04.499591"}, "licenses": [{"id": 1, "name": "Attribution-NonCommercial-ShareAlike License", "url": "http://creativecommons.org/licenses/by-nc-sa/2.0/"}], "categories": [{"id": 1, "name": "rivet", "supercategory": "class"}], "images": [{"id": 1, "file_name": "02144.png", "width": 6576, "height": 4384, "date_captured": "2021-01-20 10:14:04.499568", "license": 1, "coco_url": "", "flickr_url": ""}, {"id": 2, "file_name": "01095.png", "width": 6576, "height": 4384, "date_captured": "2021-01-20 10:14:04.499568", "license": 1, "coco_url": "", "flickr_url": ""}, {"id": 3, "file_name": "01064.png", "width": 6576, "height": 4384, "date_captured": "2021-01-20 10:14:04.499568", "license": 1, "coco_url": "", "flickr_url": ""}, {"id": 4, "file_name": "01052.png", "width": 6576, "height": 4384, "date_captured": "2021-01-20 10:14:04.499568", "license": 1, "coco_url": "", "flickr_url": ""}, {"id": 5, "file_name": "01083.png", "width": 6576, "height": 4384, "date_captured": "2021-01-20 10:14:04.499568", "license": 1, "coco_url": "", "flickr_url": ""}, {"id": 6, "file_name": "01013.png", "width": 6576, "height": 4384, "date_captured": "2021-01-20 10:14:04.499568", "license": 1, "coco_url": "", "flickr_url": ""}, {"id": 7, "file_name": "02132.png", "width": 6576, "height": 4384, "date_captured": "2021-01-20 10:14:04.499568", "license": 1, "coco_url": "", "flickr_url": ""}, {"id": 8, "file_name": "02154.png", "width": 6576, "height": 4384, "date_captured": "2021-01-20 10:14:04.499568", "license": 1, "coco_url": "", "flickr_url": ""}, {"id": 9, "file_name": "03014.png", "width": 6576, "height": 4384, "date_captured": "2021-01-20 10:14:04.499568", "license": 1, "coco_url": "", "flickr_url": ""}], "annotations": [{"id": 1, "image_id": 1, "category_id": 1, "iscrowd": 0, "area": 931959, "bbox": [0.0, 0.0, 6576.0, 4384.0], "segmentation": [[2570.0, 18.5, 2551.0, 16.5, 2543.5, 9.0, 2541.5, 0.0, 2584.0, 0, 2581.5, 12.0, 2570.0, 18.5], [3327.0, 58.5, 3307.0, 55.

  2. I write the config file referring to the tutorial

    as the detection object is small, I also change the resize image from 1330 800 to 2660 1600 in the base file

    
# The new config inherits a base config to highlight the necessary modification
_base_ = './mask_rcnn_r50_caffe_fpn_mstrain-poly_1x_coco.py'

We also need to change the num_classes in head to match the dataset's annotation

model = dict( roi_head=dict( bbox_head=dict(num_classes=1), mask_head=dict(num_classes=1)))

Modify dataset related settings

dataset_type = 'COCODataset' classes = ('rivet',) data = dict( train=dict(

explicitly add your class names to the field classes

    classes=classes,
    ann_file='/data/zhoutianyi/mmdetection/customizeddataset/TrainandTest (1)/train/AirPlaneHead_MaskTrain.json',
    img_prefix='/data/zhoutianyi/mmdetection/customizeddataset/TrainandTest (1)/train/original_images'),
val=dict(
    classes=classes,
    ann_file='/data/zhoutianyi/mmdetection/customizeddataset/TrainandTest (1)/test/AirPlaneHead_MaskTest.json',
    img_prefix='/data/zhoutianyi/mmdetection/customizeddataset/TrainandTest (1)/test/original_images'),
test=dict(
    classes=classes,
    ann_file='/data/zhoutianyi/mmdetection/customizeddataset/TrainandTest (1)/test/AirPlaneHead_MaskTest.json',
    img_prefix='/data/zhoutianyi/mmdetection/customizeddataset/TrainandTest (1)/test/original_images'))

load_from = '/data/zhoutianyi/mmdetection/checkpoints/mask_rcnn_r50_caffe_fpn_1x_coco_bbox_mAP-0.38__segm_mAP-0.344_20200504_231812-0ebd1859.pth'


3. I train the mask RCNN model

$ python3 tools/train.py /data/zhoutianyi/mmdetection/configs/mask_rcnn/my_custom_config.py --gpu-ids 3


**but the ar ap keep 0 or -1, and test.py fail to detect anything**

#### here's the log

sys.platform: linux Python: 3.7.9 (default, Aug 31 2020, 12:42:55) [GCC 7.3.0] CUDA available: True GPU 0,1,2,3: GeForce RTX 2080 Ti CUDA_HOME: /usr/local/cuda NVCC: Cuda compilation tools, release 10.2, V10.2.89 GCC: gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0 PyTorch: 1.7.0 PyTorch compiling details: PyTorch built with:

TorchVision: 0.8.1 OpenCV: 4.5.1 MMCV: 1.2.5 MMCV Compiler: GCC 7.5 MMCV CUDA Compiler: 10.2 MMDetection: 2.8.0+e1dd6e2

2021-01-21 03:03:12,766 - mmdet - INFO - Distributed training: False 2021-01-21 03:03:14,471 - mmdet - INFO - Config: model = dict( type='MaskRCNN', pretrained='open-mmlab://detectron2/resnet50_caffe', backbone=dict( type='ResNet', depth=50, num_stages=4, out_indices=(0, 1, 2, 3), frozen_stages=1, norm_cfg=dict(type='BN', requires_grad=False), norm_eval=True, style='caffe'), neck=dict( type='FPN', in_channels=[256, 512, 1024, 2048], out_channels=256, num_outs=5), rpn_head=dict( type='RPNHead', in_channels=256, feat_channels=256, anchor_generator=dict( type='AnchorGenerator', scales=[8], ratios=[0.5, 1.0, 2.0], strides=[4, 8, 16, 32, 64]), bbox_coder=dict( type='DeltaXYWHBBoxCoder', target_means=[0.0, 0.0, 0.0, 0.0], target_stds=[1.0, 1.0, 1.0, 1.0]), loss_cls=dict( type='CrossEntropyLoss', use_sigmoid=True, loss_weight=1.0), loss_bbox=dict(type='L1Loss', loss_weight=1.0)), roi_head=dict( type='StandardRoIHead', bbox_roi_extractor=dict( type='SingleRoIExtractor', roi_layer=dict(type='RoIAlign', output_size=7, sampling_ratio=0), out_channels=256, featmap_strides=[4, 8, 16, 32]), bbox_head=dict( type='Shared2FCBBoxHead', in_channels=256, fc_out_channels=1024, roi_feat_size=7, num_classes=1, bbox_coder=dict( type='DeltaXYWHBBoxCoder', target_means=[0.0, 0.0, 0.0, 0.0], target_stds=[0.1, 0.1, 0.2, 0.2]), reg_class_agnostic=False, loss_cls=dict( type='CrossEntropyLoss', use_sigmoid=False, loss_weight=1.0), loss_bbox=dict(type='L1Loss', loss_weight=1.0)), mask_roi_extractor=dict( type='SingleRoIExtractor', roi_layer=dict(type='RoIAlign', output_size=14, sampling_ratio=0), out_channels=256, featmap_strides=[4, 8, 16, 32]), mask_head=dict( type='FCNMaskHead', num_convs=4, in_channels=256, conv_out_channels=256, num_classes=1, loss_mask=dict( type='CrossEntropyLoss', use_mask=True, loss_weight=1.0))), train_cfg=dict( rpn=dict( assigner=dict( type='MaxIoUAssigner', pos_iou_thr=0.7, neg_iou_thr=0.3, min_pos_iou=0.3, match_low_quality=True, ignore_iof_thr=-1), sampler=dict( type='RandomSampler', num=256, pos_fraction=0.5, neg_pos_ub=-1, add_gt_as_proposals=False), allowed_border=-1, pos_weight=-1, debug=False), rpn_proposal=dict( nms_across_levels=False, nms_pre=2000, nms_post=1000, max_num=1000, nms_thr=0.7, min_bbox_size=0), rcnn=dict( assigner=dict( type='MaxIoUAssigner', pos_iou_thr=0.5, neg_iou_thr=0.5, min_pos_iou=0.5, match_low_quality=True, ignore_iof_thr=-1), sampler=dict( type='RandomSampler', num=512, pos_fraction=0.25, neg_pos_ub=-1, add_gt_as_proposals=True), mask_size=28, pos_weight=-1, debug=False)), test_cfg=dict( rpn=dict( nms_across_levels=False, nms_pre=1000, nms_post=1000, max_num=1000, nms_thr=0.7, min_bbox_size=0), rcnn=dict( score_thr=0.05, nms=dict(type='nms', iou_threshold=0.5), max_per_img=100, mask_thr_binary=0.5))) dataset_type = 'COCODataset' data_root = 'data/coco/' img_norm_cfg = dict( mean=[103.53, 116.28, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False) train_pipeline = [ dict(type='LoadImageFromFile'), dict( type='LoadAnnotations', with_bbox=True, with_mask=True, poly2mask=False), dict( type='Resize', img_scale=[(1333, 640), (1333, 672), (1333, 704), (1333, 736), (1333, 768), (1333, 800)], multiscale_mode='value', keep_ratio=True), dict(type='RandomFlip', flip_ratio=0.5), dict( type='Normalize', mean=[103.53, 116.28, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False), dict(type='Pad', size_divisor=32), dict(type='DefaultFormatBundle'), dict(type='Collect', keys=['img', 'gt_bboxes', 'gt_labels', 'gt_masks']) ] test_pipeline = [ dict(type='LoadImageFromFile'), dict( type='MultiScaleFlipAug', img_scale=(1333, 800), flip=False, transforms=[ dict(type='Resize', keep_ratio=True), dict(type='RandomFlip'), dict( type='Normalize', mean=[103.53, 116.28, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False), dict(type='Pad', size_divisor=32), dict(type='ImageToTensor', keys=['img']), dict(type='Collect', keys=['img']) ]) ] data = dict( samples_per_gpu=2, workers_per_gpu=2, train=dict( type='CocoDataset', ann_file= '/data/zhoutianyi/mmdetection/customizeddataset/TrainandTest (1)/train/AirPlaneHead_MaskTrain.json', img_prefix= '/data/zhoutianyi/mmdetection/customizeddataset/TrainandTest (1)/train/original_images', pipeline=[ dict(type='LoadImageFromFile'), dict( type='LoadAnnotations', with_bbox=True, with_mask=True, poly2mask=False), dict( type='Resize', img_scale=[(1333, 640), (1333, 672), (1333, 704), (1333, 736), (1333, 768), (1333, 800)], multiscale_mode='value', keep_ratio=True), dict(type='RandomFlip', flip_ratio=0.5), dict( type='Normalize', mean=[103.53, 116.28, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False), dict(type='Pad', size_divisor=32), dict(type='DefaultFormatBundle'), dict( type='Collect', keys=['img', 'gt_bboxes', 'gt_labels', 'gt_masks']) ], classes=('rivet', )), val=dict( type='CocoDataset', ann_file= '/data/zhoutianyi/mmdetection/customizeddataset/TrainandTest (1)/test/AirPlaneHead_MaskTest.json', img_prefix= '/data/zhoutianyi/mmdetection/customizeddataset/TrainandTest (1)/test/original_images', pipeline=[ dict(type='LoadImageFromFile'), dict( type='MultiScaleFlipAug', img_scale=(1333, 800), flip=False, transforms=[ dict(type='Resize', keep_ratio=True), dict(type='RandomFlip'), dict( type='Normalize', mean=[103.53, 116.28, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False), dict(type='Pad', size_divisor=32), dict(type='ImageToTensor', keys=['img']), dict(type='Collect', keys=['img']) ]) ], classes=('rivet', )), test=dict( type='CocoDataset', ann_file= '/data/zhoutianyi/mmdetection/customizeddataset/TrainandTest (1)/test/AirPlaneHead_MaskTest.json', img_prefix= '/data/zhoutianyi/mmdetection/customizeddataset/TrainandTest (1)/test/original_images', pipeline=[ dict(type='LoadImageFromFile'), dict( type='MultiScaleFlipAug', img_scale=(1333, 800), flip=False, transforms=[ dict(type='Resize', keep_ratio=True), dict(type='RandomFlip'), dict( type='Normalize', mean=[103.53, 116.28, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False), dict(type='Pad', size_divisor=32), dict(type='ImageToTensor', keys=['img']), dict(type='Collect', keys=['img']) ]) ], classes=('rivet', ))) evaluation = dict(metric=['bbox', 'segm']) optimizer = dict(type='SGD', lr=0.02, momentum=0.9, weight_decay=0.0001) optimizer_config = dict(grad_clip=None) lr_config = dict( policy='step', warmup='linear', warmup_iters=500, warmup_ratio=0.001, step=[8, 11]) total_epochs = 12 checkpoint_config = dict(interval=1) log_config = dict(interval=50, hooks=[dict(type='TextLoggerHook')]) dist_params = dict(backend='nccl') log_level = 'INFO' load_from = '/data/zhoutianyi/mmdetection/checkpoints/mask_rcnn_r50_caffe_fpn_1x_coco_bbox_mAP-0.38__segm_mAP-0.344_20200504_231812-0ebd1859.pth' resume_from = None workflow = [('train', 1)] classes = ('rivet', ) work_dir = './work_dirs/my_custom_config' gpu_ids = [3]

2021-01-21 03:03:15,003 - mmdet - INFO - load model from: open-mmlab://detectron2/resnet50_caffe 2021-01-21 03:03:15,128 - mmdet - WARNING - The model and loaded state dict do not match exactly

unexpected key in source state_dict: conv1.bias

loading annotations into memory... Done (t=0.09s) creating index... index created! loading annotations into memory... Done (t=0.04s) creating index... index created! 2021-01-21 03:03:18,371 - mmdet - INFO - load checkpoint from /data/zhoutianyi/mmdetection/checkpoints/mask_rcnn_r50_caffe_fpn_1x_coco_bbox_mAP-0.38__segm_mAP-0.344_20200504_231812-0ebd1859.pth 2021-01-21 03:03:18,645 - mmdet - WARNING - The model and loaded state dict do not match exactly

size mismatch for roi_head.bbox_head.fc_cls.weight: copying a param with shape torch.Size([81, 1024]) from checkpoint, the shape in current model is torch.Size([2, 1024]). size mismatch for roi_head.bbox_head.fc_cls.bias: copying a param with shape torch.Size([81]) from checkpoint, the shape in current model is torch.Size([2]). size mismatch for roi_head.bbox_head.fc_reg.weight: copying a param with shape torch.Size([320, 1024]) from checkpoint, the shape in current model is torch.Size([4, 1024]). size mismatch for roi_head.bbox_head.fc_reg.bias: copying a param with shape torch.Size([320]) from checkpoint, the shape in current model is torch.Size([4]). size mismatch for roi_head.mask_head.conv_logits.weight: copying a param with shape torch.Size([80, 256, 1, 1]) from checkpoint, the shape in current model is torch.Size([1, 256, 1, 1]). size mismatch for roi_head.mask_head.conv_logits.bias: copying a param with shape torch.Size([80]) from checkpoint, the shape in current model is torch.Size([1]). 2021-01-21 03:03:18,655 - mmdet - INFO - Start running, host: root@zhangsy, work_dir: /data/zhoutianyi/mmdetection/work_dirs/my_custom_config 2021-01-21 03:03:18,655 - mmdet - INFO - workflow: [('train', 1)], max: 12 epochs 2021-01-21 03:03:26,595 - mmdet - INFO - Saving checkpoint at 1 epochs [>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 2/2, 0.0 task/s, elapsed: 44s, ETA: 0s2021-01-21 03:04:18,619 - mmdet - INFO - Evaluating bbox... Loading and preparing results... DONE (t=0.00s) creating index... index created! Running per image evaluation... Evaluate annotation type bbox DONE (t=0.02s). Accumulating evaluation results... DONE (t=0.01s). Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.018 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=1000 ] = 0.056 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=1000 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = -1.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.044 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.400 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=300 ] = 0.400 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=1000 ] = 0.400 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = -1.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.400 2021-01-21 03:04:18,649 - mmdet - INFO - Evaluating segm... Loading and preparing results... DONE (t=0.01s) creating index... index created! Running per image evaluation... Evaluate annotation type segm DONE (t=0.11s). Accumulating evaluation results... DONE (t=0.01s). Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=1000 ] = 0.000 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=1000 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = -1.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=300 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=1000 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = -1.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.000 2021-01-21 03:04:18,783 - mmdet - INFO - Exp name: my_custom_config.py 2021-01-21 03:04:18,783 - mmdet - INFO - Epoch(val) [1][5] bbox_mAP: 0.0180, bbox_mAP_50: 0.0560, bbox_mAP_75: 0.0000, bbox_mAP_s: -1.0000, bbox_mAP_m: -1.0000, bbox_mAP_l: 0.0440, bbox_mAP_copypaste: 0.018 0.056 0.000 -1.000 -1.000 0.044, segm_mAP: 0.0000, segm_mAP_50: 0.0000, segm_mAP_75: 0.0000, segm_mAP_s: -1.0000, segm_mAP_m: -1.0000, segm_mAP_l: 0.0000, segm_mAP_copypaste: 0.000 0.000 0.000 -1.000 -1.000 0.000 2021-01-21 03:04:24,668 - mmdet - INFO - Saving checkpoint at 2 epochs [>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 2/2, 0.0 task/s, elapsed: 41s, ETA: 0s2021-01-21 03:05:12,380 - mmdet - INFO - Evaluating bbox... Loading and preparing results... DONE (t=0.00s) creating index... index created! Running per image evaluation... Evaluate annotation type bbox DONE (t=0.02s). Accumulating evaluation results... DONE (t=0.01s). Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.008 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=1000 ] = 0.025 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=1000 ] = 0.007 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = -1.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.105 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.450 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=300 ] = 0.450 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=1000 ] = 0.450 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = -1.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.450 2021-01-21 03:05:12,417 - mmdet - INFO - Evaluating segm... Loading and preparing results... DONE (t=0.01s) creating index... index created! Running per image evaluation... Evaluate annotation type segm DONE (t=0.06s). Accumulating evaluation results... DONE (t=0.01s). Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=1000 ] = 0.000 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=1000 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = -1.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=300 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=1000 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = -1.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.000 2021-01-21 03:05:12,592 - mmdet - INFO - Exp name: my_custom_config.py 2021-01-21 03:05:12,592 - mmdet - INFO - Epoch(val) [2][5] bbox_mAP: 0.0080, bbox_mAP_50: 0.0250, bbox_mAP_75: 0.0070, bbox_mAP_s: -1.0000, bbox_mAP_m: -1.0000, bbox_mAP_l: 0.1050, bbox_mAP_copypaste: 0.008 0.025 0.007 -1.000 -1.000 0.105, segm_mAP: 0.0000, segm_mAP_50: 0.0000, segm_mAP_75: 0.0000, segm_mAP_s: -1.0000, segm_mAP_m: -1.0000, segm_mAP_l: 0.0000, segm_mAP_copypaste: 0.000 0.000 0.000 -1.000 -1.000 0.000

XinyuSun commented 3 years ago

there must be something wrong with your json file. see my issue and check your annotation json file whether id in annotations section is always 0 which should increase from 0 to the length of annotations

KevinZhoutianyi commented 3 years ago

there must be something wrong with your json file. see my issue and check your annotation json file whether id in annotations section is always 0 which should increase from 0 to the length of annotations

Thanks for your help 

"annotations": [{"id": 1, "image_id": 1, "category_id": 1, "iscrowd": 0, "area": 931959, "bbox": [0.0, 0.0, 6576.0, 4384.0], "segmentation": [[2570.0, 9.5, 3976.0, 3041.5], ..., [1482.0, 3044.5, 1454.0, 3042.5, 1441.5, 3028.0, 1439. 5, 4382.0, 3843.0, 4370.5, 3870.0, 4366.5, 3889.0, 4373.5, 3895.0, 4383.5], [2622.0, 4383.5, 2573.5, 4382.0, 2584.0, 4374.5, 2605.0, 4373.5, 2620.0, 4379.5, 2622.0, 4383.5]], "width": 6576, "height": 4384}, {"id": 2, "image_id": 2, "category_id": 1, "iscrowd": 0, "area": 911784, "bbox": [0.0, 0.0, 6576.0, 4332.0], "segmentation": [[4229.0, 83.5, 4196.0, 81.5, 4185.0, 75....

This is my annotation part in the json file. In my case I only need to detect one type of object in the images and I do increase the id for each image.

KevinZhoutianyi commented 3 years ago

@xvjiarui please

xvjiarui commented 3 years ago

Hi @KevinZhoutianyi There is a warning that your checkpoint weight does not match. Could you please check it?

KevinZhoutianyi commented 3 years ago

Hi @xvjiarui Thanks for you help

I just exactly followed your second tutorial. I tried to use the pre-trained model and the code gave me the 'mismatch' warning. Even I did not used the pre-trained model when training, the AR AP values are still -1 and 0.

And here's my config:

# The new config inherits a base config to highlight the necessary modification
_base_ = './mask_rcnn_r50_caffe_fpn_mstrain-poly_1x_coco.py'

# We also need to change the num_classes in head to match the dataset's annotation
model = dict(
    roi_head=dict(
        bbox_head=dict(num_classes=1),
        mask_head=dict(num_classes=1)))

# Modify dataset related settings
dataset_type = 'COCODataset'
classes = ('rivet',)
data = dict(
    train=dict(
        # explicitly add your class names to the field `classes`
        classes=classes,
        ann_file='/data/zhoutianyi/mmdetection/customizeddataset/TrainandTest (1)/train/AirPlaneHead_MaskTrain.json',
        img_prefix='/data/zhoutianyi/mmdetection/customizeddataset/TrainandTest (1)/train/original_images/'),
    val=dict(
        classes=classes,
        ann_file='/data/zhoutianyi/mmdetection/customizeddataset/TrainandTest (1)/test/AirPlaneHead_MaskTest.json',
        img_prefix='/data/zhoutianyi/mmdetection/customizeddataset/TrainandTest (1)/test/original_images/'),
    test=dict(
        classes=classes,
        ann_file='/data/zhoutianyi/mmdetection/customizeddataset/TrainandTest (1)/test/AirPlaneHead_MaskTest.json',
        img_prefix='/data/zhoutianyi/mmdetection/customizeddataset/TrainandTest (1)/test/original_images/'))
load_from = '/data/zhoutianyi/mmdetection/checkpoints/mask_rcnn_r50_caffe_fpn_mstrain-poly_3x_coco_bbox_mAP-0.408__segm_mAP-0.37_20200504_163245-42aa3d00.pth'

Another point is that sometimes train.py seems gave the correct result but it will go back to -1 and 0 later

Accumulating evaluation results...
DONE (t=0.01s).
Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.015
Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.033
Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.008
Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = -1.000
Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.039
Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.500
Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.500
Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.500
Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = -1.000
Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.500
2021-01-26 07:27:16,725 - mmdet - INFO - Evaluating segm...
Loading and preparing results...
DONE (t=0.00s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *segm*
DONE (t=0.01s).
Accumulating evaluation results...
DONE (t=0.00s).
Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000
Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.000
Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.000
Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = -1.000
Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.000
Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000
Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.000
Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.000
Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = -1.000
Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.000
2021-01-26 07:27:16,745 - mmdet - INFO - Exp name: my_custom_config.py
2021-01-26 07:27:16,745 - mmdet - INFO - Epoch(val) [5][5]      bbox_mAP: 0.0150, bbox_mAP_50: 0.0330, bbox_mAP_75: 0.0080, bbox_mAP_s: -1.0000, bbox_mAP_m: -1.0000, bbox_mAP_l: 0.0390, bbox_mAP_copypaste: 0.015 0.033 0.008 -1.000 -1.000 0.039, segm_mAP: 0.0000, segm_mAP_50: 0
KevinZhoutianyi commented 3 years ago

@hhaAndroid plzzzz

DormouseLi commented 3 years ago

Hello! I have the same problem. Have you solved it?

YTEP-ZHI commented 3 years ago

I just encountered the same problem, and haven't found any helpful solutions.

not-nam-or-am-i commented 3 years ago

My result also has some -1 AP values

Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.943
Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.970
Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.970
Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = -1.000
Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.943
Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.974
Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.974
Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.974
Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = -1.000
Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.974
OrderedDict([('bbox_mAP', 0.9), ('bbox_mAP_50', 0.926), ('bbox_mAP_75', 0.926), ('bbox_mAP_s', -1.0), ('bbox_mAP_m', -1.0), ('bbox_mAP_l', 0.9), ('bbox_mAP_copypaste', '0.900 0.926 0.926 -1.000 -1.000 0.900'), ('mAP', 0.943), ('mAP_50', 0.97), ('mAP_75', 0.97), ('mAP_s', -1.0), ('mAP_m', -1.0), ('mAP_l', 0.943)])
liyewu commented 3 years ago

I met the same problem, the ap/ar values keep zeros.

RangiLyu commented 3 years ago

The AP and AR of area= small or medium are -1 means that there is no small and medium size object in your dataset. You can refer to https://cocodataset.org/ to see the definition of these metrics

BIGWangYuDong commented 3 years ago

image

According to cocotools evaluate code line 452 image

If there are no small or medium objects, the value will be set to -1.