microsoft / SoftTeacher

Semi-Supervised Learning, Object Detection, ICCV2021
MIT License
904 stars 123 forks source link

Custom Dataset training error #73

Closed lucasmarchd01 closed 3 years ago

lucasmarchd01 commented 3 years ago

Hi there, I am hoping for some insight as to why the training on my custom dataset isn't working. My dataset has one class and I have changed the config file to reflect this along with the annotation files. "Cautery Tool" Is the name of the class. Why is it using that name to open a file? Here is the error:

(softteacher) lucasm@pmous008:~/softteacher$ bash tools/dist_train.sh configs/my_custom_config.py 1 2021-10-22 11:41:27,064 - mmdet.ssod - INFO - [<StreamHandler (INFO)>, <FileHandler /home/lucasm/softteacher/work_dirs/my_custom_config/20211022_114127.log (INFO)>] 2021-10-22 11:41:27,064 - mmdet.ssod - INFO - Environment info:

sys.platform: linux Python: 3.6.13 |Anaconda, Inc.| (default, Jun 4 2021, 14:25:59) [GCC 7.5.0] CUDA available: True GPU 0: TITAN RTX GPU 1: GeForce GTX 1080 Ti CUDA_HOME: /usr/local/cuda NVCC: Cuda compilation tools, release 10.1, V10.1.243 GCC: gcc (Ubuntu 6.5.0-2ubuntu1~16.04) 6.5.0 20181026 PyTorch: 1.6.0+cu101 PyTorch compiling details: PyTorch built with:

TorchVision: 0.7.0+cu101 OpenCV: 4.5.3 MMCV: 1.3.14 MMCV Compiler: GCC 7.3 MMCV CUDA Compiler: 10.1 MMDetection: 2.17.0+5be1762

2021-10-22 11:41:29,446 - mmdet.ssod - INFO - Distributed training: True 2021-10-22 11:41:31,955 - mmdet.ssod - INFO - Config: model = dict( type='SoftTeacher', model=dict( type='FasterRCNN', backbone=dict( type='ResNet', depth=50, num_stages=4, out_indices=(0, 1, 2, 3), frozen_stages=1, norm_cfg=dict(type='BN', requires_grad=False), norm_eval=True, style='caffe', init_cfg=dict( type='Pretrained', checkpoint='open-mmlab://detectron2/resnet50_caffe')), neck=dict( type='FPN', in_channels=[256, 512, 1024, 2048], out_channels=256, num_outs=5), rpn_head=dict( type='RPNHead', in_channels=256, feat_channels=256, anchor_generator=dict( type='AnchorGenerator', scales=[8], ratios=[0.5, 1.0, 2.0], strides=[4, 8, 16, 32, 64]), bbox_coder=dict( type='DeltaXYWHBBoxCoder', target_means=[0.0, 0.0, 0.0, 0.0], target_stds=[1.0, 1.0, 1.0, 1.0]), loss_cls=dict( type='CrossEntropyLoss', use_sigmoid=True, loss_weight=1.0), loss_bbox=dict(type='L1Loss', loss_weight=1.0)), roi_head=dict( type='StandardRoIHead', bbox_roi_extractor=dict( type='SingleRoIExtractor', roi_layer=dict( type='RoIAlign', output_size=7, sampling_ratio=0), out_channels=256, featmap_strides=[4, 8, 16, 32]), bbox_head=[ dict(type='Shared2FCBBoxHead', num_classes=1), dict(type='Shared2FCBBoxHead', num_classes=1), dict(type='Shared2FCBBoxHead', num_classes=1) ]), train_cfg=dict( rpn=dict( assigner=dict( type='MaxIoUAssigner', pos_iou_thr=0.7, neg_iou_thr=0.3, min_pos_iou=0.3, match_low_quality=True, ignore_iof_thr=-1), sampler=dict( type='RandomSampler', num=256, pos_fraction=0.5, neg_pos_ub=-1, add_gt_as_proposals=False), allowed_border=-1, pos_weight=-1, debug=False), rpn_proposal=dict( nms_pre=2000, max_per_img=1000, nms=dict(type='nms', iou_threshold=0.7), min_bbox_size=0), rcnn=dict( assigner=dict( type='MaxIoUAssigner', pos_iou_thr=0.5, neg_iou_thr=0.5, min_pos_iou=0.5, match_low_quality=False, ignore_iof_thr=-1), sampler=dict( type='RandomSampler', num=512, pos_fraction=0.25, neg_pos_ub=-1, add_gt_as_proposals=True), pos_weight=-1, debug=False)), test_cfg=dict( rpn=dict( nms_pre=1000, max_per_img=1000, nms=dict(type='nms', iou_threshold=0.7), min_bbox_size=0), rcnn=dict( score_thr=0.05, nms=dict(type='nms', iou_threshold=0.5), max_per_img=100))), train_cfg=dict( use_teacher_proposal=False, pseudo_label_initial_score_thr=0.5, rpn_pseudo_threshold=0.9, cls_pseudo_threshold=0.9, reg_pseudo_threshold=0.01, jitter_times=10, jitter_scale=0.06, min_pseduo_box_size=0, unsup_weight=2.0), test_cfg=dict(inference_on='student')) dataset_type = 'CocoDataset' data_root = 'data/coco/' img_norm_cfg = dict( mean=[103.53, 116.28, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False) train_pipeline = [ dict(type='LoadImageFromFile'), dict(type='LoadAnnotations', with_bbox=True), dict( type='Sequential', transforms=[ dict( type='RandResize', img_scale=[(1333, 400), (1333, 1200)], multiscale_mode='range', keep_ratio=True), dict(type='RandFlip', flip_ratio=0.5), dict( type='OneOf', transforms=[ dict(type='Identity'), dict(type='AutoContrast'), dict(type='RandEqualize'), dict(type='RandSolarize'), dict(type='RandColor'), dict(type='RandContrast'), dict(type='RandBrightness'), dict(type='RandSharpness'), dict(type='RandPosterize') ]) ], record=True), dict(type='Pad', size_divisor=32), dict( type='Normalize', mean=[103.53, 116.28, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False), dict(type='ExtraAttrs', tag='sup'), dict(type='DefaultFormatBundle'), dict( type='Collect', keys=['img', 'gt_bboxes', 'gt_labels'], meta_keys=('filename', 'ori_shape', 'img_shape', 'img_norm_cfg', 'pad_shape', 'scale_factor', 'tag')) ] test_pipeline = [ dict(type='LoadImageFromFile'), dict( type='MultiScaleFlipAug', img_scale=(1333, 800), flip=False, transforms=[ dict(type='Resize', keep_ratio=True), dict(type='RandomFlip'), dict( type='Normalize', mean=[103.53, 116.28, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False), dict(type='Pad', size_divisor=32), dict(type='ImageToTensor', keys=['img']), dict(type='Collect', keys=['img']) ]) ] data = dict( samples_per_gpu=8, workers_per_gpu=8, train=dict( type='SemiDataset', sup=dict( type='CocoDataset', ann_file= 'SoftTeacher_Custom/data/coco/annotations/SoftTeacherTrain.json', img_prefix='SoftTeacher_Custom/data/coco/train2017', pipeline=[ dict(type='LoadImageFromFile'), dict(type='LoadAnnotations', with_bbox=True), dict( type='Sequential', transforms=[ dict( type='RandResize', img_scale=[(1333, 400), (1333, 1200)], multiscale_mode='range', keep_ratio=True), dict(type='RandFlip', flip_ratio=0.5), dict( type='OneOf', transforms=[ dict(type='Identity'), dict(type='AutoContrast'), dict(type='RandEqualize'), dict(type='RandSolarize'), dict(type='RandColor'), dict(type='RandContrast'), dict(type='RandBrightness'), dict(type='RandSharpness'), dict(type='RandPosterize') ]) ], record=True), dict(type='Pad', size_divisor=32), dict( type='Normalize', mean=[103.53, 116.28, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False), dict(type='ExtraAttrs', tag='sup'), dict(type='DefaultFormatBundle'), dict( type='Collect', keys=['img', 'gt_bboxes', 'gt_labels'], meta_keys=('filename', 'ori_shape', 'img_shape', 'img_norm_cfg', 'pad_shape', 'scale_factor', 'tag')) ]), unsup=dict( type='CocoDataset', ann_file= 'SoftTeacher_Custom/data/coco/annotations/SoftTeacherUnlabeled.json', img_prefix='SoftTeacher_Custom/data/coco/unlabeled2017', pipeline=[ dict(type='LoadImageFromFile'), dict(type='PseudoSamples', with_bbox=True), dict( type='MultiBranch', unsup_teacher=[ dict( type='Sequential', transforms=[ dict( type='RandResize', img_scale=[(1333, 400), (1333, 1200)], multiscale_mode='range', keep_ratio=True), dict(type='RandFlip', flip_ratio=0.5), dict( type='ShuffledSequential', transforms=[ dict( type='OneOf', transforms=[ dict(type='Identity'), dict(type='AutoContrast'), dict(type='RandEqualize'), dict(type='RandSolarize'), dict(type='RandColor'), dict(type='RandContrast'), dict(type='RandBrightness'), dict(type='RandSharpness'), dict(type='RandPosterize') ]), dict( type='OneOf', transforms=[{ 'type': 'RandTranslate', 'x': (-0.1, 0.1) }, { 'type': 'RandTranslate', 'y': (-0.1, 0.1) }, { 'type': 'RandRotate', 'angle': (-30, 30) }, [{ 'type': 'RandShear', 'x': (-30, 30) }, { 'type': 'RandShear', 'y': (-30, 30) }]]) ]), dict( type='RandErase', n_iterations=(1, 5), size=[0, 0.2], squared=True) ], record=True), dict(type='Pad', size_divisor=32), dict( type='Normalize', mean=[103.53, 116.28, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False), dict(type='ExtraAttrs', tag='unsup_student'), dict(type='DefaultFormatBundle'), dict( type='Collect', keys=['img', 'gt_bboxes', 'gt_labels'], meta_keys=('filename', 'ori_shape', 'img_shape', 'img_norm_cfg', 'pad_shape', 'scale_factor', 'tag', 'transform_matrix')) ], unsup_student=[ dict( type='Sequential', transforms=[ dict( type='RandResize', img_scale=[(1333, 400), (1333, 1200)], multiscale_mode='range', keep_ratio=True), dict(type='RandFlip', flip_ratio=0.5) ], record=True), dict(type='Pad', size_divisor=32), dict( type='Normalize', mean=[103.53, 116.28, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False), dict(type='ExtraAttrs', tag='unsup_teacher'), dict(type='DefaultFormatBundle'), dict( type='Collect', keys=['img', 'gt_bboxes', 'gt_labels'], meta_keys=('filename', 'ori_shape', 'img_shape', 'img_norm_cfg', 'pad_shape', 'scale_factor', 'tag', 'transform_matrix')) ]) ], filter_empty_gt=False)), val=dict( type='CocoDataset', ann_file='SoftTeacher_Custom/data/coco/annotations/SoftTeacherVal.json', img_prefix='SoftTeacher_Custom/data/coco/val2017', pipeline=[ dict(type='LoadImageFromFile'), dict( type='MultiScaleFlipAug', img_scale=(1333, 800), flip=False, transforms=[ dict(type='Resize', keep_ratio=True), dict(type='RandomFlip'), dict( type='Normalize', mean=[103.53, 116.28, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False), dict(type='Pad', size_divisor=32), dict(type='ImageToTensor', keys=['img']), dict(type='Collect', keys=['img']) ]) ], classes='Cautery Tool'), test=dict( type='CocoDataset', ann_file='data/coco/annotations/instances_val2017.json', img_prefix='data/coco/val2017/', pipeline=[ dict(type='LoadImageFromFile'), dict( type='MultiScaleFlipAug', img_scale=(1333, 800), flip=False, transforms=[ dict(type='Resize', keep_ratio=True), dict(type='RandomFlip'), dict( type='Normalize', mean=[103.53, 116.28, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False), dict(type='Pad', size_divisor=32), dict(type='ImageToTensor', keys=['img']), dict(type='Collect', keys=['img']) ]) ]), sampler=dict( train=dict( type='SemiBalanceSampler', sample_ratio=[1, 1], by_prob=True, epoch_length=7330))) evaluation = dict(interval=4000, metric='bbox', type='SubModulesDistEvalHook') optimizer = dict(type='SGD', lr=0.01, momentum=0.9, weight_decay=0.0001) optimizer_config = dict(grad_clip=None) lr_config = dict( policy='step', warmup='linear', warmup_iters=500, warmup_ratio=0.001, step=[480000, 640000]) runner = dict(type='IterBasedRunner', max_iters=720000) checkpoint_config = dict(interval=4000, by_epoch=False, max_keep_ckpts=20) log_config = dict( interval=50, hooks=[ dict(type='TextLoggerHook', by_epoch=False), dict( type='WandbLoggerHook', init_kwargs=dict( project='pre_release', name='my_custom_config', config=dict( work_dirs='./work_dirs/my_custom_config', total_step=720000)), by_epoch=False) ]) custom_hooks = [ dict(type='NumClassCheckHook'), dict(type='WeightSummary'), dict(type='MeanTeacher', momentum=0.999, interval=1, warm_up=0) ] dist_params = dict(backend='nccl') log_level = 'INFO' load_from = None resume_from = None workflow = [('train', 1)] mmdet_base = '../../thirdparty/mmdetection/configs/base' strong_pipeline = [ dict( type='Sequential', transforms=[ dict( type='RandResize', img_scale=[(1333, 400), (1333, 1200)], multiscale_mode='range', keep_ratio=True), dict(type='RandFlip', flip_ratio=0.5), dict( type='ShuffledSequential', transforms=[ dict( type='OneOf', transforms=[ dict(type='Identity'), dict(type='AutoContrast'), dict(type='RandEqualize'), dict(type='RandSolarize'), dict(type='RandColor'), dict(type='RandContrast'), dict(type='RandBrightness'), dict(type='RandSharpness'), dict(type='RandPosterize') ]), dict( type='OneOf', transforms=[{ 'type': 'RandTranslate', 'x': (-0.1, 0.1) }, { 'type': 'RandTranslate', 'y': (-0.1, 0.1) }, { 'type': 'RandRotate', 'angle': (-30, 30) }, [{ 'type': 'RandShear', 'x': (-30, 30) }, { 'type': 'RandShear', 'y': (-30, 30) }]]) ]), dict( type='RandErase', n_iterations=(1, 5), size=[0, 0.2], squared=True) ], record=True), dict(type='Pad', size_divisor=32), dict( type='Normalize', mean=[103.53, 116.28, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False), dict(type='ExtraAttrs', tag='unsup_student'), dict(type='DefaultFormatBundle'), dict( type='Collect', keys=['img', 'gt_bboxes', 'gt_labels'], meta_keys=('filename', 'ori_shape', 'img_shape', 'img_norm_cfg', 'pad_shape', 'scale_factor', 'tag', 'transform_matrix')) ] weak_pipeline = [ dict( type='Sequential', transforms=[ dict( type='RandResize', img_scale=[(1333, 400), (1333, 1200)], multiscale_mode='range', keep_ratio=True), dict(type='RandFlip', flip_ratio=0.5) ], record=True), dict(type='Pad', size_divisor=32), dict( type='Normalize', mean=[103.53, 116.28, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False), dict(type='ExtraAttrs', tag='unsup_teacher'), dict(type='DefaultFormatBundle'), dict( type='Collect', keys=['img', 'gt_bboxes', 'gt_labels'], meta_keys=('filename', 'ori_shape', 'img_shape', 'img_norm_cfg', 'pad_shape', 'scale_factor', 'tag', 'transform_matrix')) ] unsup_pipeline = [ dict(type='LoadImageFromFile'), dict(type='PseudoSamples', with_bbox=True), dict( type='MultiBranch', unsup_teacher=[ dict( type='Sequential', transforms=[ dict( type='RandResize', img_scale=[(1333, 400), (1333, 1200)], multiscale_mode='range', keep_ratio=True), dict(type='RandFlip', flip_ratio=0.5), dict( type='ShuffledSequential', transforms=[ dict( type='OneOf', transforms=[ dict(type='Identity'), dict(type='AutoContrast'), dict(type='RandEqualize'), dict(type='RandSolarize'), dict(type='RandColor'), dict(type='RandContrast'), dict(type='RandBrightness'), dict(type='RandSharpness'), dict(type='RandPosterize') ]), dict( type='OneOf', transforms=[{ 'type': 'RandTranslate', 'x': (-0.1, 0.1) }, { 'type': 'RandTranslate', 'y': (-0.1, 0.1) }, { 'type': 'RandRotate', 'angle': (-30, 30) }, [{ 'type': 'RandShear', 'x': (-30, 30) }, { 'type': 'RandShear', 'y': (-30, 30) }]]) ]), dict( type='RandErase', n_iterations=(1, 5), size=[0, 0.2], squared=True) ], record=True), dict(type='Pad', size_divisor=32), dict( type='Normalize', mean=[103.53, 116.28, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False), dict(type='ExtraAttrs', tag='unsup_student'), dict(type='DefaultFormatBundle'), dict( type='Collect', keys=['img', 'gt_bboxes', 'gt_labels'], meta_keys=('filename', 'ori_shape', 'img_shape', 'img_norm_cfg', 'pad_shape', 'scale_factor', 'tag', 'transform_matrix')) ], unsup_student=[ dict( type='Sequential', transforms=[ dict( type='RandResize', img_scale=[(1333, 400), (1333, 1200)], multiscale_mode='range', keep_ratio=True), dict(type='RandFlip', flip_ratio=0.5) ], record=True), dict(type='Pad', size_divisor=32), dict( type='Normalize', mean=[103.53, 116.28, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False), dict(type='ExtraAttrs', tag='unsup_teacher'), dict(type='DefaultFormatBundle'), dict( type='Collect', keys=['img', 'gt_bboxes', 'gt_labels'], meta_keys=('filename', 'ori_shape', 'img_shape', 'img_norm_cfg', 'pad_shape', 'scale_factor', 'tag', 'transform_matrix')) ]) ] fp16 = dict(loss_scale='dynamic') classes = 'Cautery Tool' work_dir = './work_dirs/my_custom_config' cfg_name = 'my_custom_config' gpu_ids = range(0, 1)

/home/lucasm/softteacher/thirdparty/mmdetection/mmdet/core/anchor/builder.py:17: UserWarning: build_anchor_generator would be deprecated soon, please use build_prior_generator 'build_anchor_generator would be deprecated soon, please use ' 2021-10-22 11:41:32,725 - mmdet.ssod - INFO - initialize ResNet with init_cfg {'type': 'Pretrained', 'checkpoint': 'open-mmlab://detectron2/resnet50_caffe'} 2021-10-22 11:41:32,726 - mmcv - INFO - load model from: open-mmlab://detectron2/resnet50_caffe 2021-10-22 11:41:32,726 - mmcv - INFO - Use load_from_openmmlab loader 2021-10-22 11:41:32,779 - mmcv - WARNING - The model and loaded state dict do not match exactly

unexpected key in source state_dict: conv1.bias

2021-10-22 11:41:32,796 - mmdet.ssod - INFO - initialize FPN with init_cfg {'type': 'Xavier', 'layer': 'Conv2d', 'distribution': 'uniform'} 2021-10-22 11:41:32,812 - mmdet.ssod - INFO - initialize RPNHead with init_cfg {'type': 'Normal', 'layer': 'Conv2d', 'std': 0.01} 2021-10-22 11:41:32,817 - mmdet.ssod - INFO - initialize Shared2FCBBoxHead with init_cfg [{'type': 'Normal', 'std': 0.01, 'override': {'name': 'fc_cls'}}, {'type': 'Normal', 'std': 0.001, 'override': {'name': 'fc_reg'}}, {'type': 'Xavier', 'layer': 'Linear', 'override': [{'name': 'shared_fcs'}, {'name': 'cls_fcs'}, {'name': 'reg_fcs'}]}] 2021-10-22 11:41:33,087 - mmdet.ssod - INFO - initialize Shared2FCBBoxHead with init_cfg [{'type': 'Normal', 'std': 0.01, 'override': {'name': 'fc_cls'}}, {'type': 'Normal', 'std': 0.001, 'override': {'name': 'fc_reg'}}, {'type': 'Xavier', 'layer': 'Linear', 'override': [{'name': 'shared_fcs'}, {'name': 'cls_fcs'}, {'name': 'reg_fcs'}]}] 2021-10-22 11:41:33,348 - mmdet.ssod - INFO - initialize Shared2FCBBoxHead with init_cfg [{'type': 'Normal', 'std': 0.01, 'override': {'name': 'fc_cls'}}, {'type': 'Normal', 'std': 0.001, 'override': {'name': 'fc_reg'}}, {'type': 'Xavier', 'layer': 'Linear', 'override': [{'name': 'shared_fcs'}, {'name': 'cls_fcs'}, {'name': 'reg_fcs'}]}] 2021-10-22 11:41:33,635 - mmdet.ssod - INFO - initialize ResNet with init_cfg {'type': 'Pretrained', 'checkpoint': 'open-mmlab://detectron2/resnet50_caffe'} 2021-10-22 11:41:33,635 - mmcv - INFO - load model from: open-mmlab://detectron2/resnet50_caffe 2021-10-22 11:41:33,635 - mmcv - INFO - Use load_from_openmmlab loader 2021-10-22 11:41:33,681 - mmcv - WARNING - The model and loaded state dict do not match exactly

unexpected key in source state_dict: conv1.bias

2021-10-22 11:41:33,698 - mmdet.ssod - INFO - initialize FPN with init_cfg {'type': 'Xavier', 'layer': 'Conv2d', 'distribution': 'uniform'} 2021-10-22 11:41:33,713 - mmdet.ssod - INFO - initialize RPNHead with init_cfg {'type': 'Normal', 'layer': 'Conv2d', 'std': 0.01} 2021-10-22 11:41:33,718 - mmdet.ssod - INFO - initialize Shared2FCBBoxHead with init_cfg [{'type': 'Normal', 'std': 0.01, 'override': {'name': 'fc_cls'}}, {'type': 'Normal', 'std': 0.001, 'override': {'name': 'fc_reg'}}, {'type': 'Xavier', 'layer': 'Linear', 'override': [{'name': 'shared_fcs'}, {'name': 'cls_fcs'}, {'name': 'reg_fcs'}]}] 2021-10-22 11:41:33,961 - mmdet.ssod - INFO - initialize Shared2FCBBoxHead with init_cfg [{'type': 'Normal', 'std': 0.01, 'override': {'name': 'fc_cls'}}, {'type': 'Normal', 'std': 0.001, 'override': {'name': 'fc_reg'}}, {'type': 'Xavier', 'layer': 'Linear', 'override': [{'name': 'shared_fcs'}, {'name': 'cls_fcs'}, {'name': 'reg_fcs'}]}] 2021-10-22 11:41:34,206 - mmdet.ssod - INFO - initialize Shared2FCBBoxHead with init_cfg [{'type': 'Normal', 'std': 0.01, 'override': {'name': 'fc_cls'}}, {'type': 'Normal', 'std': 0.001, 'override': {'name': 'fc_reg'}}, {'type': 'Xavier', 'layer': 'Linear', 'override': [{'name': 'shared_fcs'}, {'name': 'cls_fcs'}, {'name': 'reg_fcs'}]}] loading annotations into memory... Done (t=0.00s) creating index... index created! loading annotations into memory... Done (t=0.00s) creating index... index created! Traceback (most recent call last): File "/home/lucasm/anaconda3/envs/softteacher/lib/python3.6/site-packages/mmcv/utils/registry.py", line 52, in build_from_cfg return obj_cls(**args) File "/home/lucasm/softteacher/thirdparty/mmdetection/mmdet/datasets/custom.py", line 74, in init self.CLASSES = self.get_classes(classes) File "/home/lucasm/softteacher/thirdparty/mmdetection/mmdet/datasets/custom.py", line 257, in get_classes class_names = mmcv.list_from_file(classes) File "/home/lucasm/anaconda3/envs/softteacher/lib/python3.6/site-packages/mmcv/fileio/parse.py", line 18, in list_from_file with open(filename, 'r', encoding=encoding) as f: FileNotFoundError: [Errno 2] No such file or directory: 'Cautery Tool'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "tools/train.py", line 198, in main() File "tools/train.py", line 193, in main meta=meta, File "/home/lucasm/softteacher/ssod/apis/train.py", line 156, in train_detector val_dataset = build_dataset(cfg.data.val, dict(test_mode=True)) File "/home/lucasm/softteacher/thirdparty/mmdetection/mmdet/datasets/builder.py", line 78, in build_dataset dataset = build_from_cfg(cfg, DATASETS, default_args) File "/home/lucasm/anaconda3/envs/softteacher/lib/python3.6/site-packages/mmcv/utils/registry.py", line 55, in build_from_cfg raise type(e)(f'{obj_cls.name}: {e}') FileNotFoundError: CocoDataset: [Errno 2] No such file or directory: 'Cautery Tool' Traceback (most recent call last): File "/home/lucasm/anaconda3/envs/softteacher/lib/python3.6/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/home/lucasm/anaconda3/envs/softteacher/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/lucasm/anaconda3/envs/softteacher/lib/python3.6/site-packages/torch/distributed/launch.py", line 261, in main() File "/home/lucasm/anaconda3/envs/softteacher/lib/python3.6/site-packages/torch/distributed/launch.py", line 257, in main cmd=cmd) subprocess.CalledProcessError: Command '['/home/lucasm/anaconda3/envs/softteacher/bin/python', '-u', 'tools/train.py', '--local_rank=0', 'configs/my_custom_config.py', '--launcher', 'pytorch']' returned non-zero exit status 1.

MendelXu commented 3 years ago

You can not specify the selected classes as classes = 'Cautery Tool'. Instead, you should write it as classes = ['Cautery Tool']. Otherwise, it will recognize the classes variable as a file path. See the docstring here https://github.com/open-mmlab/mmdetection/blob/9874180a015beee874c904357fe4ed4fab4b46a4/mmdet/datasets/custom.py#L243

lucasmarchd01 commented 3 years ago

Thanks so much, this fixed the error. I've come across a new error in training. Are empty bounding box annotations allowed in the dataset? I suspect this could be where it's coming from.

/home/lucasm/softteacher/thirdparty/mmdetection/mmdet/core/anchor/anchor_generator.py:324: UserWarning: grid_anchors would be deprecated soon. Please use grid_priors warnings.warn('grid_anchors would be deprecated soon. ' /home/lucasm/softteacher/thirdparty/mmdetection/mmdet/core/anchor/anchor_generator.py:361: UserWarning: single_level_grid_anchors would be deprecated soon. Please use single_level_grid_priors 'single_level_grid_anchors would be deprecated soon. ' Traceback (most recent call last): File "tools/train.py", line 198, in main() File "tools/train.py", line 193, in main meta=meta, File "/home/lucasm/softteacher/ssod/apis/train.py", line 206, in train_detector runner.run(data_loaders, cfg.workflow) File "/home/lucasm/anaconda3/envs/softteacher/lib/python3.6/site-packages/mmcv/runner/iter_based_runner.py", line 133, in run iter_runner(iter_loaders[i], kwargs) File "/home/lucasm/anaconda3/envs/softteacher/lib/python3.6/site-packages/mmcv/runner/iter_based_runner.py", line 60, in train outputs = self.model.train_step(data_batch, self.optimizer, kwargs) File "/home/lucasm/anaconda3/envs/softteacher/lib/python3.6/site-packages/mmcv/parallel/distributed.py", line 52, in train_step output = self.module.train_step(inputs[0], kwargs[0]) File "/home/lucasm/softteacher/thirdparty/mmdetection/mmdet/models/detectors/base.py", line 238, in train_step losses = self(data) File "/home/lucasm/anaconda3/envs/softteacher/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl result = self.forward(input, kwargs) File "/home/lucasm/anaconda3/envs/softteacher/lib/python3.6/site-packages/mmcv/runner/fp16_utils.py", line 128, in new_func output = old_func(new_args, new_kwargs) File "/home/lucasm/softteacher/thirdparty/mmdetection/mmdet/models/detectors/base.py", line 172, in forward return self.forward_train(img, img_metas, kwargs) File "/home/lucasm/softteacher/ssod/models/soft_teacher.py", line 44, in forward_train sup_loss = self.student.forward_train(data_groups["sup"]) File "/home/lucasm/softteacher/thirdparty/mmdetection/mmdet/models/detectors/two_stage.py", line 150, in forward_train kwargs) File "/home/lucasm/softteacher/thirdparty/mmdetection/mmdet/models/roi_heads/standard_roi_head.py", line 106, in forward_train img_metas) File "/home/lucasm/softteacher/thirdparty/mmdetection/mmdet/models/roi_heads/standard_roi_head.py", line 135, in _bbox_forward_train bbox_results = self._bbox_forward(x, rois) File "/home/lucasm/softteacher/thirdparty/mmdetection/mmdet/models/roi_heads/standard_roi_head.py", line 125, in _bbox_forward cls_score, bbox_pred = self.bbox_head(bbox_feats) File "/home/lucasm/anaconda3/envs/softteacher/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl result = self.forward(input, kwargs) File "/home/lucasm/anaconda3/envs/softteacher/lib/python3.6/site-packages/torch/nn/modules/container.py", line 117, in forward input = module(input) File "/home/lucasm/anaconda3/envs/softteacher/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl result = self.forward(*input, **kwargs) File "/home/lucasm/softteacher/thirdparty/mmdetection/mmdet/models/roi_heads/bbox_heads/convfc_bbox_head.py", line 162, in forward x = x.flatten(1) AttributeError: 'tuple' object has no attribute 'flatten'

MendelXu commented 3 years ago

For labeled data, samples with empty annotations will be filtered out during preprocessing. So I think the error may not be related to your concern and it may be related to https://github.com/open-mmlab/mmdetection/issues/6141

lucasmarchd01 commented 3 years ago

Thank you, modifying the config file as that issue suggested solved the error and training on the custom data starts.

pynsigrid commented 2 years ago

Hi, @lucasmarchd01 @MendelXu . I met the same question with yours. As the issues mentioned above, I add some words in the train config: configs/soft_teacher/soft_teacher_faster_rcnn_r50_caffe_fpn_coco_full_720k.py: classes = ['chocolate'] But the running results, as my prints showed, seem to be still the 'coco classes': ('person', 'bicycle', 'car'...)

Could you please tell me where to modify if training on custom dataset.

MendelXu commented 2 years ago

Do you add classes = ['chocolate'] in the data dict like

data=dict(
  train=dict(sup=dict(
...
classes=['chocolate']
...
),unsup=dict(
...
classes=['chocolate']
...
)
),val=dict(
...
classes=['chocolate']
...
))

?