kekedan commented 5 years ago

I train data with one class, it's failure。 the config is modified on faster_rcnn_r50_fpn_1x_voc0712.py。 I change the num_classes to 2 ,it will failure, however when the num_classes is 21, it success。 also，when num_classes>=3 ，it can run success。

wthat's wrong ? can you help me ,thanks! @hellock

model settings

model = dict( type='FasterRCNN', pretrained='modelzoo://resnet50', backbone=dict( type='ResNet', depth=50, num_stages=4, out_indices=(0, 1, 2, 3), frozen_stages=1, style='pytorch'), neck=dict( type='FPN', in_channels=[256, 512, 1024, 2048], out_channels=256, num_outs=5), rpn_head=dict( type='RPNHead', in_channels=256, feat_channels=256, anchor_scales=[8], anchor_ratios=[0.5, 1.0, 2.0], anchor_strides=[4, 8, 16, 32, 64], target_means=[.0, .0, .0, .0], target_stds=[1.0, 1.0, 1.0, 1.0], use_sigmoid_cls=True), bbox_roi_extractor=dict( type='SingleRoIExtractor', roi_layer=dict(type='RoIAlign', out_size=7, sample_num=2), out_channels=256, featmap_strides=[4, 8, 16, 32]), bbox_head=dict( type='SharedFCBBoxHead', num_fcs=2, in_channels=256, fc_out_channels=1024, roi_feat_size=7, num_classes=2, target_means=[0., 0., 0., 0.], target_stds=[0.1, 0.1, 0.2, 0.2], reg_class_agnostic=False))

model training and testing settings

train_cfg = dict( rpn=dict( assigner=dict( type='MaxIoUAssigner', pos_iou_thr=0.7, neg_iou_thr=0.3, min_pos_iou=0.3, ignore_iof_thr=-1), sampler=dict( type='RandomSampler', num=256, pos_fraction=0.5, neg_pos_ub=-1, add_gt_as_proposals=False), allowed_border=0, pos_weight=-1, smoothl1_beta=1 / 9.0, debug=False), rcnn=dict( assigner=dict( type='MaxIoUAssigner', pos_iou_thr=0.5, neg_iou_thr=0.5, min_pos_iou=0.5, ignore_iof_thr=-1), sampler=dict( type='RandomSampler', num=512, pos_fraction=0.25, neg_pos_ub=-1, add_gt_as_proposals=True), pos_weight=-1, debug=False)) test_cfg = dict( rpn=dict( nms_across_levels=False, nms_pre=2000, nms_post=2000, max_num=2000, nms_thr=0.7, min_bbox_size=0), rcnn=dict( score_thr=0.05, nms=dict(type='nms', iou_thr=0.5), max_per_img=100)

soft-nms is also supported for rcnn testing

# e.g., nms=dict(type='soft_nms', iou_thr=0.5, min_score=0.05)

)

dataset settings

dataset_type = 'VOCDataset' data_root = 'data/VOCdevkit/' img_norm_cfg = dict( mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True) data = dict( imgs_per_gpu=2, workers_per_gpu=2, train=dict( type='RepeatDataset', # to avoid reloading datasets frequently times=3, dataset=dict( type=dataset_type, ann_file=[ data_root + 'VOC2007/ImageSets/Main/train.txt', ], img_prefix=[data_root + 'VOC2007/'], img_scale=(1000, 600), img_norm_cfg=img_norm_cfg, size_divisor=32, flip_ratio=0.5, with_mask=False, with_crowd=True, with_label=True)), val=dict( type=dataset_type, ann_file=data_root + 'VOC2007/ImageSets/Main/test.txt', img_prefix=data_root + 'VOC2007/', img_scale=(1000, 600), img_norm_cfg=img_norm_cfg, size_divisor=32, flip_ratio=0, with_mask=False, with_crowd=True, with_label=True), test=dict( type=dataset_type, ann_file=data_root + 'VOC2007/ImageSets/Main/test.txt', img_prefix=data_root + 'VOC2007/', img_scale=(1000, 600), img_norm_cfg=img_norm_cfg, size_divisor=32, flip_ratio=0, with_mask=False, with_label=False, test_mode=True))

optimizer

optimizer = dict(type='SGD', lr=0.01, momentum=0.9, weight_decay=0.0001) optimizer_config = dict(grad_clip=dict(max_norm=35, norm_type=2))

learning policy

lr_config = dict(policy='step', step=[3]) # actual epoch = 3 * 3 = 9 checkpoint_config = dict(interval=1)

yapf:disable

log_config = dict( interval=50, hooks=[ dict(type='TextLoggerHook'),

dict(type='TensorboardLoggerHook')

])

yapf:enable

runtime settings

total_epochs = 4 # actual epoch = 4 * 3 = 12 dist_params = dict(backend='nccl') log_level = 'INFO' work_dir = './work_dirs/faster_rcnn_r50_fpn_1x_voc0712' load_from = None resume_from = None workflow = [('train', 1)] `

the error：

...... /opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THCUNN/ClassNLLCriterion.cu:56: void ClassNLLCriterion_updateOutput_no_reduce_kernel(int, THCDeviceTensor<Dtype, 2, int, DefaultPtrTraits>, THCDeviceTensor<long, 1, int, DefaultPtrTraits>, THCDeviceTensor<Dtype, 1, int, DefaultPtrTraits>, Dtype , int, int) [with Dtype = float]: block: [0,0,0], thread: [613,0,0] Assertion cur_target >= 0 && cur_target < n_classes failed. /opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THCUNN/ClassNLLCriterion.cu:56: void ClassNLLCriterion_updateOutput_no_reduce_kernel(int, THCDeviceTensor<Dtype, 2, int, DefaultPtrTraits>, THCDeviceTensor<long, 1, int, DefaultPtrTraits>, THCDeviceTensor<Dtype, 1, int, DefaultPtrTraits>, Dtype , int, int) [with Dtype = float]: block: [0,0,0], thread: [614,0,0] Assertion cur_target >= 0 && cur_target < n_classes failed. /opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THCUNN/ClassNLLCriterion.cu:56: void ClassNLLCriterion_updateOutput_no_reduce_kernel(int, THCDeviceTensor<Dtype, 2, int, DefaultPtrTraits>, THCDeviceTensor<long, 1, int, DefaultPtrTraits>, THCDeviceTensor<Dtype, 1, int, DefaultPtrTraits>, Dtype , int, int) [with Dtype = float]: block: [0,0,0], thread: [615,0,0] Assertion cur_target >= 0 && cur_target < n_classes failed. /opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THCUNN/ClassNLLCriterion.cu:56: void ClassNLLCriterion_updateOutput_no_reduce_kernel(int, THCDeviceTensor<Dtype, 2, int, DefaultPtrTraits>, THCDeviceTensor<long, 1, int, DefaultPtrTraits>, THCDeviceTensor<Dtype, 1, int, DefaultPtrTraits>, Dtype , int, int) [with Dtype = float]: block: [0,0,0], thread: [616,0,0] Assertion cur_target >= 0 && cur_target < n_classes failed. /opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THCUNN/ClassNLLCriterion.cu:56: void ClassNLLCriterion_updateOutput_no_reduce_kernel(int, THCDeviceTensor<Dtype, 2, int, DefaultPtrTraits>, THCDeviceTensor<long, 1, int, DefaultPtrTraits>, THCDeviceTensor<Dtype, 1, int, DefaultPtrTraits>, Dtype , int, int) [with Dtype = float]: block: [0,0,0], thread: [617,0,0] Assertion cur_target >= 0 && cur_target < n_classes failed. /opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THCUNN/ClassNLLCriterion.cu:56: void ClassNLLCriterion_updateOutput_no_reduce_kernel(int, THCDeviceTensor<Dtype, 2, int, DefaultPtrTraits>, THCDeviceTensor<long, 1, int, DefaultPtrTraits>, THCDeviceTensor<Dtype, 1, int, DefaultPtrTraits>, Dtype , int, int) [with Dtype = float]: block: [0,0,0], thread: [618,0,0] Assertion cur_target >= 0 && cur_target < n_classes failed. /opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THCUNN/ClassNLLCriterion.cu:56: void ClassNLLCriterion_updateOutput_no_reduce_kernel(int, THCDeviceTensor<Dtype, 2, int, DefaultPtrTraits>, THCDeviceTensor<long, 1, int, DefaultPtrTraits>, THCDeviceTensor<Dtype, 1, int, DefaultPtrTraits>, Dtype , int, int) [with Dtype = float]: block: [0,0,0], thread: [619,0,0] Assertion cur_target >= 0 && cur_target < n_classes failed. /opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THCUNN/ClassNLLCriterion.cu:56: void ClassNLLCriterion_updateOutput_no_reduce_kernel(int, THCDeviceTensor<Dtype, 2, int, DefaultPtrTraits>, THCDeviceTensor<long, 1, int, DefaultPtrTraits>, THCDeviceTensor<Dtype, 1, int, DefaultPtrTraits>, Dtype , int, int) [with Dtype = float]: block: [0,0,0], thread: [620,0,0] Assertion cur_target >= 0 && cur_target < n_classes failed. /opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THCUNN/ClassNLLCriterion.cu:56: void ClassNLLCriterion_updateOutput_no_reduce_kernel(int, THCDeviceTensor<Dtype, 2, int, DefaultPtrTraits>, THCDeviceTensor<long, 1, int, DefaultPtrTraits>, THCDeviceTensor<Dtype, 1, int, DefaultPtrTraits>, Dtype , int, int) [with Dtype = float]: block: [0,0,0], thread: [621,0,0] Assertion cur_target >= 0 && cur_target < n_classes failed. /opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THCUNN/ClassNLLCriterion.cu:56: void ClassNLLCriterion_updateOutput_no_reduce_kernel(int, THCDeviceTensor<Dtype, 2, int, DefaultPtrTraits>, THCDeviceTensor<long, 1, int, DefaultPtrTraits>, THCDeviceTensor<Dtype, 1, int, DefaultPtrTraits>, Dtype , int, int) [with Dtype = float]: block: [0,0,0], thread: [622,0,0] Assertion cur_target >= 0 && cur_target < n_classes failed. /opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THCUNN/ClassNLLCriterion.cu:56: void ClassNLLCriterion_updateOutput_no_reduce_kernel(int, THCDeviceTensor<Dtype, 2, int, DefaultPtrTraits>, THCDeviceTensor<long, 1, int, DefaultPtrTraits>, THCDeviceTensor<Dtype, 1, int, DefaultPtrTraits>, Dtype , int, int) [with Dtype = float]: block: [0,0,0], thread: [623,0,0] Assertion cur_target >= 0 && cur_target < n_classes failed. /opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THCUNN/ClassNLLCriterion.cu:56: void ClassNLLCriterion_updateOutput_no_reduce_kernel(int, THCDeviceTensor<Dtype, 2, int, DefaultPtrTraits>, THCDeviceTensor<long, 1, int, DefaultPtrTraits>, THCDeviceTensor<Dtype, 1, int, DefaultPtrTraits>, Dtype , int, int) [with Dtype = float]: block: [0,0,0], thread: [624,0,0] Assertion cur_target >= 0 && cur_target < n_classes failed. /opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THCUNN/ClassNLLCriterion.cu:56: void ClassNLLCriterion_updateOutput_no_reduce_kernel(int, THCDeviceTensor<Dtype, 2, int, DefaultPtrTraits>, THCDeviceTensor<long, 1, int, DefaultPtrTraits>, THCDeviceTensor<Dtype, 1, int, DefaultPtrTraits>, Dtype , int, int) [with Dtype = float]: block: [0,0,0], thread: [625,0,0] Assertion cur_target >= 0 && cur_target < n_classes failed. /opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THCUNN/ClassNLLCriterion.cu:56: void ClassNLLCriterion_updateOutput_no_reduce_kernel(int, THCDeviceTensor<Dtype, 2, int, DefaultPtrTraits>, THCDeviceTensor<long, 1, int, DefaultPtrTraits>, THCDeviceTensor<Dtype, 1, int, DefaultPtrTraits>, Dtype , int, int) [with Dtype = float]: block: [0,0,0], thread: [626,0,0] Assertion cur_target >= 0 && cur_target < n_classes failed. /opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THCUNN/ClassNLLCriterion.cu:56: void ClassNLLCriterion_updateOutput_no_reduce_kernel(int, THCDeviceTensor<Dtype, 2, int, DefaultPtrTraits>, THCDeviceTensor<long, 1, int, DefaultPtrTraits>, THCDeviceTensor<Dtype, 1, int, DefaultPtrTraits>, Dtype , int, int) [with Dtype = float]: block: [0,0,0], thread: [627,0,0] Assertion cur_target >= 0 && cur_target < n_classes failed. /opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THCUNN/ClassNLLCriterion.cu:56: void ClassNLLCriterion_updateOutput_no_reduce_kernel(int, THCDeviceTensor<Dtype, 2, int, DefaultPtrTraits>, THCDeviceTensor<long, 1, int, DefaultPtrTraits>, THCDeviceTensor<Dtype, 1, int, DefaultPtrTraits>, Dtype , int, int) [with Dtype = float]: block: [0,0,0], thread: [628,0,0] Assertion cur_target >= 0 && cur_target < n_classes failed. /opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THCUNN/ClassNLLCriterion.cu:56: void ClassNLLCriterion_updateOutput_no_reduce_kernel(int, THCDeviceTensor<Dtype, 2, int, DefaultPtrTraits>, THCDeviceTensor<long, 1, int, DefaultPtrTraits>, THCDeviceTensor<Dtype, 1, int, DefaultPtrTraits>, Dtype , int, int) [with Dtype = float]: block: [0,0,0], thread: [629,0,0] Assertion cur_target >= 0 && cur_target < n_classes failed. /opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THCUNN/ClassNLLCriterion.cu:56: void ClassNLLCriterion_updateOutput_no_reduce_kernel(int, THCDeviceTensor<Dtype, 2, int, DefaultPtrTraits>, THCDeviceTensor<long, 1, int, DefaultPtrTraits>, THCDeviceTensor<Dtype, 1, int, DefaultPtrTraits>, Dtype , int, int) [with Dtype = float]: block: [0,0,0], thread: [630,0,0] Assertion cur_target >= 0 && cur_target < n_classes failed. /opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THCUNN/ClassNLLCriterion.cu:56: void ClassNLLCriterion_updateOutput_no_reduce_kernel(int, THCDeviceTensor<Dtype, 2, int, DefaultPtrTraits>, THCDeviceTensor<long, 1, int, DefaultPtrTraits>, THCDeviceTensor<Dtype, 1, int, DefaultPtrTraits>, Dtype , int, int) [with Dtype = float]: block: [0,0,0], thread: [631,0,0] Assertion cur_target >= 0 && cur_target < n_classes failed. /opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THCUNN/ClassNLLCriterion.cu:56: void ClassNLLCriterion_updateOutput_no_reduce_kernel(int, THCDeviceTensor<Dtype, 2, int, DefaultPtrTraits>, THCDeviceTensor<long, 1, int, DefaultPtrTraits>, THCDeviceTensor<Dtype, 1, int, DefaultPtrTraits>, Dtype , int, int) [with Dtype = float]: block: [0,0,0], thread: [632,0,0] Assertion cur_target >= 0 && cur_target < n_classes failed. /opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THCUNN/ClassNLLCriterion.cu:56: void ClassNLLCriterion_updateOutput_no_reduce_kernel(int, THCDeviceTensor<Dtype, 2, int, DefaultPtrTraits>, THCDeviceTensor<long, 1, int, DefaultPtrTraits>, THCDeviceTensor<Dtype, 1, int, DefaultPtrTraits>, Dtype , int, int) [with Dtype = float]: block: [0,0,0], thread: [633,0,0] Assertion cur_target >= 0 && cur_target < n_classes failed. /opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THCUNN/ClassNLLCriterion.cu:56: void ClassNLLCriterion_updateOutput_no_reduce_kernel(int, THCDeviceTensor<Dtype, 2, int, DefaultPtrTraits>, THCDeviceTensor<long, 1, int, DefaultPtrTraits>, THCDeviceTensor<Dtype, 1, int, DefaultPtrTraits>, Dtype , int, int) [with Dtype = float]: block: [0,0,0], thread: [634,0,0] Assertion cur_target >= 0 && cur_target < n_classes failed. /opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THCUNN/ClassNLLCriterion.cu:56: void ClassNLLCriterion_updateOutput_no_reduce_kernel(int, THCDeviceTensor<Dtype, 2, int, DefaultPtrTraits>, THCDeviceTensor<long, 1, int, DefaultPtrTraits>, THCDeviceTensor<Dtype, 1, int, DefaultPtrTraits>, Dtype , int, int) [with Dtype = float]: block: [0,0,0], thread: [635,0,0] Assertion cur_target >= 0 && cur_target < n_classes failed. /opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THCUNN/ClassNLLCriterion.cu:56: void ClassNLLCriterion_updateOutput_no_reduce_kernel(int, THCDeviceTensor<Dtype, 2, int, DefaultPtrTraits>, THCDeviceTensor<long, 1, int, DefaultPtrTraits>, THCDeviceTensor<Dtype, 1, int, DefaultPtrTraits>, Dtype , int, int) [with Dtype = float]: block: [0,0,0], thread: [636,0,0] Assertion cur_target >= 0 && cur_target < n_classes failed. /opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THCUNN/ClassNLLCriterion.cu:56: void ClassNLLCriterion_updateOutput_no_reduce_kernel(int, THCDeviceTensor<Dtype, 2, int, DefaultPtrTraits>, THCDeviceTensor<long, 1, int, DefaultPtrTraits>, THCDeviceTensor<Dtype, 1, int, DefaultPtrTraits>, Dtype , int, int) [with Dtype = float]: block: [0,0,0], thread: [637,0,0] Assertion cur_target >= 0 && cur_target < n_classes failed. /opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THCUNN/ClassNLLCriterion.cu:56: void ClassNLLCriterion_updateOutput_no_reduce_kernel(int, THCDeviceTensor<Dtype, 2, int, DefaultPtrTraits>, THCDeviceTensor<long, 1, int, DefaultPtrTraits>, THCDeviceTensor<Dtype, 1, int, DefaultPtrTraits>, Dtype , int, int) [with Dtype = float]: block: [0,0,0], thread: [638,0,0] Assertion cur_target >= 0 && cur_target < n_classes failed. /opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THCUNN/ClassNLLCriterion.cu:56: void ClassNLLCriterion_updateOutput_no_reduce_kernel(int, THCDeviceTensor<Dtype, 2, int, DefaultPtrTraits>, THCDeviceTensor<long, 1, int, DefaultPtrTraits>, THCDeviceTensor<Dtype, 1, int, DefaultPtrTraits>, Dtype *, int, int) [with Dtype = float]: block: [0,0,0], thread: [639,0,0] Assertion cur_target >= 0 && cur_target < n_classes failed. terminate called without an active exception Traceback (most recent call last): File "./tools/train.py", line 90, in main() File "./tools/train.py", line 86, in main logger=logger) File "/root/anaconda3/envs/pth1/lib/python3.6/site-packages/mmdet-0.6rc0+254077b-py3.6.egg/mmdet/apis/train.py", line 57, in train_detector _dist_train(model, dataset, cfg, validate=validate) File "/root/anaconda3/envs/pth1/lib/python3.6/site-packages/mmdet-0.6rc0+254077b-py3.6.egg/mmdet/apis/train.py", line 96, in _dist_train runner.run(data_loaders, cfg.workflow, cfg.total_epochs) File "/root/anaconda3/envs/pth1/lib/python3.6/site-packages/mmcv/runner/runner.py", line 355, in run epoch_runner(data_loaders[i], kwargs) File "/root/anaconda3/envs/pth1/lib/python3.6/site-packages/mmcv/runner/runner.py", line 261, in train self.model, data_batch, train_mode=True, kwargs) File "/root/anaconda3/envs/pth1/lib/python3.6/site-packages/mmdet-0.6rc0+254077b-py3.6.egg/mmdet/apis/train.py", line 38, in batch_processor loss, log_vars = parse_losses(losses) File "/root/anaconda3/envs/pth1/lib/python3.6/site-packages/mmdet-0.6rc0+254077b-py3.6.egg/mmdet/apis/train.py", line 22, in parse_losses log_vars[loss_name] = sum(_loss.mean() for _loss in loss_value) File "/root/anaconda3/envs/pth1/lib/python3.6/site-packages/mmdet-0.6rc0+254077b-py3.6.egg/mmdet/apis/train.py", line 22, in log_vars[loss_name] = sum(_loss.mean() for _loss in loss_value) RuntimeError: CUDA error: device-side assert triggered terminate called without an active exception terminate called without an active exception

hellock commented 5 years ago

num_classes should match the actual classes of your dataset and it cannot be arbitrary.

Bigwode commented 5 years ago

num_classes should be be equal or greater than the number of classes of your dataset add one(background).

kekedan commented 5 years ago

num_classes should match the actual classes of your dataset and it cannot be arbitrary.

thanks~

ghost commented 5 years ago

Means that changing the name in the CLASSES inside the CocoDataset class and value of num_classes at configs/script.py and mmdet/models/bbox_heads/bbox_head.py. If i have only 1 class, I should put 2 for num_classes right?

Just need to reconfirm. Thanks for helping!

njustymk commented 5 years ago

Means that changing the name in the CLASSES inside the CocoDataset class and value of num_classes at configs/script.py and mmdet/models/bbox_heads/bbox_head.py. If i have only 1 class, I should put 2 for num_classes right?

Just need to reconfirm. Thanks for helping!

i have the same isue ,have you solved it? i modefy 1,CLASSES in CocoDataset 2,num_classes in confige file

but i get the same ploblem with you,can you tell me how to solve it?thank you !