search

joe-siyuan-qiao / DetectoRS

DetectoRS: Detecting Objects with Recursive Feature Pyramid and Switchable Atrous Convolution
Apache License 2.0
1.14k stars 176 forks source link

RuntimeError: copy_if failed to synchronize: device-side assert triggered terminate called after throwing an instance of 'c10::Error' #13

Closed MotiBaadror closed 4 years ago

MotiBaadror commented 4 years ago 
sys.platform: linux
Python: 3.6.9 (default, Apr 18 2020, 01:56:04) [GCC 8.4.0]
CUDA available: True
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 10.1, V10.1.243
GPU 0: Tesla K80
GCC: gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
PyTorch: 1.4.0+cu100
PyTorch compiling details: PyTorch built with:
  - GCC 7.3
  - Intel(R) Math Kernel Library Version 2019.0.4 Product Build 20190411 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v0.21.1 (Git Hash 7d2fd500bc78936d1d648ca713b901012f470dbc)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - NNPACK is enabled
  - CUDA Runtime 10.0
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_37,code=compute_37
  - CuDNN 7.6.3
  - Magma 2.5.1
  - Build settings: BLAS=MKL, BUILD_NAMEDTENSOR=OFF, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -fopenmp -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Wno-stringop-overflow, DISABLE_NUMA=1, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_STATIC_DISPATCH=OFF, 

TorchVision: 0.5.0+cu100
OpenCV: 4.1.2
MMCV: 0.5.9
MMDetection: 1.1.0+bccc528
MMDetection Compiler: GCC 7.5
MMDetection CUDA Compiler: 10.1
------------------------------------------------------------

2020-06-08 08:59:36,404 - mmdet - INFO - Distributed training: False
2020-06-08 08:59:36,404 - mmdet - INFO - Config:
/content/drive/My Drive/DetectoRS/configs/DetectoRS/DetectoRS_mstrain_400_1200_x101_32x4d_40e.py
conv_cfg = dict(type='ConvAWS')
# model settings
model = dict(
    type='RecursiveFeaturePyramid',
    rfp_steps=2,
    rfp_sharing=False,
    stage_with_rfp=(False, True, True, True),
    num_stages=3,
    pretrained='open-mmlab://resnext101_32x4d',
    interleaved=True,
    mask_info_flow=True,
    backbone=dict(
        type='ResNeXt',
        depth=101,
        groups=32,
        base_width=4,
        num_stages=4,
        out_indices=(0, 1, 2, 3),
        frozen_stages=1,
        conv_cfg=conv_cfg,
        sac=dict(type='SAC', use_deform=True),
        stage_with_sac=(False, True, True, True),
        norm_cfg=dict(type='BN', requires_grad=True),
        style='pytorch'),
    neck=dict(
        type='FPN',
        in_channels=[256, 512, 1024, 2048],
        out_channels=256,
        num_outs=5),
    rpn_head=dict(
        type='RPNHead',
        in_channels=256,
        feat_channels=256,
        anchor_scales=[8],
        anchor_ratios=[0.5, 1.0, 2.0],
        anchor_strides=[4, 8, 16, 32, 64],
        target_means=[.0, .0, .0, .0],
        target_stds=[1.0, 1.0, 1.0, 1.0],
        loss_cls=dict(
            type='CrossEntropyLoss', use_sigmoid=True, loss_weight=1.0),
        loss_bbox=dict(type='SmoothL1Loss', beta=1.0 / 9.0, loss_weight=1.0)),
    bbox_roi_extractor=dict(
        type='SingleRoIExtractor',
        roi_layer=dict(type='RoIAlign', out_size=7, sample_num=2),
        out_channels=256,
        featmap_strides=[4, 8, 16, 32]),
    bbox_head=[
        dict(
            type='SharedFCBBoxHead',
            num_fcs=2,
            in_channels=256,
            fc_out_channels=1024,
            roi_feat_size=7,
            num_classes=1,
            target_means=[0., 0., 0., 0.],
            target_stds=[0.1, 0.1, 0.2, 0.2],
            reg_class_agnostic=True,
            loss_cls=dict(
                type='CrossEntropyLoss', use_sigmoid=False, loss_weight=1.0),
            loss_bbox=dict(type='SmoothL1Loss', beta=1.0, loss_weight=1.0)),
        dict(
            type='SharedFCBBoxHead',
            num_fcs=2,
            in_channels=256,
            fc_out_channels=1024,
            roi_feat_size=7,
            num_classes=2,
            target_means=[0., 0., 0., 0.],
            target_stds=[0.05, 0.05, 0.1, 0.1],
            reg_class_agnostic=True,
            loss_cls=dict(
                type='CrossEntropyLoss', use_sigmoid=False, loss_weight=1.0),
            loss_bbox=dict(type='SmoothL1Loss', beta=1.0, loss_weight=1.0)),
        dict(
            type='SharedFCBBoxHead',
            num_fcs=2,
            in_channels=256,
            fc_out_channels=1024,
            roi_feat_size=7,
            num_classes=2,
            target_means=[0., 0., 0., 0.],
            target_stds=[0.033, 0.033, 0.067, 0.067],
            reg_class_agnostic=True,
            loss_cls=dict(
                type='CrossEntropyLoss', use_sigmoid=False, loss_weight=1.0),
            loss_bbox=dict(type='SmoothL1Loss', beta=1.0, loss_weight=1.0))
    ],
    mask_roi_extractor=dict(
        type='SingleRoIExtractor',
        roi_layer=dict(type='RoIAlign', out_size=14, sample_num=2),
        out_channels=256,
        featmap_strides=[4, 8, 16, 32]),
    mask_head=[
        dict(
            type='HTCMaskHead',
            with_conv_res=False,
            num_convs=4,
            in_channels=256,
            conv_out_channels=256,
            num_classes=1,
            loss_mask=dict(
                type='CrossEntropyLoss', use_mask=True, loss_weight=1.0)),
        dict(
            type='HTCMaskHead',
            num_convs=4,
            in_channels=256,
            conv_out_channels=256,
            num_classes=1,
            loss_mask=dict(
                type='CrossEntropyLoss', use_mask=True, loss_weight=1.0)),
        dict(
            type='HTCMaskHead',
            num_convs=4,
            in_channels=256,
            conv_out_channels=256,
            num_classes=1,
            loss_mask=dict(
                type='CrossEntropyLoss', use_mask=True, loss_weight=1.0))
    ],
    semantic_roi_extractor=dict(
        type='SingleRoIExtractor',
        roi_layer=dict(type='RoIAlign', out_size=14, sample_num=2),
        out_channels=256,
        featmap_strides=[8]),
    semantic_head=dict(
        type='FusedSemanticHead',
        num_ins=5,
        fusion_level=1,
        num_convs=4,
        in_channels=256,
        conv_out_channels=256,
        num_classes=1,
        ignore_label=255,
        loss_weight=0.2))
# model training and testing settings
train_cfg = dict(
    rpn=dict(
        assigner=dict(
            type='MaxIoUAssigner',
            pos_iou_thr=0.7,
            neg_iou_thr=0.3,
            min_pos_iou=0.3,
            ignore_iof_thr=-1),
        sampler=dict(
            type='RandomSampler',
            num=256,
            pos_fraction=0.5,
            neg_pos_ub=-1,
            add_gt_as_proposals=False),
        allowed_border=0,
        pos_weight=-1,
        debug=False),
    rpn_proposal=dict(
        nms_across_levels=False,
        nms_pre=2000,
        nms_post=2000,
        max_num=2000,
        nms_thr=0.7,
        min_bbox_size=0),
    rcnn=[
        dict(
            assigner=dict(
                type='MaxIoUAssigner',
                pos_iou_thr=0.5,
                neg_iou_thr=0.5,
                min_pos_iou=0.5,
                ignore_iof_thr=-1),
            sampler=dict(
                type='RandomSampler',
                num=512,
                pos_fraction=0.25,
                neg_pos_ub=-1,
                add_gt_as_proposals=True),
            mask_size=28,
            pos_weight=-1,
            debug=False),
        dict(
            assigner=dict(
                type='MaxIoUAssigner',
                pos_iou_thr=0.6,
                neg_iou_thr=0.6,
                min_pos_iou=0.6,
                ignore_iof_thr=-1),
            sampler=dict(
                type='RandomSampler',
                num=512,
                pos_fraction=0.25,
                neg_pos_ub=-1,
                add_gt_as_proposals=True),
            mask_size=28,
            pos_weight=-1,
            debug=False),
        dict(
            assigner=dict(
                type='MaxIoUAssigner',
                pos_iou_thr=0.7,
                neg_iou_thr=0.7,
                min_pos_iou=0.7,
                ignore_iof_thr=-1),
            sampler=dict(
                type='RandomSampler',
                num=512,
                pos_fraction=0.25,
                neg_pos_ub=-1,
                add_gt_as_proposals=True),
            mask_size=28,
            pos_weight=-1,
            debug=False)
    ],
    stage_loss_weights=[1, 0.5, 0.25])
test_cfg = dict(
    rpn=dict(
        nms_across_levels=False,
        nms_pre=1000,
        nms_post=1000,
        max_num=1000,
        nms_thr=0.7,
        min_bbox_size=0),
    rcnn=dict(
        score_thr=0.001,
        nms=dict(type='soft_nms', iou_thr=0.5),
        max_per_img=100,
        mask_thr_binary=0.5))
# dataset settings
dataset_type = 'CocoDataset'
data_root = '/content/drive/My Drive/collab/data/'
img_norm_cfg = dict(
    mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True)
train_pipeline = [
    dict(type='LoadImageFromFile'),
    dict(
        type='LoadAnnotations', with_bbox=True, with_mask=True, with_seg=True),
    dict(
        type='Resize',
        img_scale=[(1333, 400), (1333, 1200)],
        multiscale_mode='range',
        keep_ratio=True),
    dict(type='RandomFlip', flip_ratio=0.5),
    dict(type='Normalize', **img_norm_cfg),
    dict(type='Pad', size_divisor=32),
    dict(type='SegRescale', scale_factor=1 / 8),
    dict(type='DefaultFormatBundle'),
    dict(
        type='Collect',
        keys=['img', 'gt_bboxes', 'gt_labels', 'gt_masks', 'gt_semantic_seg']),
]
test_pipeline = [
    dict(type='LoadImageFromFile'),
    dict(
        type='MultiScaleFlipAug',
        img_scale=(1333, 800),
        flip=False,
        transforms=[
            dict(type='Resize', keep_ratio=True),
            dict(type='RandomFlip', flip_ratio=0.5),
            dict(type='Normalize', **img_norm_cfg),
            dict(type='Pad', size_divisor=32),
            dict(type='ImageToTensor', keys=['img']),
            dict(type='Collect', keys=['img']),
        ])
]
data = dict(
    imgs_per_gpu=1,
    workers_per_gpu=1,
    train=dict(
        type=dataset_type,
        ann_file=data_root + 'annotations/instances_train2017.json',
        img_prefix=data_root + 'train2017/',
        seg_prefix=data_root + 'train2017/',
        pipeline=train_pipeline),
    val=dict(
        type=dataset_type,
        ann_file=data_root + 'annotations/instances_val2017.json',
        img_prefix=data_root + 'val2017/',
        pipeline=test_pipeline),
    test=dict(
        type=dataset_type,
        ann_file=data_root + 'annotations/instances_val2017.json',
        img_prefix=data_root + 'val2017/',
        pipeline=test_pipeline))
evaluation = dict(interval=1, metric=['bbox', 'segm'])
# optimizer
optimizer = dict(type='SGD', lr=0.02, momentum=0.9, weight_decay=0.0001)
optimizer_config = dict(grad_clip=dict(max_norm=35, norm_type=2))
# learning policy
lr_config = dict(
    policy='step',
    warmup='linear',
    warmup_iters=500,
    warmup_ratio=1.0 / 3,
    step=[36, 39])
checkpoint_config = dict(interval=1)
# yapf:disable
log_config = dict(
    interval=50,
    hooks=[
        dict(type='TextLoggerHook'),
        # dict(type='TensorboardLoggerHook')
    ])
# yapf:enable
# runtime settings
total_epochs = 40
dist_params = dict(backend='nccl')
log_level = 'INFO'
work_dir = './work_dirs/DetectoRS_mstrain_400_1200_x101_32x4d_40e.py'
load_from = None
resume_from = None
workflow = [('train', 1)]

2020-06-08 08:59:39,001 - mmdet - INFO - load model from: open-mmlab://resnext101_32x4d
2020-06-08 08:59:39,627 - mmdet - WARNING - The model and loaded state dict do not match exactly

missing keys in source state_dict: conv1.weight_gamma, conv1.weight_beta, layer1.0.conv1.weight_gamma, layer1.0.conv1.weight_beta, layer1.0.conv2.weight_gamma, layer1.0.conv2.weight_beta, layer1.0.conv3.weight_gamma, layer1.0.conv3.weight_beta, layer1.0.downsample.0.weight_gamma, layer1.0.downsample.0.weight_beta, layer1.1.conv1.weight_gamma, layer1.1.conv1.weight_beta, layer1.1.conv2.weight_gamma, layer1.1.conv2.weight_beta, layer1.1.conv3.weight_gamma, layer1.1.conv3.weight_beta, layer1.2.conv1.weight_gamma, layer1.2.conv1.weight_beta, layer1.2.conv2.weight_gamma, layer1.2.conv2.weight_beta, layer1.2.conv3.weight_gamma, layer1.2.conv3.weight_beta, layer2.0.conv1.weight_gamma, layer2.0.conv1.weight_beta, layer2.0.conv2.weight_diff, layer2.0.conv2.weight_gamma, layer2.0.conv2.weight_beta, layer2.0.conv2.switch.weight, layer2.0.conv2.switch.bias, layer2.0.conv2.pre_context.weight, layer2.0.conv2.pre_context.bias, layer2.0.conv2.post_context.weight, layer2.0.conv2.post_context.bias, layer2.0.conv2.offset_s.weight, layer2.0.conv2.offset_s.bias, layer2.0.conv2.offset_l.weight, layer2.0.conv2.offset_l.bias, layer2.0.conv3.weight_gamma, layer2.0.conv3.weight_beta, layer2.0.downsample.0.weight_gamma, layer2.0.downsample.0.weight_beta, layer2.1.conv1.weight_gamma, layer2.1.conv1.weight_beta, layer2.1.conv2.weight_diff, layer2.1.conv2.weight_gamma, layer2.1.conv2.weight_beta, layer2.1.conv2.switch.weight, layer2.1.conv2.switch.bias, layer2.1.conv2.pre_context.weight, layer2.1.conv2.pre_context.bias, layer2.1.conv2.post_context.weight, layer2.1.conv2.post_context.bias, layer2.1.conv2.offset_s.weight, layer2.1.conv2.offset_s.bias, layer2.1.conv2.offset_l.weight, layer2.1.conv2.offset_l.bias, layer2.1.conv3.weight_gamma, layer2.1.conv3.weight_beta, layer2.2.conv1.weight_gamma, layer2.2.conv1.weight_beta, layer2.2.conv2.weight_diff, layer2.2.conv2.weight_gamma, layer2.2.conv2.weight_beta, layer2.2.conv2.switch.weight, layer2.2.conv2.switch.bias, layer2.2.conv2.pre_context.weight, layer2.2.conv2.pre_context.bias, layer2.2.conv2.post_context.weight, layer2.2.conv2.post_context.bias, layer2.2.conv2.offset_s.weight, layer2.2.conv2.offset_s.bias, layer2.2.conv2.offset_l.weight, layer2.2.conv2.offset_l.bias, layer2.2.conv3.weight_gamma, layer2.2.conv3.weight_beta, layer2.3.conv1.weight_gamma, layer2.3.conv1.weight_beta, layer2.3.conv2.weight_diff, layer2.3.conv2.weight_gamma, layer2.3.conv2.weight_beta, layer2.3.conv2.switch.weight, layer2.3.conv2.switch.bias, layer2.3.conv2.pre_context.weight, layer2.3.conv2.pre_context.bias, layer2.3.conv2.post_context.weight, layer2.3.conv2.post_context.bias, layer2.3.conv2.offset_s.weight, layer2.3.conv2.offset_s.bias, layer2.3.conv2.offset_l.weight, layer2.3.conv2.offset_l.bias, layer2.3.conv3.weight_gamma, layer2.3.conv3.weight_beta, layer3.0.conv1.weight_gamma, layer3.0.conv1.weight_beta, layer3.0.conv2.weight_diff, layer3.0.conv2.weight_gamma, layer3.0.conv2.weight_beta, layer3.0.conv2.switch.weight, layer3.0.conv2.switch.bias, layer3.0.conv2.pre_context.weight, layer3.0.conv2.pre_context.bias, layer3.0.conv2.post_context.weight, layer3.0.conv2.post_context.bias, layer3.0.conv2.offset_s.weight, layer3.0.conv2.offset_s.bias, layer3.0.conv2.offset_l.weight, layer3.0.conv2.offset_l.bias, layer3.0.conv3.weight_gamma, layer3.0.conv3.weight_beta, layer3.0.downsample.0.weight_gamma, layer3.0.downsample.0.weight_beta, layer3.1.conv1.weight_gamma, layer3.1.conv1.weight_beta, layer3.1.conv2.weight_diff, layer3.1.conv2.weight_gamma, layer3.1.conv2.weight_beta, layer3.1.conv2.switch.weight, layer3.1.conv2.switch.bias, layer3.1.conv2.pre_context.weight, layer3.1.conv2.pre_context.bias, layer3.1.conv2.post_context.weight, layer3.1.conv2.post_context.bias, layer3.1.conv2.offset_s.weight, layer3.1.conv2.offset_s.bias, layer3.1.conv2.offset_l.weight, layer3.1.conv2.offset_l.bias, layer3.1.conv3.weight_gamma, layer3.1.conv3.weight_beta, layer3.2.conv1.weight_gamma, layer3.2.conv1.weight_beta, layer3.2.conv2.weight_diff, layer3.2.conv2.weight_gamma, layer3.2.conv2.weight_beta, layer3.2.conv2.switch.weight, layer3.2.conv2.switch.bias, layer3.2.conv2.pre_context.weight, layer3.2.conv2.pre_context.bias, layer3.2.conv2.post_context.weight, layer3.2.conv2.post_context.bias, layer3.2.conv2.offset_s.weight, layer3.2.conv2.offset_s.bias, layer3.2.conv2.offset_l.weight, layer3.2.conv2.offset_l.bias, layer3.2.conv3.weight_gamma, layer3.2.conv3.weight_beta, layer3.3.conv1.weight_gamma, layer3.3.conv1.weight_beta, layer3.3.conv2.weight_diff, layer3.3.conv2.weight_gamma, layer3.3.conv2.weight_beta, layer3.3.conv2.switch.weight, layer3.3.conv2.switch.bias, layer3.3.conv2.pre_context.weight, layer3.3.conv2.pre_context.bias, layer3.3.conv2.post_context.weight, layer3.3.conv2.post_context.bias, layer3.3.conv2.offset_s.weight, layer3.3.conv2.offset_s.bias, layer3.3.conv2.offset_l.weight, layer3.3.conv2.offset_l.bias, layer3.3.conv3.weight_gamma, layer3.3.conv3.weight_beta, layer3.4.conv1.weight_gamma, layer3.4.conv1.weight_beta, layer3.4.conv2.weight_diff, layer3.4.conv2.weight_gamma, layer3.4.conv2.weight_beta, layer3.4.conv2.switch.weight, layer3.4.conv2.switch.bias, layer3.4.conv2.pre_context.weight, layer3.4.conv2.pre_context.bias, layer3.4.conv2.post_context.weight, layer3.4.conv2.post_context.bias, layer3.4.conv2.offset_s.weight, layer3.4.conv2.offset_s.bias, layer3.4.conv2.offset_l.weight, layer3.4.conv2.offset_l.bias, layer3.4.conv3.weight_gamma, layer3.4.conv3.weight_beta, layer3.5.conv1.weight_gamma, layer3.5.conv1.weight_beta, layer3.5.conv2.weight_diff, layer3.5.conv2.weight_gamma, layer3.5.conv2.weight_beta, layer3.5.conv2.switch.weight, layer3.5.conv2.switch.bias, layer3.5.conv2.pre_context.weight, layer3.5.conv2.pre_context.bias, layer3.5.conv2.post_context.weight, layer3.5.conv2.post_context.bias, layer3.5.conv2.offset_s.weight, layer3.5.conv2.offset_s.bias, layer3.5.conv2.offset_l.weight, layer3.5.conv2.offset_l.bias, layer3.5.conv3.weight_gamma, layer3.5.conv3.weight_beta, layer3.6.conv1.weight_gamma, layer3.6.conv1.weight_beta, layer3.6.conv2.weight_diff, layer3.6.conv2.weight_gamma, layer3.6.conv2.weight_beta, layer3.6.conv2.switch.weight, layer3.6.conv2.switch.bias, layer3.6.conv2.pre_context.weight, layer3.6.conv2.pre_context.bias, layer3.6.conv2.post_context.weight, layer3.6.conv2.post_context.bias, layer3.6.conv2.offset_s.weight, layer3.6.conv2.offset_s.bias, layer3.6.conv2.offset_l.weight, layer3.6.conv2.offset_l.bias, layer3.6.conv3.weight_gamma, layer3.6.conv3.weight_beta, layer3.7.conv1.weight_gamma, layer3.7.conv1.weight_beta, layer3.7.conv2.weight_diff, layer3.7.conv2.weight_gamma, layer3.7.conv2.weight_beta, layer3.7.conv2.switch.weight, layer3.7.conv2.switch.bias, layer3.7.conv2.pre_context.weight, layer3.7.conv2.pre_context.bias, layer3.7.conv2.post_context.weight, layer3.7.conv2.post_context.bias, layer3.7.conv2.offset_s.weight, layer3.7.conv2.offset_s.bias, layer3.7.conv2.offset_l.weight, layer3.7.conv2.offset_l.bias, layer3.7.conv3.weight_gamma, layer3.7.conv3.weight_beta, layer3.8.conv1.weight_gamma, layer3.8.conv1.weight_beta, layer3.8.conv2.weight_diff, layer3.8.conv2.weight_gamma, layer3.8.conv2.weight_beta, layer3.8.conv2.switch.weight, layer3.8.conv2.switch.bias, layer3.8.conv2.pre_context.weight, layer3.8.conv2.pre_context.bias, layer3.8.conv2.post_context.weight, layer3.8.conv2.post_context.bias, layer3.8.conv2.offset_s.weight, layer3.8.conv2.offset_s.bias, layer3.8.conv2.offset_l.weight, layer3.8.conv2.offset_l.bias, layer3.8.conv3.weight_gamma, layer3.8.conv3.weight_beta, layer3.9.conv1.weight_gamma, layer3.9.conv1.weight_beta, layer3.9.conv2.weight_diff, layer3.9.conv2.weight_gamma, layer3.9.conv2.weight_beta, layer3.9.conv2.switch.weight, layer3.9.conv2.switch.bias, layer3.9.conv2.pre_context.weight, layer3.9.conv2.pre_context.bias, layer3.9.conv2.post_context.weight, layer3.9.conv2.post_context.bias, layer3.9.conv2.offset_s.weight, layer3.9.conv2.offset_s.bias, layer3.9.conv2.offset_l.weight, layer3.9.conv2.offset_l.bias, layer3.9.conv3.weight_gamma, layer3.9.conv3.weight_beta, layer3.10.conv1.weight_gamma, layer3.10.conv1.weight_beta, layer3.10.conv2.weight_diff, layer3.10.conv2.weight_gamma, layer3.10.conv2.weight_beta, layer3.10.conv2.switch.weight, layer3.10.conv2.switch.bias, layer3.10.conv2.pre_context.weight, layer3.10.conv2.pre_context.bias, layer3.10.conv2.post_context.weight, layer3.10.conv2.post_context.bias, layer3.10.conv2.offset_s.weight, layer3.10.conv2.offset_s.bias, layer3.10.conv2.offset_l.weight, layer3.10.conv2.offset_l.bias, layer3.10.conv3.weight_gamma, layer3.10.conv3.weight_beta, layer3.11.conv1.weight_gamma, layer3.11.conv1.weight_beta, layer3.11.conv2.weight_diff, layer3.11.conv2.weight_gamma, layer3.11.conv2.weight_beta, layer3.11.conv2.switch.weight, layer3.11.conv2.switch.bias, layer3.11.conv2.pre_context.weight, layer3.11.conv2.pre_context.bias, layer3.11.conv2.post_context.weight, layer3.11.conv2.post_context.bias, layer3.11.conv2.offset_s.weight, layer3.11.conv2.offset_s.bias, layer3.11.conv2.offset_l.weight, layer3.11.conv2.offset_l.bias, layer3.11.conv3.weight_gamma, layer3.11.conv3.weight_beta, layer3.12.conv1.weight_gamma, layer3.12.conv1.weight_beta, layer3.12.conv2.weight_diff, layer3.12.conv2.weight_gamma, layer3.12.conv2.weight_beta, layer3.12.conv2.switch.weight, layer3.12.conv2.switch.bias, layer3.12.conv2.pre_context.weight, layer3.12.conv2.pre_context.bias, layer3.12.conv2.post_context.weight, layer3.12.conv2.post_context.bias, layer3.12.conv2.offset_s.weight, layer3.12.conv2.offset_s.bias, layer3.12.conv2.offset_l.weight, layer3.12.conv2.offset_l.bias, layer3.12.conv3.weight_gamma, layer3.12.conv3.weight_beta, layer3.13.conv1.weight_gamma, layer3.13.conv1.weight_beta, layer3.13.conv2.weight_diff, layer3.13.conv2.weight_gamma, layer3.13.conv2.weight_beta, layer3.13.conv2.switch.weight, layer3.13.conv2.switch.bias, layer3.13.conv2.pre_context.weight, layer3.13.conv2.pre_context.bias, layer3.13.conv2.post_context.weight, layer3.13.conv2.post_context.bias, layer3.13.conv2.offset_s.weight, layer3.13.conv2.offset_s.bias, layer3.13.conv2.offset_l.weight, layer3.13.conv2.offset_l.bias, layer3.13.conv3.weight_gamma, layer3.13.conv3.weight_beta, layer3.14.conv1.weight_gamma, layer3.14.conv1.weight_beta, layer3.14.conv2.weight_diff, layer3.14.conv2.weight_gamma, layer3.14.conv2.weight_beta, layer3.14.conv2.switch.weight, layer3.14.conv2.switch.bias, layer3.14.conv2.pre_context.weight, layer3.14.conv2.pre_context.bias, layer3.14.conv2.post_context.weight, layer3.14.conv2.post_context.bias, layer3.14.conv2.offset_s.weight, layer3.14.conv2.offset_s.bias, layer3.14.conv2.offset_l.weight, layer3.14.conv2.offset_l.bias, layer3.14.conv3.weight_gamma, layer3.14.conv3.weight_beta, layer3.15.conv1.weight_gamma, layer3.15.conv1.weight_beta, layer3.15.conv2.weight_diff, layer3.15.conv2.weight_gamma, layer3.15.conv2.weight_beta, layer3.15.conv2.switch.weight, layer3.15.conv2.switch.bias, layer3.15.conv2.pre_context.weight, layer3.15.conv2.pre_context.bias, layer3.15.conv2.post_context.weight, layer3.15.conv2.post_context.bias, layer3.15.conv2.offset_s.weight, layer3.15.conv2.offset_s.bias, layer3.15.conv2.offset_l.weight, layer3.15.conv2.offset_l.bias, layer3.15.conv3.weight_gamma, layer3.15.conv3.weight_beta, layer3.16.conv1.weight_gamma, layer3.16.conv1.weight_beta, layer3.16.conv2.weight_diff, layer3.16.conv2.weight_gamma, layer3.16.conv2.weight_beta, layer3.16.conv2.switch.weight, layer3.16.conv2.switch.bias, layer3.16.conv2.pre_context.weight, layer3.16.conv2.pre_context.bias, layer3.16.conv2.post_context.weight, layer3.16.conv2.post_context.bias, layer3.16.conv2.offset_s.weight, layer3.16.conv2.offset_s.bias, layer3.16.conv2.offset_l.weight, layer3.16.conv2.offset_l.bias, layer3.16.conv3.weight_gamma, layer3.16.conv3.weight_beta, layer3.17.conv1.weight_gamma, layer3.17.conv1.weight_beta, layer3.17.conv2.weight_diff, layer3.17.conv2.weight_gamma, layer3.17.conv2.weight_beta, layer3.17.conv2.switch.weight, layer3.17.conv2.switch.bias, layer3.17.conv2.pre_context.weight, layer3.17.conv2.pre_context.bias, layer3.17.conv2.post_context.weight, layer3.17.conv2.post_context.bias, layer3.17.conv2.offset_s.weight, layer3.17.conv2.offset_s.bias, layer3.17.conv2.offset_l.weight, layer3.17.conv2.offset_l.bias, layer3.17.conv3.weight_gamma, layer3.17.conv3.weight_beta, layer3.18.conv1.weight_gamma, layer3.18.conv1.weight_beta, layer3.18.conv2.weight_diff, layer3.18.conv2.weight_gamma, layer3.18.conv2.weight_beta, layer3.18.conv2.switch.weight, layer3.18.conv2.switch.bias, layer3.18.conv2.pre_context.weight, layer3.18.conv2.pre_context.bias, layer3.18.conv2.post_context.weight, layer3.18.conv2.post_context.bias, layer3.18.conv2.offset_s.weight, layer3.18.conv2.offset_s.bias, layer3.18.conv2.offset_l.weight, layer3.18.conv2.offset_l.bias, layer3.18.conv3.weight_gamma, layer3.18.conv3.weight_beta, layer3.19.conv1.weight_gamma, layer3.19.conv1.weight_beta, layer3.19.conv2.weight_diff, layer3.19.conv2.weight_gamma, layer3.19.conv2.weight_beta, layer3.19.conv2.switch.weight, layer3.19.conv2.switch.bias, layer3.19.conv2.pre_context.weight, layer3.19.conv2.pre_context.bias, layer3.19.conv2.post_context.weight, layer3.19.conv2.post_context.bias, layer3.19.conv2.offset_s.weight, layer3.19.conv2.offset_s.bias, layer3.19.conv2.offset_l.weight, layer3.19.conv2.offset_l.bias, layer3.19.conv3.weight_gamma, layer3.19.conv3.weight_beta, layer3.20.conv1.weight_gamma, layer3.20.conv1.weight_beta, layer3.20.conv2.weight_diff, layer3.20.conv2.weight_gamma, layer3.20.conv2.weight_beta, layer3.20.conv2.switch.weight, layer3.20.conv2.switch.bias, layer3.20.conv2.pre_context.weight, layer3.20.conv2.pre_context.bias, layer3.20.conv2.post_context.weight, layer3.20.conv2.post_context.bias, layer3.20.conv2.offset_s.weight, layer3.20.conv2.offset_s.bias, layer3.20.conv2.offset_l.weight, layer3.20.conv2.offset_l.bias, layer3.20.conv3.weight_gamma, layer3.20.conv3.weight_beta, layer3.21.conv1.weight_gamma, layer3.21.conv1.weight_beta, layer3.21.conv2.weight_diff, layer3.21.conv2.weight_gamma, layer3.21.conv2.weight_beta, layer3.21.conv2.switch.weight, layer3.21.conv2.switch.bias, layer3.21.conv2.pre_context.weight, layer3.21.conv2.pre_context.bias, layer3.21.conv2.post_context.weight, layer3.21.conv2.post_context.bias, layer3.21.conv2.offset_s.weight, layer3.21.conv2.offset_s.bias, layer3.21.conv2.offset_l.weight, layer3.21.conv2.offset_l.bias, layer3.21.conv3.weight_gamma, layer3.21.conv3.weight_beta, layer3.22.conv1.weight_gamma, layer3.22.conv1.weight_beta, layer3.22.conv2.weight_diff, layer3.22.conv2.weight_gamma, layer3.22.conv2.weight_beta, layer3.22.conv2.switch.weight, layer3.22.conv2.switch.bias, layer3.22.conv2.pre_context.weight, layer3.22.conv2.pre_context.bias, layer3.22.conv2.post_context.weight, layer3.22.conv2.post_context.bias, layer3.22.conv2.offset_s.weight, layer3.22.conv2.offset_s.bias, layer3.22.conv2.offset_l.weight, layer3.22.conv2.offset_l.bias, layer3.22.conv3.weight_gamma, layer3.22.conv3.weight_beta, layer4.0.conv1.weight_gamma, layer4.0.conv1.weight_beta, layer4.0.conv2.weight_diff, layer4.0.conv2.weight_gamma, layer4.0.conv2.weight_beta, layer4.0.conv2.switch.weight, layer4.0.conv2.switch.bias, layer4.0.conv2.pre_context.weight, layer4.0.conv2.pre_context.bias, layer4.0.conv2.post_context.weight, layer4.0.conv2.post_context.bias, layer4.0.conv2.offset_s.weight, layer4.0.conv2.offset_s.bias, layer4.0.conv2.offset_l.weight, layer4.0.conv2.offset_l.bias, layer4.0.conv3.weight_gamma, layer4.0.conv3.weight_beta, layer4.0.downsample.0.weight_gamma, layer4.0.downsample.0.weight_beta, layer4.1.conv1.weight_gamma, layer4.1.conv1.weight_beta, layer4.1.conv2.weight_diff, layer4.1.conv2.weight_gamma, layer4.1.conv2.weight_beta, layer4.1.conv2.switch.weight, layer4.1.conv2.switch.bias, layer4.1.conv2.pre_context.weight, layer4.1.conv2.pre_context.bias, layer4.1.conv2.post_context.weight, layer4.1.conv2.post_context.bias, layer4.1.conv2.offset_s.weight, layer4.1.conv2.offset_s.bias, layer4.1.conv2.offset_l.weight, layer4.1.conv2.offset_l.bias, layer4.1.conv3.weight_gamma, layer4.1.conv3.weight_beta, layer4.2.conv1.weight_gamma, layer4.2.conv1.weight_beta, layer4.2.conv2.weight_diff, layer4.2.conv2.weight_gamma, layer4.2.conv2.weight_beta, layer4.2.conv2.switch.weight, layer4.2.conv2.switch.bias, layer4.2.conv2.pre_context.weight, layer4.2.conv2.pre_context.bias, layer4.2.conv2.post_context.weight, layer4.2.conv2.post_context.bias, layer4.2.conv2.offset_s.weight, layer4.2.conv2.offset_s.bias, layer4.2.conv2.offset_l.weight, layer4.2.conv2.offset_l.bias, layer4.2.conv3.weight_gamma, layer4.2.conv3.weight_beta

2020-06-08 08:59:42,601 - mmdet - WARNING - The model and loaded state dict do not match exactly

missing keys in source state_dict: conv1.weight_gamma, conv1.weight_beta, layer1.0.conv1.weight_gamma, layer1.0.conv1.weight_beta, layer1.0.conv2.weight_gamma, layer1.0.conv2.weight_beta, layer1.0.conv3.weight_gamma, layer1.0.conv3.weight_beta, layer1.0.downsample.0.weight_gamma, layer1.0.downsample.0.weight_beta, layer1.1.conv1.weight_gamma, layer1.1.conv1.weight_beta, layer1.1.conv2.weight_gamma, layer1.1.conv2.weight_beta, layer1.1.conv3.weight_gamma, layer1.1.conv3.weight_beta, layer1.2.conv1.weight_gamma, layer1.2.conv1.weight_beta, layer1.2.conv2.weight_gamma, layer1.2.conv2.weight_beta, layer1.2.conv3.weight_gamma, layer1.2.conv3.weight_beta, layer2.0.conv1.weight_gamma, layer2.0.conv1.weight_beta, layer2.0.conv2.weight_diff, layer2.0.conv2.weight_gamma, layer2.0.conv2.weight_beta, layer2.0.conv2.switch.weight, layer2.0.conv2.switch.bias, layer2.0.conv2.pre_context.weight, layer2.0.conv2.pre_context.bias, layer2.0.conv2.post_context.weight, layer2.0.conv2.post_context.bias, layer2.0.conv2.offset_s.weight, layer2.0.conv2.offset_s.bias, layer2.0.conv2.offset_l.weight, layer2.0.conv2.offset_l.bias, layer2.0.conv3.weight_gamma, layer2.0.conv3.weight_beta, layer2.0.downsample.0.weight_gamma, layer2.0.downsample.0.weight_beta, layer2.0.rfp_conv.weight, layer2.0.rfp_conv.bias, layer2.1.conv1.weight_gamma, layer2.1.conv1.weight_beta, layer2.1.conv2.weight_diff, layer2.1.conv2.weight_gamma, layer2.1.conv2.weight_beta, layer2.1.conv2.switch.weight, layer2.1.conv2.switch.bias, layer2.1.conv2.pre_context.weight, layer2.1.conv2.pre_context.bias, layer2.1.conv2.post_context.weight, layer2.1.conv2.post_context.bias, layer2.1.conv2.offset_s.weight, layer2.1.conv2.offset_s.bias, layer2.1.conv2.offset_l.weight, layer2.1.conv2.offset_l.bias, layer2.1.conv3.weight_gamma, layer2.1.conv3.weight_beta, layer2.2.conv1.weight_gamma, layer2.2.conv1.weight_beta, layer2.2.conv2.weight_diff, layer2.2.conv2.weight_gamma, layer2.2.conv2.weight_beta, layer2.2.conv2.switch.weight, layer2.2.conv2.switch.bias, layer2.2.conv2.pre_context.weight, layer2.2.conv2.pre_context.bias, layer2.2.conv2.post_context.weight, layer2.2.conv2.post_context.bias, layer2.2.conv2.offset_s.weight, layer2.2.conv2.offset_s.bias, layer2.2.conv2.offset_l.weight, layer2.2.conv2.offset_l.bias, layer2.2.conv3.weight_gamma, layer2.2.conv3.weight_beta, layer2.3.conv1.weight_gamma, layer2.3.conv1.weight_beta, layer2.3.conv2.weight_diff, layer2.3.conv2.weight_gamma, layer2.3.conv2.weight_beta, layer2.3.conv2.switch.weight, layer2.3.conv2.switch.bias, layer2.3.conv2.pre_context.weight, layer2.3.conv2.pre_context.bias, layer2.3.conv2.post_context.weight, layer2.3.conv2.post_context.bias, layer2.3.conv2.offset_s.weight, layer2.3.conv2.offset_s.bias, layer2.3.conv2.offset_l.weight, layer2.3.conv2.offset_l.bias, layer2.3.conv3.weight_gamma, layer2.3.conv3.weight_beta, layer3.0.conv1.weight_gamma, layer3.0.conv1.weight_beta, layer3.0.conv2.weight_diff, layer3.0.conv2.weight_gamma, layer3.0.conv2.weight_beta, layer3.0.conv2.switch.weight, layer3.0.conv2.switch.bias, layer3.0.conv2.pre_context.weight, layer3.0.conv2.pre_context.bias, layer3.0.conv2.post_context.weight, layer3.0.conv2.post_context.bias, layer3.0.conv2.offset_s.weight, layer3.0.conv2.offset_s.bias, layer3.0.conv2.offset_l.weight, layer3.0.conv2.offset_l.bias, layer3.0.conv3.weight_gamma, layer3.0.conv3.weight_beta, layer3.0.downsample.0.weight_gamma, layer3.0.downsample.0.weight_beta, layer3.0.rfp_conv.weight, layer3.0.rfp_conv.bias, layer3.1.conv1.weight_gamma, layer3.1.conv1.weight_beta, layer3.1.conv2.weight_diff, layer3.1.conv2.weight_gamma, layer3.1.conv2.weight_beta, layer3.1.conv2.switch.weight, layer3.1.conv2.switch.bias, layer3.1.conv2.pre_context.weight, layer3.1.conv2.pre_context.bias, layer3.1.conv2.post_context.weight, layer3.1.conv2.post_context.bias, layer3.1.conv2.offset_s.weight, layer3.1.conv2.offset_s.bias, layer3.1.conv2.offset_l.weight, layer3.1.conv2.offset_l.bias, layer3.1.conv3.weight_gamma, layer3.1.conv3.weight_beta, layer3.2.conv1.weight_gamma, layer3.2.conv1.weight_beta, layer3.2.conv2.weight_diff, layer3.2.conv2.weight_gamma, layer3.2.conv2.weight_beta, layer3.2.conv2.switch.weight, layer3.2.conv2.switch.bias, layer3.2.conv2.pre_context.weight, layer3.2.conv2.pre_context.bias, layer3.2.conv2.post_context.weight, layer3.2.conv2.post_context.bias, layer3.2.conv2.offset_s.weight, layer3.2.conv2.offset_s.bias, layer3.2.conv2.offset_l.weight, layer3.2.conv2.offset_l.bias, layer3.2.conv3.weight_gamma, layer3.2.conv3.weight_beta, layer3.3.conv1.weight_gamma, layer3.3.conv1.weight_beta, layer3.3.conv2.weight_diff, layer3.3.conv2.weight_gamma, layer3.3.conv2.weight_beta, layer3.3.conv2.switch.weight, layer3.3.conv2.switch.bias, layer3.3.conv2.pre_context.weight, layer3.3.conv2.pre_context.bias, layer3.3.conv2.post_context.weight, layer3.3.conv2.post_context.bias, layer3.3.conv2.offset_s.weight, layer3.3.conv2.offset_s.bias, layer3.3.conv2.offset_l.weight, layer3.3.conv2.offset_l.bias, layer3.3.conv3.weight_gamma, layer3.3.conv3.weight_beta, layer3.4.conv1.weight_gamma, layer3.4.conv1.weight_beta, layer3.4.conv2.weight_diff, layer3.4.conv2.weight_gamma, layer3.4.conv2.weight_beta, layer3.4.conv2.switch.weight, layer3.4.conv2.switch.bias, layer3.4.conv2.pre_context.weight, layer3.4.conv2.pre_context.bias, layer3.4.conv2.post_context.weight, layer3.4.conv2.post_context.bias, layer3.4.conv2.offset_s.weight, layer3.4.conv2.offset_s.bias, layer3.4.conv2.offset_l.weight, layer3.4.conv2.offset_l.bias, layer3.4.conv3.weight_gamma, layer3.4.conv3.weight_beta, layer3.5.conv1.weight_gamma, layer3.5.conv1.weight_beta, layer3.5.conv2.weight_diff, layer3.5.conv2.weight_gamma, layer3.5.conv2.weight_beta, layer3.5.conv2.switch.weight, layer3.5.conv2.switch.bias, layer3.5.conv2.pre_context.weight, layer3.5.conv2.pre_context.bias, layer3.5.conv2.post_context.weight, layer3.5.conv2.post_context.bias, layer3.5.conv2.offset_s.weight, layer3.5.conv2.offset_s.bias, layer3.5.conv2.offset_l.weight, layer3.5.conv2.offset_l.bias, layer3.5.conv3.weight_gamma, layer3.5.conv3.weight_beta, layer3.6.conv1.weight_gamma, layer3.6.conv1.weight_beta, layer3.6.conv2.weight_diff, layer3.6.conv2.weight_gamma, layer3.6.conv2.weight_beta, layer3.6.conv2.switch.weight, layer3.6.conv2.switch.bias, layer3.6.conv2.pre_context.weight, layer3.6.conv2.pre_context.bias, layer3.6.conv2.post_context.weight, layer3.6.conv2.post_context.bias, layer3.6.conv2.offset_s.weight, layer3.6.conv2.offset_s.bias, layer3.6.conv2.offset_l.weight, layer3.6.conv2.offset_l.bias, layer3.6.conv3.weight_gamma, layer3.6.conv3.weight_beta, layer3.7.conv1.weight_gamma, layer3.7.conv1.weight_beta, layer3.7.conv2.weight_diff, layer3.7.conv2.weight_gamma, layer3.7.conv2.weight_beta, layer3.7.conv2.switch.weight, layer3.7.conv2.switch.bias, layer3.7.conv2.pre_context.weight, layer3.7.conv2.pre_context.bias, layer3.7.conv2.post_context.weight, layer3.7.conv2.post_context.bias, layer3.7.conv2.offset_s.weight, layer3.7.conv2.offset_s.bias, layer3.7.conv2.offset_l.weight, layer3.7.conv2.offset_l.bias, layer3.7.conv3.weight_gamma, layer3.7.conv3.weight_beta, layer3.8.conv1.weight_gamma, layer3.8.conv1.weight_beta, layer3.8.conv2.weight_diff, layer3.8.conv2.weight_gamma, layer3.8.conv2.weight_beta, layer3.8.conv2.switch.weight, layer3.8.conv2.switch.bias, layer3.8.conv2.pre_context.weight, layer3.8.conv2.pre_context.bias, layer3.8.conv2.post_context.weight, layer3.8.conv2.post_context.bias, layer3.8.conv2.offset_s.weight, layer3.8.conv2.offset_s.bias, layer3.8.conv2.offset_l.weight, layer3.8.conv2.offset_l.bias, layer3.8.conv3.weight_gamma, layer3.8.conv3.weight_beta, layer3.9.conv1.weight_gamma, layer3.9.conv1.weight_beta, layer3.9.conv2.weight_diff, layer3.9.conv2.weight_gamma, layer3.9.conv2.weight_beta, layer3.9.conv2.switch.weight, layer3.9.conv2.switch.bias, layer3.9.conv2.pre_context.weight, layer3.9.conv2.pre_context.bias, layer3.9.conv2.post_context.weight, layer3.9.conv2.post_context.bias, layer3.9.conv2.offset_s.weight, layer3.9.conv2.offset_s.bias, layer3.9.conv2.offset_l.weight, layer3.9.conv2.offset_l.bias, layer3.9.conv3.weight_gamma, layer3.9.conv3.weight_beta, layer3.10.conv1.weight_gamma, layer3.10.conv1.weight_beta, layer3.10.conv2.weight_diff, layer3.10.conv2.weight_gamma, layer3.10.conv2.weight_beta, layer3.10.conv2.switch.weight, layer3.10.conv2.switch.bias, layer3.10.conv2.pre_context.weight, layer3.10.conv2.pre_context.bias, layer3.10.conv2.post_context.weight, layer3.10.conv2.post_context.bias, layer3.10.conv2.offset_s.weight, layer3.10.conv2.offset_s.bias, layer3.10.conv2.offset_l.weight, layer3.10.conv2.offset_l.bias, layer3.10.conv3.weight_gamma, layer3.10.conv3.weight_beta, layer3.11.conv1.weight_gamma, layer3.11.conv1.weight_beta, layer3.11.conv2.weight_diff, layer3.11.conv2.weight_gamma, layer3.11.conv2.weight_beta, layer3.11.conv2.switch.weight, layer3.11.conv2.switch.bias, layer3.11.conv2.pre_context.weight, layer3.11.conv2.pre_context.bias, layer3.11.conv2.post_context.weight, layer3.11.conv2.post_context.bias, layer3.11.conv2.offset_s.weight, layer3.11.conv2.offset_s.bias, layer3.11.conv2.offset_l.weight, layer3.11.conv2.offset_l.bias, layer3.11.conv3.weight_gamma, layer3.11.conv3.weight_beta, layer3.12.conv1.weight_gamma, layer3.12.conv1.weight_beta, layer3.12.conv2.weight_diff, layer3.12.conv2.weight_gamma, layer3.12.conv2.weight_beta, layer3.12.conv2.switch.weight, layer3.12.conv2.switch.bias, layer3.12.conv2.pre_context.weight, layer3.12.conv2.pre_context.bias, layer3.12.conv2.post_context.weight, layer3.12.conv2.post_context.bias, layer3.12.conv2.offset_s.weight, layer3.12.conv2.offset_s.bias, layer3.12.conv2.offset_l.weight, layer3.12.conv2.offset_l.bias, layer3.12.conv3.weight_gamma, layer3.12.conv3.weight_beta, layer3.13.conv1.weight_gamma, layer3.13.conv1.weight_beta, layer3.13.conv2.weight_diff, layer3.13.conv2.weight_gamma, layer3.13.conv2.weight_beta, layer3.13.conv2.switch.weight, layer3.13.conv2.switch.bias, layer3.13.conv2.pre_context.weight, layer3.13.conv2.pre_context.bias, layer3.13.conv2.post_context.weight, layer3.13.conv2.post_context.bias, layer3.13.conv2.offset_s.weight, layer3.13.conv2.offset_s.bias, layer3.13.conv2.offset_l.weight, layer3.13.conv2.offset_l.bias, layer3.13.conv3.weight_gamma, layer3.13.conv3.weight_beta, layer3.14.conv1.weight_gamma, layer3.14.conv1.weight_beta, layer3.14.conv2.weight_diff, layer3.14.conv2.weight_gamma, layer3.14.conv2.weight_beta, layer3.14.conv2.switch.weight, layer3.14.conv2.switch.bias, layer3.14.conv2.pre_context.weight, layer3.14.conv2.pre_context.bias, layer3.14.conv2.post_context.weight, layer3.14.conv2.post_context.bias, layer3.14.conv2.offset_s.weight, layer3.14.conv2.offset_s.bias, layer3.14.conv2.offset_l.weight, layer3.14.conv2.offset_l.bias, layer3.14.conv3.weight_gamma, layer3.14.conv3.weight_beta, layer3.15.conv1.weight_gamma, layer3.15.conv1.weight_beta, layer3.15.conv2.weight_diff, layer3.15.conv2.weight_gamma, layer3.15.conv2.weight_beta, layer3.15.conv2.switch.weight, layer3.15.conv2.switch.bias, layer3.15.conv2.pre_context.weight, layer3.15.conv2.pre_context.bias, layer3.15.conv2.post_context.weight, layer3.15.conv2.post_context.bias, layer3.15.conv2.offset_s.weight, layer3.15.conv2.offset_s.bias, layer3.15.conv2.offset_l.weight, layer3.15.conv2.offset_l.bias, layer3.15.conv3.weight_gamma, layer3.15.conv3.weight_beta, layer3.16.conv1.weight_gamma, layer3.16.conv1.weight_beta, layer3.16.conv2.weight_diff, layer3.16.conv2.weight_gamma, layer3.16.conv2.weight_beta, layer3.16.conv2.switch.weight, layer3.16.conv2.switch.bias, layer3.16.conv2.pre_context.weight, layer3.16.conv2.pre_context.bias, layer3.16.conv2.post_context.weight, layer3.16.conv2.post_context.bias, layer3.16.conv2.offset_s.weight, layer3.16.conv2.offset_s.bias, layer3.16.conv2.offset_l.weight, layer3.16.conv2.offset_l.bias, layer3.16.conv3.weight_gamma, layer3.16.conv3.weight_beta, layer3.17.conv1.weight_gamma, layer3.17.conv1.weight_beta, layer3.17.conv2.weight_diff, layer3.17.conv2.weight_gamma, layer3.17.conv2.weight_beta, layer3.17.conv2.switch.weight, layer3.17.conv2.switch.bias, layer3.17.conv2.pre_context.weight, layer3.17.conv2.pre_context.bias, layer3.17.conv2.post_context.weight, layer3.17.conv2.post_context.bias, layer3.17.conv2.offset_s.weight, layer3.17.conv2.offset_s.bias, layer3.17.conv2.offset_l.weight, layer3.17.conv2.offset_l.bias, layer3.17.conv3.weight_gamma, layer3.17.conv3.weight_beta, layer3.18.conv1.weight_gamma, layer3.18.conv1.weight_beta, layer3.18.conv2.weight_diff, layer3.18.conv2.weight_gamma, layer3.18.conv2.weight_beta, layer3.18.conv2.switch.weight, layer3.18.conv2.switch.bias, layer3.18.conv2.pre_context.weight, layer3.18.conv2.pre_context.bias, layer3.18.conv2.post_context.weight, layer3.18.conv2.post_context.bias, layer3.18.conv2.offset_s.weight, layer3.18.conv2.offset_s.bias, layer3.18.conv2.offset_l.weight, layer3.18.conv2.offset_l.bias, layer3.18.conv3.weight_gamma, layer3.18.conv3.weight_beta, layer3.19.conv1.weight_gamma, layer3.19.conv1.weight_beta, layer3.19.conv2.weight_diff, layer3.19.conv2.weight_gamma, layer3.19.conv2.weight_beta, layer3.19.conv2.switch.weight, layer3.19.conv2.switch.bias, layer3.19.conv2.pre_context.weight, layer3.19.conv2.pre_context.bias, layer3.19.conv2.post_context.weight, layer3.19.conv2.post_context.bias, layer3.19.conv2.offset_s.weight, layer3.19.conv2.offset_s.bias, layer3.19.conv2.offset_l.weight, layer3.19.conv2.offset_l.bias, layer3.19.conv3.weight_gamma, layer3.19.conv3.weight_beta, layer3.20.conv1.weight_gamma, layer3.20.conv1.weight_beta, layer3.20.conv2.weight_diff, layer3.20.conv2.weight_gamma, layer3.20.conv2.weight_beta, layer3.20.conv2.switch.weight, layer3.20.conv2.switch.bias, layer3.20.conv2.pre_context.weight, layer3.20.conv2.pre_context.bias, layer3.20.conv2.post_context.weight, layer3.20.conv2.post_context.bias, layer3.20.conv2.offset_s.weight, layer3.20.conv2.offset_s.bias, layer3.20.conv2.offset_l.weight, layer3.20.conv2.offset_l.bias, layer3.20.conv3.weight_gamma, layer3.20.conv3.weight_beta, layer3.21.conv1.weight_gamma, layer3.21.conv1.weight_beta, layer3.21.conv2.weight_diff, layer3.21.conv2.weight_gamma, layer3.21.conv2.weight_beta, layer3.21.conv2.switch.weight, layer3.21.conv2.switch.bias, layer3.21.conv2.pre_context.weight, layer3.21.conv2.pre_context.bias, layer3.21.conv2.post_context.weight, layer3.21.conv2.post_context.bias, layer3.21.conv2.offset_s.weight, layer3.21.conv2.offset_s.bias, layer3.21.conv2.offset_l.weight, layer3.21.conv2.offset_l.bias, layer3.21.conv3.weight_gamma, layer3.21.conv3.weight_beta, layer3.22.conv1.weight_gamma, layer3.22.conv1.weight_beta, layer3.22.conv2.weight_diff, layer3.22.conv2.weight_gamma, layer3.22.conv2.weight_beta, layer3.22.conv2.switch.weight, layer3.22.conv2.switch.bias, layer3.22.conv2.pre_context.weight, layer3.22.conv2.pre_context.bias, layer3.22.conv2.post_context.weight, layer3.22.conv2.post_context.bias, layer3.22.conv2.offset_s.weight, layer3.22.conv2.offset_s.bias, layer3.22.conv2.offset_l.weight, layer3.22.conv2.offset_l.bias, layer3.22.conv3.weight_gamma, layer3.22.conv3.weight_beta, layer4.0.conv1.weight_gamma, layer4.0.conv1.weight_beta, layer4.0.conv2.weight_diff, layer4.0.conv2.weight_gamma, layer4.0.conv2.weight_beta, layer4.0.conv2.switch.weight, layer4.0.conv2.switch.bias, layer4.0.conv2.pre_context.weight, layer4.0.conv2.pre_context.bias, layer4.0.conv2.post_context.weight, layer4.0.conv2.post_context.bias, layer4.0.conv2.offset_s.weight, layer4.0.conv2.offset_s.bias, layer4.0.conv2.offset_l.weight, layer4.0.conv2.offset_l.bias, layer4.0.conv3.weight_gamma, layer4.0.conv3.weight_beta, layer4.0.downsample.0.weight_gamma, layer4.0.downsample.0.weight_beta, layer4.0.rfp_conv.weight, layer4.0.rfp_conv.bias, layer4.1.conv1.weight_gamma, layer4.1.conv1.weight_beta, layer4.1.conv2.weight_diff, layer4.1.conv2.weight_gamma, layer4.1.conv2.weight_beta, layer4.1.conv2.switch.weight, layer4.1.conv2.switch.bias, layer4.1.conv2.pre_context.weight, layer4.1.conv2.pre_context.bias, layer4.1.conv2.post_context.weight, layer4.1.conv2.post_context.bias, layer4.1.conv2.offset_s.weight, layer4.1.conv2.offset_s.bias, layer4.1.conv2.offset_l.weight, layer4.1.conv2.offset_l.bias, layer4.1.conv3.weight_gamma, layer4.1.conv3.weight_beta, layer4.2.conv1.weight_gamma, layer4.2.conv1.weight_beta, layer4.2.conv2.weight_diff, layer4.2.conv2.weight_gamma, layer4.2.conv2.weight_beta, layer4.2.conv2.switch.weight, layer4.2.conv2.switch.bias, layer4.2.conv2.pre_context.weight, layer4.2.conv2.pre_context.bias, layer4.2.conv2.post_context.weight, layer4.2.conv2.post_context.bias, layer4.2.conv2.offset_s.weight, layer4.2.conv2.offset_s.bias, layer4.2.conv2.offset_l.weight, layer4.2.conv2.offset_l.bias, layer4.2.conv3.weight_gamma, layer4.2.conv3.weight_beta

loading annotations into memory...
Done (t=0.00s)
creating index...
index created!
2020-06-08 08:59:45,129 - mmdet - INFO - Start running, host: root@c73feb983947, work_dir: /content/drive/My Drive/DetectoRS/work_dirs/DetectoRS_mstrain_400_1200_x101_32x4d_40e.py
2020-06-08 08:59:45,130 - mmdet - INFO - workflow: [('train', 1)], max: 40 epochs
error in deformable_im2col: no kernel image is available for execution on the device
error in deformable_im2col: no kernel image is available for execution on the device
error in deformable_im2col: no kernel image is available for execution on the device
error in deformable_im2col: no kernel image is available for execution on the device
error in deformable_im2col: no kernel image is available for execution on the device
error in deformable_im2col: no kernel image is available for execution on the device
error in deformable_im2col: no kernel image is available for execution on the device
error in deformable_im2col: no kernel image is available for execution on the device
error in deformable_im2col: no kernel image is available for execution on the device
error in deformable_im2col: no kernel image is available for execution on the device
error in deformable_im2col: no kernel image is available for execution on the device
error in deformable_im2col: no kernel image is available for execution on the device
error in deformable_im2col: no kernel image is available for execution on the device
error in deformable_im2col: no kernel image is available for execution on the device
error in deformable_im2col: no kernel image is available for execution on the device
error in deformable_im2col: no kernel image is available for execution on the device
error in deformable_im2col: no kernel image is available for execution on the device
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [93,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [94,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [95,0,0] Assertion `t >= 0 && t < n_classes` failed.
Traceback (most recent call last):
  File "/content/drive/My Drive/DetectoRS/tools/train.py", line 142, in <module>
    main()
  File "/content/drive/My Drive/DetectoRS/tools/train.py", line 138, in main
    meta=meta)
  File "/content/drive/My Drive/DetectoRS/mmdet/apis/train.py", line 111, in train_detector
    meta=meta)
  File "/content/drive/My Drive/DetectoRS/mmdet/apis/train.py", line 242, in _non_dist_train
    runner.run(data_loaders, cfg.workflow, cfg.total_epochs)
  File "/usr/local/lib/python3.6/dist-packages/mmcv/runner/runner.py", line 384, in run
    epoch_runner(data_loaders[i], **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/mmcv/runner/runner.py", line 283, in train
    self.model, data_batch, train_mode=True, **kwargs)
  File "/content/drive/My Drive/DetectoRS/mmdet/apis/train.py", line 75, in batch_processor
    losses = model(**data)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/data_parallel.py", line 150, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/content/drive/My Drive/DetectoRS/mmdet/core/fp16/decorators.py", line 49, in new_func
    return old_func(*args, **kwargs)
  File "/content/drive/My Drive/DetectoRS/mmdet/models/detectors/base.py", line 147, in forward
    return self.forward_train(img, img_metas, **kwargs)
  File "/content/drive/My Drive/DetectoRS/mmdet/models/detectors/htc.py", line 252, in forward_train
    gt_labels[j])
  File "/content/drive/My Drive/DetectoRS/mmdet/core/bbox/assigners/max_iou_assigner.py", line 113, in assign
    assign_result = self.assign_wrt_overlaps(overlaps, gt_labels)
  File "/content/drive/My Drive/DetectoRS/mmdet/core/bbox/assigners/max_iou_assigner.py", line 166, in assign_wrt_overlaps
    & (max_overlaps < self.neg_iou_thr)] = 0
RuntimeError: copy_if failed to synchronize: device-side assert triggered
terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: device-side assert triggered (insert_events at /pytorch/c10/cuda/CUDACachingAllocator.cpp:764)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x33 (0x7f200fd5c193 in /usr/local/lib/python3.6/dist-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x17f66 (0x7f200ff99f66 in /usr/local/lib/python3.6/dist-packages/torch/lib/libc10_cuda.so)
frame #2: <unknown function> + 0x19cbd (0x7f200ff9bcbd in /usr/local/lib/python3.6/dist-packages/torch/lib/libc10_cuda.so)
frame #3: c10::TensorImpl::release_resources() + 0x4d (0x7f200fd4c63d in /usr/local/lib/python3.6/dist-packages/torch/lib/libc10.so)
frame #4: <unknown function> + 0x67a902 (0x7f2058d83902 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
frame #5: <unknown function> + 0x67a9a6 (0x7f2058d839a6 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
frame #6: python3() [0x54ef16]
frame #7: python3() [0x572890]
frame #8: python3() [0x54ec1b]
frame #9: python3() [0x588948]
frame #10: python3() [0x5ad438]
frame #11: python3() [0x5ad44e]
frame #12: python3() [0x5ad44e]
frame #13: python3() [0x5ad44e]
frame #14: python3() [0x5ad44e]
frame #15: python3() [0x5ad44e]
frame #16: python3() [0x5ad44e]
frame #17: python3() [0x5ad44e]
frame #18: python3() [0x5ad44e]
frame #19: python3() [0x56b276]
<omitting python frames>
frame #25: __libc_start_main + 0xe7 (0x7f20643f3b97 in /lib/x86_64-linux-gnu/libc.so.6)
joe-siyuan-qiao commented 4 years ago

Hi, it seems that the issue might be related to the architecture of the GPU you're using. Please consider setting the use_deform in SAC to False. It may help if it was DCN that stopped the program.

MotiBaadror commented 4 years ago

Thank you for your response. Part of the error has gone from your solution But still left with below error. 

2020-06-09 04:24:59,731 - mmdet - INFO - Environment info:
------------------------------------------------------------
sys.platform: linux
Python: 3.6.9 (default, Apr 18 2020, 01:56:04) [GCC 8.4.0]
CUDA available: True
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 10.1, V10.1.243
GPU 0: Tesla K80
GCC: gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
PyTorch: 1.4.0+cu100
PyTorch compiling details: PyTorch built with:
  - GCC 7.3
  - Intel(R) Math Kernel Library Version 2019.0.4 Product Build 20190411 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v0.21.1 (Git Hash 7d2fd500bc78936d1d648ca713b901012f470dbc)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - NNPACK is enabled
  - CUDA Runtime 10.0
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_37,code=compute_37
  - CuDNN 7.6.3
  - Magma 2.5.1
  - Build settings: BLAS=MKL, BUILD_NAMEDTENSOR=OFF, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -fopenmp -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Wno-stringop-overflow, DISABLE_NUMA=1, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_STATIC_DISPATCH=OFF, 

TorchVision: 0.5.0+cu100
OpenCV: 4.1.2
MMCV: 0.5.9
MMDetection: 1.1.0+bccc528
MMDetection Compiler: GCC 7.5
MMDetection CUDA Compiler: 10.1
------------------------------------------------------------

2020-06-09 04:24:59,732 - mmdet - INFO - Distributed training: False
2020-06-09 04:24:59,732 - mmdet - INFO - Config:
/content/drive/My Drive/DetectoRS/configs/DetectoRS/DetectoRS_mstrain_400_1200_x101_32x4d_40e.py
conv_cfg = dict(type='ConvAWS')
# model settings
model = dict(
    type='RecursiveFeaturePyramid',
    rfp_steps=2,
    rfp_sharing=False,
    stage_with_rfp=(False, True, True, True),
    num_stages=3,
    pretrained='open-mmlab://resnext101_32x4d',
    interleaved=True,
    mask_info_flow=True,
    backbone=dict(
        type='ResNeXt',
        depth=101,
        groups=32,
        base_width=4,
        num_stages=4,
        out_indices=(0, 1, 2, 3),
        frozen_stages=1,
        conv_cfg=conv_cfg,
        sac=dict(type='SAC', use_deform=False),
        stage_with_sac=(False, True, True, True),
        norm_cfg=dict(type='BN', requires_grad=True),
        style='pytorch'),
    neck=dict(
        type='FPN',
        in_channels=[256, 512, 1024, 2048],
        out_channels=256,
        num_outs=5),
    rpn_head=dict(
        type='RPNHead',
        in_channels=256,
        feat_channels=256,
        anchor_scales=[8],
        anchor_ratios=[0.5, 1.0, 2.0],
        anchor_strides=[4, 8, 16, 32, 64],
        target_means=[.0, .0, .0, .0],
        target_stds=[1.0, 1.0, 1.0, 1.0],
        loss_cls=dict(
            type='CrossEntropyLoss', use_sigmoid=True, loss_weight=1.0),
        loss_bbox=dict(type='SmoothL1Loss', beta=1.0 / 9.0, loss_weight=1.0)),
    bbox_roi_extractor=dict(
        type='SingleRoIExtractor',
        roi_layer=dict(type='RoIAlign', out_size=7, sample_num=2),
        out_channels=256,
        featmap_strides=[4, 8, 16, 32]),
    bbox_head=[
        dict(
            type='SharedFCBBoxHead',
            num_fcs=2,
            in_channels=256,
            fc_out_channels=1024,
            roi_feat_size=7,
            num_classes=2,
            target_means=[0., 0., 0., 0.],
            target_stds=[0.1, 0.1, 0.2, 0.2],
            reg_class_agnostic=True,
            loss_cls=dict(
                type='CrossEntropyLoss', use_sigmoid=False, loss_weight=1.0),
            loss_bbox=dict(type='SmoothL1Loss', beta=1.0, loss_weight=1.0)),
        dict(
            type='SharedFCBBoxHead',
            num_fcs=2,
            in_channels=256,
            fc_out_channels=1024,
            roi_feat_size=7,
            num_classes=2,
            target_means=[0., 0., 0., 0.],
            target_stds=[0.05, 0.05, 0.1, 0.1],
            reg_class_agnostic=True,
            loss_cls=dict(
                type='CrossEntropyLoss', use_sigmoid=False, loss_weight=1.0),
            loss_bbox=dict(type='SmoothL1Loss', beta=1.0, loss_weight=1.0)),
        dict(
            type='SharedFCBBoxHead',
            num_fcs=2,
            in_channels=256,
            fc_out_channels=1024,
            roi_feat_size=7,
            num_classes=2,
            target_means=[0., 0., 0., 0.],
            target_stds=[0.033, 0.033, 0.067, 0.067],
            reg_class_agnostic=True,
            loss_cls=dict(
                type='CrossEntropyLoss', use_sigmoid=False, loss_weight=1.0),
            loss_bbox=dict(type='SmoothL1Loss', beta=1.0, loss_weight=1.0))
    ],
    mask_roi_extractor=dict(
        type='SingleRoIExtractor',
        roi_layer=dict(type='RoIAlign', out_size=14, sample_num=2),
        out_channels=256,
        featmap_strides=[4, 8, 16, 32]),
    mask_head=[
        dict(
            type='HTCMaskHead',
            with_conv_res=False,
            num_convs=4,
            in_channels=256,
            conv_out_channels=256,
            num_classes=1,
            loss_mask=dict(
                type='CrossEntropyLoss', use_mask=True, loss_weight=1.0)),
        dict(
            type='HTCMaskHead',
            num_convs=4,
            in_channels=256,
            conv_out_channels=256,
            num_classes=1,
            loss_mask=dict(
                type='CrossEntropyLoss', use_mask=True, loss_weight=1.0)),
        dict(
            type='HTCMaskHead',
            num_convs=4,
            in_channels=256,
            conv_out_channels=256,
            num_classes=1,
            loss_mask=dict(
                type='CrossEntropyLoss', use_mask=True, loss_weight=1.0))
    ],
    semantic_roi_extractor=dict(
        type='SingleRoIExtractor',
        roi_layer=dict(type='RoIAlign', out_size=14, sample_num=2),
        out_channels=256,
        featmap_strides=[8]),
    semantic_head=dict(
        type='FusedSemanticHead',
        num_ins=5,
        fusion_level=1,
        num_convs=4,
        in_channels=256,
        conv_out_channels=256,
        num_classes=1,
        ignore_label=-1,
        loss_weight=0.2))
# model training and testing settings
train_cfg = dict(
    rpn=dict(
        assigner=dict(
            type='MaxIoUAssigner',
            pos_iou_thr=0.7,
            neg_iou_thr=0.3,
            min_pos_iou=0.3,
            ignore_iof_thr=-1),
        sampler=dict(
            type='RandomSampler',
            num=256,
            pos_fraction=0.5,
            neg_pos_ub=-1,
            add_gt_as_proposals=False),
        allowed_border=0,
        pos_weight=-1,
        debug=False),
    rpn_proposal=dict(
        nms_across_levels=False,
        nms_pre=2000,
        nms_post=2000,
        max_num=2000,
        nms_thr=0.7,
        min_bbox_size=0),
    rcnn=[
        dict(
            assigner=dict(
                type='MaxIoUAssigner',
                pos_iou_thr=0.5,
                neg_iou_thr=0.5,
                min_pos_iou=0.5,
                ignore_iof_thr=-1),
            sampler=dict(
                type='RandomSampler',
                num=512,
                pos_fraction=0.25,
                neg_pos_ub=-1,
                add_gt_as_proposals=True),
            mask_size=28,
            pos_weight=-1,
            debug=False),
        dict(
            assigner=dict(
                type='MaxIoUAssigner',
                pos_iou_thr=0.6,
                neg_iou_thr=0.6,
                min_pos_iou=0.6,
                ignore_iof_thr=-1),
            sampler=dict(
                type='RandomSampler',
                num=512,
                pos_fraction=0.25,
                neg_pos_ub=-1,
                add_gt_as_proposals=True),
            mask_size=28,
            pos_weight=-1,
            debug=False),
        dict(
            assigner=dict(
                type='MaxIoUAssigner',
                pos_iou_thr=0.7,
                neg_iou_thr=0.7,
                min_pos_iou=0.7,
                ignore_iof_thr=-1),
            sampler=dict(
                type='RandomSampler',
                num=512,
                pos_fraction=0.25,
                neg_pos_ub=-1,
                add_gt_as_proposals=True),
            mask_size=28,
            pos_weight=-1,
            debug=False)
    ],
    stage_loss_weights=[1, 0.5, 0.25])
test_cfg = dict(
    rpn=dict(
        nms_across_levels=False,
        nms_pre=1000,
        nms_post=1000,
        max_num=1000,
        nms_thr=0.7,
        min_bbox_size=0),
    rcnn=dict(
        score_thr=0.001,
        nms=dict(type='soft_nms', iou_thr=0.5),
        max_per_img=100,
        mask_thr_binary=0.5))
# dataset settings
dataset_type = 'CocoDataset'
data_root = '/content/drive/My Drive/collab/data/'
img_norm_cfg = dict(
    mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True)
train_pipeline = [
    dict(type='LoadImageFromFile'),
    dict(
        type='LoadAnnotations', with_bbox=True, with_mask=True, with_seg=True),
    dict(
        type='Resize',
        img_scale=[(1333, 400), (1333, 1200)],
        multiscale_mode='range',
        keep_ratio=True),
    dict(type='RandomFlip', flip_ratio=0.5),
    dict(type='Normalize', **img_norm_cfg),
    dict(type='Pad', size_divisor=32),
    dict(type='SegRescale', scale_factor=1 / 8),
    dict(type='DefaultFormatBundle'),
    dict(
        type='Collect',
        keys=['img', 'gt_bboxes', 'gt_labels', 'gt_masks', 'gt_semantic_seg']),
]
test_pipeline = [
    dict(type='LoadImageFromFile'),
    dict(
        type='MultiScaleFlipAug',
        img_scale=(1333, 800),
        flip=False,
        transforms=[
            dict(type='Resize', keep_ratio=True),
            dict(type='RandomFlip', flip_ratio=0.5),
            dict(type='Normalize', **img_norm_cfg),
            dict(type='Pad', size_divisor=32),
            dict(type='ImageToTensor', keys=['img']),
            dict(type='Collect', keys=['img']),
        ])
]
data = dict(
    imgs_per_gpu=1,
    workers_per_gpu=1,
    train=dict(
        type=dataset_type,
        ann_file=data_root + 'annotations/instances_train2017.json',
        img_prefix=data_root + 'train2017/',
        seg_prefix=data_root + 'train2017/',
        pipeline=train_pipeline),
    val=dict(
        type=dataset_type,
        ann_file=data_root + 'annotations/instances_val2017.json',
        img_prefix=data_root + 'val2017/',
        pipeline=test_pipeline),
    test=dict(
        type=dataset_type,
        ann_file=data_root + 'annotations/instances_val2017.json',
        img_prefix=data_root + 'val2017/',
        pipeline=test_pipeline))
evaluation = dict(interval=1, metric=['bbox', 'segm'])
# optimizer
optimizer = dict(type='SGD', lr=0.02, momentum=0.9, weight_decay=0.0001)
optimizer_config = dict(grad_clip=dict(max_norm=35, norm_type=2))
# learning policy
lr_config = dict(
    policy='step',
    warmup='linear',
    warmup_iters=500,
    warmup_ratio=1.0 / 3,
    step=[36, 39])
checkpoint_config = dict(interval=1)
# yapf:disable
log_config = dict(
    interval=50,
    hooks=[
        dict(type='TextLoggerHook'),
        # dict(type='TensorboardLoggerHook')
    ])
# yapf:enable
# runtime settings
total_epochs = 40
dist_params = dict(backend='nccl')
log_level = 'INFO'
work_dir = './work_dirs/DetectoRS_mstrain_400_1200_x101_32x4d_40e.py'
load_from = None
resume_from = None
workflow = [('train', 1)]

2020-06-09 04:25:02,066 - mmdet - INFO - load model from: open-mmlab://resnext101_32x4d
Downloading: "https://open-mmlab.s3.ap-northeast-2.amazonaws.com/pretrain/third_party/resnext101_32x4d-a5af3160.pth" to /root/.cache/torch/checkpoints/resnext101_32x4d-a5af3160.pth
100% 161M/161M [00:05<00:00, 33.0MB/s]
2020-06-09 04:25:08,172 - mmdet - WARNING - The model and loaded state dict do not match exactly

missing keys in source state_dict: conv1.weight_gamma, conv1.weight_beta, layer1.0.conv1.weight_gamma, layer1.0.conv1.weight_beta, layer1.0.conv2.weight_gamma, layer1.0.conv2.weight_beta, layer1.0.conv3.weight_gamma, layer1.0.conv3.weight_beta, layer1.0.downsample.0.weight_gamma, layer1.0.downsample.0.weight_beta, layer1.1.conv1.weight_gamma, layer1.1.conv1.weight_beta, layer1.1.conv2.weight_gamma, layer1.1.conv2.weight_beta, layer1.1.conv3.weight_gamma, layer1.1.conv3.weight_beta, layer1.2.conv1.weight_gamma, layer1.2.conv1.weight_beta, layer1.2.conv2.weight_gamma, layer1.2.conv2.weight_beta, layer1.2.conv3.weight_gamma, layer1.2.conv3.weight_beta, layer2.0.conv1.weight_gamma, layer2.0.conv1.weight_beta, layer2.0.conv2.weight_diff, layer2.0.conv2.weight_gamma, layer2.0.conv2.weight_beta, layer2.0.conv2.switch.weight, layer2.0.conv2.switch.bias, layer2.0.conv2.pre_context.weight, layer2.0.conv2.pre_context.bias, layer2.0.conv2.post_context.weight, layer2.0.conv2.post_context.bias, layer2.0.conv3.weight_gamma, layer2.0.conv3.weight_beta, layer2.0.downsample.0.weight_gamma, layer2.0.downsample.0.weight_beta, layer2.1.conv1.weight_gamma, layer2.1.conv1.weight_beta, layer2.1.conv2.weight_diff, layer2.1.conv2.weight_gamma, layer2.1.conv2.weight_beta, layer2.1.conv2.switch.weight, layer2.1.conv2.switch.bias, layer2.1.conv2.pre_context.weight, layer2.1.conv2.pre_context.bias, layer2.1.conv2.post_context.weight, layer2.1.conv2.post_context.bias, layer2.1.conv3.weight_gamma, layer2.1.conv3.weight_beta, layer2.2.conv1.weight_gamma, layer2.2.conv1.weight_beta, layer2.2.conv2.weight_diff, layer2.2.conv2.weight_gamma, layer2.2.conv2.weight_beta, layer2.2.conv2.switch.weight, layer2.2.conv2.switch.bias, layer2.2.conv2.pre_context.weight, layer2.2.conv2.pre_context.bias, layer2.2.conv2.post_context.weight, layer2.2.conv2.post_context.bias, layer2.2.conv3.weight_gamma, layer2.2.conv3.weight_beta, layer2.3.conv1.weight_gamma, layer2.3.conv1.weight_beta, layer2.3.conv2.weight_diff, layer2.3.conv2.weight_gamma, layer2.3.conv2.weight_beta, layer2.3.conv2.switch.weight, layer2.3.conv2.switch.bias, layer2.3.conv2.pre_context.weight, layer2.3.conv2.pre_context.bias, layer2.3.conv2.post_context.weight, layer2.3.conv2.post_context.bias, layer2.3.conv3.weight_gamma, layer2.3.conv3.weight_beta, layer3.0.conv1.weight_gamma, layer3.0.conv1.weight_beta, layer3.0.conv2.weight_diff, layer3.0.conv2.weight_gamma, layer3.0.conv2.weight_beta, layer3.0.conv2.switch.weight, layer3.0.conv2.switch.bias, layer3.0.conv2.pre_context.weight, layer3.0.conv2.pre_context.bias, layer3.0.conv2.post_context.weight, layer3.0.conv2.post_context.bias, layer3.0.conv3.weight_gamma, layer3.0.conv3.weight_beta, layer3.0.downsample.0.weight_gamma, layer3.0.downsample.0.weight_beta, layer3.1.conv1.weight_gamma, layer3.1.conv1.weight_beta, layer3.1.conv2.weight_diff, layer3.1.conv2.weight_gamma, layer3.1.conv2.weight_beta, layer3.1.conv2.switch.weight, layer3.1.conv2.switch.bias, layer3.1.conv2.pre_context.weight, layer3.1.conv2.pre_context.bias, layer3.1.conv2.post_context.weight, layer3.1.conv2.post_context.bias, layer3.1.conv3.weight_gamma, layer3.1.conv3.weight_beta, layer3.2.conv1.weight_gamma, layer3.2.conv1.weight_beta, layer3.2.conv2.weight_diff, layer3.2.conv2.weight_gamma, layer3.2.conv2.weight_beta, layer3.2.conv2.switch.weight, layer3.2.conv2.switch.bias, layer3.2.conv2.pre_context.weight, layer3.2.conv2.pre_context.bias, layer3.2.conv2.post_context.weight, layer3.2.conv2.post_context.bias, layer3.2.conv3.weight_gamma, layer3.2.conv3.weight_beta, layer3.3.conv1.weight_gamma, layer3.3.conv1.weight_beta, layer3.3.conv2.weight_diff, layer3.3.conv2.weight_gamma, layer3.3.conv2.weight_beta, layer3.3.conv2.switch.weight, layer3.3.conv2.switch.bias, layer3.3.conv2.pre_context.weight, layer3.3.conv2.pre_context.bias, layer3.3.conv2.post_context.weight, layer3.3.conv2.post_context.bias, layer3.3.conv3.weight_gamma, layer3.3.conv3.weight_beta, layer3.4.conv1.weight_gamma, layer3.4.conv1.weight_beta, layer3.4.conv2.weight_diff, layer3.4.conv2.weight_gamma, layer3.4.conv2.weight_beta, layer3.4.conv2.switch.weight, layer3.4.conv2.switch.bias, layer3.4.conv2.pre_context.weight, layer3.4.conv2.pre_context.bias, layer3.4.conv2.post_context.weight, layer3.4.conv2.post_context.bias, layer3.4.conv3.weight_gamma, layer3.4.conv3.weight_beta, layer3.5.conv1.weight_gamma, layer3.5.conv1.weight_beta, layer3.5.conv2.weight_diff, layer3.5.conv2.weight_gamma, layer3.5.conv2.weight_beta, layer3.5.conv2.switch.weight, layer3.5.conv2.switch.bias, layer3.5.conv2.pre_context.weight, layer3.5.conv2.pre_context.bias, layer3.5.conv2.post_context.weight, layer3.5.conv2.post_context.bias, layer3.5.conv3.weight_gamma, layer3.5.conv3.weight_beta, layer3.6.conv1.weight_gamma, layer3.6.conv1.weight_beta, layer3.6.conv2.weight_diff, layer3.6.conv2.weight_gamma, layer3.6.conv2.weight_beta, layer3.6.conv2.switch.weight, layer3.6.conv2.switch.bias, layer3.6.conv2.pre_context.weight, layer3.6.conv2.pre_context.bias, layer3.6.conv2.post_context.weight, layer3.6.conv2.post_context.bias, layer3.6.conv3.weight_gamma, layer3.6.conv3.weight_beta, layer3.7.conv1.weight_gamma, layer3.7.conv1.weight_beta, layer3.7.conv2.weight_diff, layer3.7.conv2.weight_gamma, layer3.7.conv2.weight_beta, layer3.7.conv2.switch.weight, layer3.7.conv2.switch.bias, layer3.7.conv2.pre_context.weight, layer3.7.conv2.pre_context.bias, layer3.7.conv2.post_context.weight, layer3.7.conv2.post_context.bias, layer3.7.conv3.weight_gamma, layer3.7.conv3.weight_beta, layer3.8.conv1.weight_gamma, layer3.8.conv1.weight_beta, layer3.8.conv2.weight_diff, layer3.8.conv2.weight_gamma, layer3.8.conv2.weight_beta, layer3.8.conv2.switch.weight, layer3.8.conv2.switch.bias, layer3.8.conv2.pre_context.weight, layer3.8.conv2.pre_context.bias, layer3.8.conv2.post_context.weight, layer3.8.conv2.post_context.bias, layer3.8.conv3.weight_gamma, layer3.8.conv3.weight_beta, layer3.9.conv1.weight_gamma, layer3.9.conv1.weight_beta, layer3.9.conv2.weight_diff, layer3.9.conv2.weight_gamma, layer3.9.conv2.weight_beta, layer3.9.conv2.switch.weight, layer3.9.conv2.switch.bias, layer3.9.conv2.pre_context.weight, layer3.9.conv2.pre_context.bias, layer3.9.conv2.post_context.weight, layer3.9.conv2.post_context.bias, layer3.9.conv3.weight_gamma, layer3.9.conv3.weight_beta, layer3.10.conv1.weight_gamma, layer3.10.conv1.weight_beta, layer3.10.conv2.weight_diff, layer3.10.conv2.weight_gamma, layer3.10.conv2.weight_beta, layer3.10.conv2.switch.weight, layer3.10.conv2.switch.bias, layer3.10.conv2.pre_context.weight, layer3.10.conv2.pre_context.bias, layer3.10.conv2.post_context.weight, layer3.10.conv2.post_context.bias, layer3.10.conv3.weight_gamma, layer3.10.conv3.weight_beta, layer3.11.conv1.weight_gamma, layer3.11.conv1.weight_beta, layer3.11.conv2.weight_diff, layer3.11.conv2.weight_gamma, layer3.11.conv2.weight_beta, layer3.11.conv2.switch.weight, layer3.11.conv2.switch.bias, layer3.11.conv2.pre_context.weight, layer3.11.conv2.pre_context.bias, layer3.11.conv2.post_context.weight, layer3.11.conv2.post_context.bias, layer3.11.conv3.weight_gamma, layer3.11.conv3.weight_beta, layer3.12.conv1.weight_gamma, layer3.12.conv1.weight_beta, layer3.12.conv2.weight_diff, layer3.12.conv2.weight_gamma, layer3.12.conv2.weight_beta, layer3.12.conv2.switch.weight, layer3.12.conv2.switch.bias, layer3.12.conv2.pre_context.weight, layer3.12.conv2.pre_context.bias, layer3.12.conv2.post_context.weight, layer3.12.conv2.post_context.bias, layer3.12.conv3.weight_gamma, layer3.12.conv3.weight_beta, layer3.13.conv1.weight_gamma, layer3.13.conv1.weight_beta, layer3.13.conv2.weight_diff, layer3.13.conv2.weight_gamma, layer3.13.conv2.weight_beta, layer3.13.conv2.switch.weight, layer3.13.conv2.switch.bias, layer3.13.conv2.pre_context.weight, layer3.13.conv2.pre_context.bias, layer3.13.conv2.post_context.weight, layer3.13.conv2.post_context.bias, layer3.13.conv3.weight_gamma, layer3.13.conv3.weight_beta, layer3.14.conv1.weight_gamma, layer3.14.conv1.weight_beta, layer3.14.conv2.weight_diff, layer3.14.conv2.weight_gamma, layer3.14.conv2.weight_beta, layer3.14.conv2.switch.weight, layer3.14.conv2.switch.bias, layer3.14.conv2.pre_context.weight, layer3.14.conv2.pre_context.bias, layer3.14.conv2.post_context.weight, layer3.14.conv2.post_context.bias, layer3.14.conv3.weight_gamma, layer3.14.conv3.weight_beta, layer3.15.conv1.weight_gamma, layer3.15.conv1.weight_beta, layer3.15.conv2.weight_diff, layer3.15.conv2.weight_gamma, layer3.15.conv2.weight_beta, layer3.15.conv2.switch.weight, layer3.15.conv2.switch.bias, layer3.15.conv2.pre_context.weight, layer3.15.conv2.pre_context.bias, layer3.15.conv2.post_context.weight, layer3.15.conv2.post_context.bias, layer3.15.conv3.weight_gamma, layer3.15.conv3.weight_beta, layer3.16.conv1.weight_gamma, layer3.16.conv1.weight_beta, layer3.16.conv2.weight_diff, layer3.16.conv2.weight_gamma, layer3.16.conv2.weight_beta, layer3.16.conv2.switch.weight, layer3.16.conv2.switch.bias, layer3.16.conv2.pre_context.weight, layer3.16.conv2.pre_context.bias, layer3.16.conv2.post_context.weight, layer3.16.conv2.post_context.bias, layer3.16.conv3.weight_gamma, layer3.16.conv3.weight_beta, layer3.17.conv1.weight_gamma, layer3.17.conv1.weight_beta, layer3.17.conv2.weight_diff, layer3.17.conv2.weight_gamma, layer3.17.conv2.weight_beta, layer3.17.conv2.switch.weight, layer3.17.conv2.switch.bias, layer3.17.conv2.pre_context.weight, layer3.17.conv2.pre_context.bias, layer3.17.conv2.post_context.weight, layer3.17.conv2.post_context.bias, layer3.17.conv3.weight_gamma, layer3.17.conv3.weight_beta, layer3.18.conv1.weight_gamma, layer3.18.conv1.weight_beta, layer3.18.conv2.weight_diff, layer3.18.conv2.weight_gamma, layer3.18.conv2.weight_beta, layer3.18.conv2.switch.weight, layer3.18.conv2.switch.bias, layer3.18.conv2.pre_context.weight, layer3.18.conv2.pre_context.bias, layer3.18.conv2.post_context.weight, layer3.18.conv2.post_context.bias, layer3.18.conv3.weight_gamma, layer3.18.conv3.weight_beta, layer3.19.conv1.weight_gamma, layer3.19.conv1.weight_beta, layer3.19.conv2.weight_diff, layer3.19.conv2.weight_gamma, layer3.19.conv2.weight_beta, layer3.19.conv2.switch.weight, layer3.19.conv2.switch.bias, layer3.19.conv2.pre_context.weight, layer3.19.conv2.pre_context.bias, layer3.19.conv2.post_context.weight, layer3.19.conv2.post_context.bias, layer3.19.conv3.weight_gamma, layer3.19.conv3.weight_beta, layer3.20.conv1.weight_gamma, layer3.20.conv1.weight_beta, layer3.20.conv2.weight_diff, layer3.20.conv2.weight_gamma, layer3.20.conv2.weight_beta, layer3.20.conv2.switch.weight, layer3.20.conv2.switch.bias, layer3.20.conv2.pre_context.weight, layer3.20.conv2.pre_context.bias, layer3.20.conv2.post_context.weight, layer3.20.conv2.post_context.bias, layer3.20.conv3.weight_gamma, layer3.20.conv3.weight_beta, layer3.21.conv1.weight_gamma, layer3.21.conv1.weight_beta, layer3.21.conv2.weight_diff, layer3.21.conv2.weight_gamma, layer3.21.conv2.weight_beta, layer3.21.conv2.switch.weight, layer3.21.conv2.switch.bias, layer3.21.conv2.pre_context.weight, layer3.21.conv2.pre_context.bias, layer3.21.conv2.post_context.weight, layer3.21.conv2.post_context.bias, layer3.21.conv3.weight_gamma, layer3.21.conv3.weight_beta, layer3.22.conv1.weight_gamma, layer3.22.conv1.weight_beta, layer3.22.conv2.weight_diff, layer3.22.conv2.weight_gamma, layer3.22.conv2.weight_beta, layer3.22.conv2.switch.weight, layer3.22.conv2.switch.bias, layer3.22.conv2.pre_context.weight, layer3.22.conv2.pre_context.bias, layer3.22.conv2.post_context.weight, layer3.22.conv2.post_context.bias, layer3.22.conv3.weight_gamma, layer3.22.conv3.weight_beta, layer4.0.conv1.weight_gamma, layer4.0.conv1.weight_beta, layer4.0.conv2.weight_diff, layer4.0.conv2.weight_gamma, layer4.0.conv2.weight_beta, layer4.0.conv2.switch.weight, layer4.0.conv2.switch.bias, layer4.0.conv2.pre_context.weight, layer4.0.conv2.pre_context.bias, layer4.0.conv2.post_context.weight, layer4.0.conv2.post_context.bias, layer4.0.conv3.weight_gamma, layer4.0.conv3.weight_beta, layer4.0.downsample.0.weight_gamma, layer4.0.downsample.0.weight_beta, layer4.1.conv1.weight_gamma, layer4.1.conv1.weight_beta, layer4.1.conv2.weight_diff, layer4.1.conv2.weight_gamma, layer4.1.conv2.weight_beta, layer4.1.conv2.switch.weight, layer4.1.conv2.switch.bias, layer4.1.conv2.pre_context.weight, layer4.1.conv2.pre_context.bias, layer4.1.conv2.post_context.weight, layer4.1.conv2.post_context.bias, layer4.1.conv3.weight_gamma, layer4.1.conv3.weight_beta, layer4.2.conv1.weight_gamma, layer4.2.conv1.weight_beta, layer4.2.conv2.weight_diff, layer4.2.conv2.weight_gamma, layer4.2.conv2.weight_beta, layer4.2.conv2.switch.weight, layer4.2.conv2.switch.bias, layer4.2.conv2.pre_context.weight, layer4.2.conv2.pre_context.bias, layer4.2.conv2.post_context.weight, layer4.2.conv2.post_context.bias, layer4.2.conv3.weight_gamma, layer4.2.conv3.weight_beta

2020-06-09 04:25:11,017 - mmdet - WARNING - The model and loaded state dict do not match exactly

missing keys in source state_dict: conv1.weight_gamma, conv1.weight_beta, layer1.0.conv1.weight_gamma, layer1.0.conv1.weight_beta, layer1.0.conv2.weight_gamma, layer1.0.conv2.weight_beta, layer1.0.conv3.weight_gamma, layer1.0.conv3.weight_beta, layer1.0.downsample.0.weight_gamma, layer1.0.downsample.0.weight_beta, layer1.1.conv1.weight_gamma, layer1.1.conv1.weight_beta, layer1.1.conv2.weight_gamma, layer1.1.conv2.weight_beta, layer1.1.conv3.weight_gamma, layer1.1.conv3.weight_beta, layer1.2.conv1.weight_gamma, layer1.2.conv1.weight_beta, layer1.2.conv2.weight_gamma, layer1.2.conv2.weight_beta, layer1.2.conv3.weight_gamma, layer1.2.conv3.weight_beta, layer2.0.conv1.weight_gamma, layer2.0.conv1.weight_beta, layer2.0.conv2.weight_diff, layer2.0.conv2.weight_gamma, layer2.0.conv2.weight_beta, layer2.0.conv2.switch.weight, layer2.0.conv2.switch.bias, layer2.0.conv2.pre_context.weight, layer2.0.conv2.pre_context.bias, layer2.0.conv2.post_context.weight, layer2.0.conv2.post_context.bias, layer2.0.conv3.weight_gamma, layer2.0.conv3.weight_beta, layer2.0.downsample.0.weight_gamma, layer2.0.downsample.0.weight_beta, layer2.0.rfp_conv.weight, layer2.0.rfp_conv.bias, layer2.1.conv1.weight_gamma, layer2.1.conv1.weight_beta, layer2.1.conv2.weight_diff, layer2.1.conv2.weight_gamma, layer2.1.conv2.weight_beta, layer2.1.conv2.switch.weight, layer2.1.conv2.switch.bias, layer2.1.conv2.pre_context.weight, layer2.1.conv2.pre_context.bias, layer2.1.conv2.post_context.weight, layer2.1.conv2.post_context.bias, layer2.1.conv3.weight_gamma, layer2.1.conv3.weight_beta, layer2.2.conv1.weight_gamma, layer2.2.conv1.weight_beta, layer2.2.conv2.weight_diff, layer2.2.conv2.weight_gamma, layer2.2.conv2.weight_beta, layer2.2.conv2.switch.weight, layer2.2.conv2.switch.bias, layer2.2.conv2.pre_context.weight, layer2.2.conv2.pre_context.bias, layer2.2.conv2.post_context.weight, layer2.2.conv2.post_context.bias, layer2.2.conv3.weight_gamma, layer2.2.conv3.weight_beta, layer2.3.conv1.weight_gamma, layer2.3.conv1.weight_beta, layer2.3.conv2.weight_diff, layer2.3.conv2.weight_gamma, layer2.3.conv2.weight_beta, layer2.3.conv2.switch.weight, layer2.3.conv2.switch.bias, layer2.3.conv2.pre_context.weight, layer2.3.conv2.pre_context.bias, layer2.3.conv2.post_context.weight, layer2.3.conv2.post_context.bias, layer2.3.conv3.weight_gamma, layer2.3.conv3.weight_beta, layer3.0.conv1.weight_gamma, layer3.0.conv1.weight_beta, layer3.0.conv2.weight_diff, layer3.0.conv2.weight_gamma, layer3.0.conv2.weight_beta, layer3.0.conv2.switch.weight, layer3.0.conv2.switch.bias, layer3.0.conv2.pre_context.weight, layer3.0.conv2.pre_context.bias, layer3.0.conv2.post_context.weight, layer3.0.conv2.post_context.bias, layer3.0.conv3.weight_gamma, layer3.0.conv3.weight_beta, layer3.0.downsample.0.weight_gamma, layer3.0.downsample.0.weight_beta, layer3.0.rfp_conv.weight, layer3.0.rfp_conv.bias, layer3.1.conv1.weight_gamma, layer3.1.conv1.weight_beta, layer3.1.conv2.weight_diff, layer3.1.conv2.weight_gamma, layer3.1.conv2.weight_beta, layer3.1.conv2.switch.weight, layer3.1.conv2.switch.bias, layer3.1.conv2.pre_context.weight, layer3.1.conv2.pre_context.bias, layer3.1.conv2.post_context.weight, layer3.1.conv2.post_context.bias, layer3.1.conv3.weight_gamma, layer3.1.conv3.weight_beta, layer3.2.conv1.weight_gamma, layer3.2.conv1.weight_beta, layer3.2.conv2.weight_diff, layer3.2.conv2.weight_gamma, layer3.2.conv2.weight_beta, layer3.2.conv2.switch.weight, layer3.2.conv2.switch.bias, layer3.2.conv2.pre_context.weight, layer3.2.conv2.pre_context.bias, layer3.2.conv2.post_context.weight, layer3.2.conv2.post_context.bias, layer3.2.conv3.weight_gamma, layer3.2.conv3.weight_beta, layer3.3.conv1.weight_gamma, layer3.3.conv1.weight_beta, layer3.3.conv2.weight_diff, layer3.3.conv2.weight_gamma, layer3.3.conv2.weight_beta, layer3.3.conv2.switch.weight, layer3.3.conv2.switch.bias, layer3.3.conv2.pre_context.weight, layer3.3.conv2.pre_context.bias, layer3.3.conv2.post_context.weight, layer3.3.conv2.post_context.bias, layer3.3.conv3.weight_gamma, layer3.3.conv3.weight_beta, layer3.4.conv1.weight_gamma, layer3.4.conv1.weight_beta, layer3.4.conv2.weight_diff, layer3.4.conv2.weight_gamma, layer3.4.conv2.weight_beta, layer3.4.conv2.switch.weight, layer3.4.conv2.switch.bias, layer3.4.conv2.pre_context.weight, layer3.4.conv2.pre_context.bias, layer3.4.conv2.post_context.weight, layer3.4.conv2.post_context.bias, layer3.4.conv3.weight_gamma, layer3.4.conv3.weight_beta, layer3.5.conv1.weight_gamma, layer3.5.conv1.weight_beta, layer3.5.conv2.weight_diff, layer3.5.conv2.weight_gamma, layer3.5.conv2.weight_beta, layer3.5.conv2.switch.weight, layer3.5.conv2.switch.bias, layer3.5.conv2.pre_context.weight, layer3.5.conv2.pre_context.bias, layer3.5.conv2.post_context.weight, layer3.5.conv2.post_context.bias, layer3.5.conv3.weight_gamma, layer3.5.conv3.weight_beta, layer3.6.conv1.weight_gamma, layer3.6.conv1.weight_beta, layer3.6.conv2.weight_diff, layer3.6.conv2.weight_gamma, layer3.6.conv2.weight_beta, layer3.6.conv2.switch.weight, layer3.6.conv2.switch.bias, layer3.6.conv2.pre_context.weight, layer3.6.conv2.pre_context.bias, layer3.6.conv2.post_context.weight, layer3.6.conv2.post_context.bias, layer3.6.conv3.weight_gamma, layer3.6.conv3.weight_beta, layer3.7.conv1.weight_gamma, layer3.7.conv1.weight_beta, layer3.7.conv2.weight_diff, layer3.7.conv2.weight_gamma, layer3.7.conv2.weight_beta, layer3.7.conv2.switch.weight, layer3.7.conv2.switch.bias, layer3.7.conv2.pre_context.weight, layer3.7.conv2.pre_context.bias, layer3.7.conv2.post_context.weight, layer3.7.conv2.post_context.bias, layer3.7.conv3.weight_gamma, layer3.7.conv3.weight_beta, layer3.8.conv1.weight_gamma, layer3.8.conv1.weight_beta, layer3.8.conv2.weight_diff, layer3.8.conv2.weight_gamma, layer3.8.conv2.weight_beta, layer3.8.conv2.switch.weight, layer3.8.conv2.switch.bias, layer3.8.conv2.pre_context.weight, layer3.8.conv2.pre_context.bias, layer3.8.conv2.post_context.weight, layer3.8.conv2.post_context.bias, layer3.8.conv3.weight_gamma, layer3.8.conv3.weight_beta, layer3.9.conv1.weight_gamma, layer3.9.conv1.weight_beta, layer3.9.conv2.weight_diff, layer3.9.conv2.weight_gamma, layer3.9.conv2.weight_beta, layer3.9.conv2.switch.weight, layer3.9.conv2.switch.bias, layer3.9.conv2.pre_context.weight, layer3.9.conv2.pre_context.bias, layer3.9.conv2.post_context.weight, layer3.9.conv2.post_context.bias, layer3.9.conv3.weight_gamma, layer3.9.conv3.weight_beta, layer3.10.conv1.weight_gamma, layer3.10.conv1.weight_beta, layer3.10.conv2.weight_diff, layer3.10.conv2.weight_gamma, layer3.10.conv2.weight_beta, layer3.10.conv2.switch.weight, layer3.10.conv2.switch.bias, layer3.10.conv2.pre_context.weight, layer3.10.conv2.pre_context.bias, layer3.10.conv2.post_context.weight, layer3.10.conv2.post_context.bias, layer3.10.conv3.weight_gamma, layer3.10.conv3.weight_beta, layer3.11.conv1.weight_gamma, layer3.11.conv1.weight_beta, layer3.11.conv2.weight_diff, layer3.11.conv2.weight_gamma, layer3.11.conv2.weight_beta, layer3.11.conv2.switch.weight, layer3.11.conv2.switch.bias, layer3.11.conv2.pre_context.weight, layer3.11.conv2.pre_context.bias, layer3.11.conv2.post_context.weight, layer3.11.conv2.post_context.bias, layer3.11.conv3.weight_gamma, layer3.11.conv3.weight_beta, layer3.12.conv1.weight_gamma, layer3.12.conv1.weight_beta, layer3.12.conv2.weight_diff, layer3.12.conv2.weight_gamma, layer3.12.conv2.weight_beta, layer3.12.conv2.switch.weight, layer3.12.conv2.switch.bias, layer3.12.conv2.pre_context.weight, layer3.12.conv2.pre_context.bias, layer3.12.conv2.post_context.weight, layer3.12.conv2.post_context.bias, layer3.12.conv3.weight_gamma, layer3.12.conv3.weight_beta, layer3.13.conv1.weight_gamma, layer3.13.conv1.weight_beta, layer3.13.conv2.weight_diff, layer3.13.conv2.weight_gamma, layer3.13.conv2.weight_beta, layer3.13.conv2.switch.weight, layer3.13.conv2.switch.bias, layer3.13.conv2.pre_context.weight, layer3.13.conv2.pre_context.bias, layer3.13.conv2.post_context.weight, layer3.13.conv2.post_context.bias, layer3.13.conv3.weight_gamma, layer3.13.conv3.weight_beta, layer3.14.conv1.weight_gamma, layer3.14.conv1.weight_beta, layer3.14.conv2.weight_diff, layer3.14.conv2.weight_gamma, layer3.14.conv2.weight_beta, layer3.14.conv2.switch.weight, layer3.14.conv2.switch.bias, layer3.14.conv2.pre_context.weight, layer3.14.conv2.pre_context.bias, layer3.14.conv2.post_context.weight, layer3.14.conv2.post_context.bias, layer3.14.conv3.weight_gamma, layer3.14.conv3.weight_beta, layer3.15.conv1.weight_gamma, layer3.15.conv1.weight_beta, layer3.15.conv2.weight_diff, layer3.15.conv2.weight_gamma, layer3.15.conv2.weight_beta, layer3.15.conv2.switch.weight, layer3.15.conv2.switch.bias, layer3.15.conv2.pre_context.weight, layer3.15.conv2.pre_context.bias, layer3.15.conv2.post_context.weight, layer3.15.conv2.post_context.bias, layer3.15.conv3.weight_gamma, layer3.15.conv3.weight_beta, layer3.16.conv1.weight_gamma, layer3.16.conv1.weight_beta, layer3.16.conv2.weight_diff, layer3.16.conv2.weight_gamma, layer3.16.conv2.weight_beta, layer3.16.conv2.switch.weight, layer3.16.conv2.switch.bias, layer3.16.conv2.pre_context.weight, layer3.16.conv2.pre_context.bias, layer3.16.conv2.post_context.weight, layer3.16.conv2.post_context.bias, layer3.16.conv3.weight_gamma, layer3.16.conv3.weight_beta, layer3.17.conv1.weight_gamma, layer3.17.conv1.weight_beta, layer3.17.conv2.weight_diff, layer3.17.conv2.weight_gamma, layer3.17.conv2.weight_beta, layer3.17.conv2.switch.weight, layer3.17.conv2.switch.bias, layer3.17.conv2.pre_context.weight, layer3.17.conv2.pre_context.bias, layer3.17.conv2.post_context.weight, layer3.17.conv2.post_context.bias, layer3.17.conv3.weight_gamma, layer3.17.conv3.weight_beta, layer3.18.conv1.weight_gamma, layer3.18.conv1.weight_beta, layer3.18.conv2.weight_diff, layer3.18.conv2.weight_gamma, layer3.18.conv2.weight_beta, layer3.18.conv2.switch.weight, layer3.18.conv2.switch.bias, layer3.18.conv2.pre_context.weight, layer3.18.conv2.pre_context.bias, layer3.18.conv2.post_context.weight, layer3.18.conv2.post_context.bias, layer3.18.conv3.weight_gamma, layer3.18.conv3.weight_beta, layer3.19.conv1.weight_gamma, layer3.19.conv1.weight_beta, layer3.19.conv2.weight_diff, layer3.19.conv2.weight_gamma, layer3.19.conv2.weight_beta, layer3.19.conv2.switch.weight, layer3.19.conv2.switch.bias, layer3.19.conv2.pre_context.weight, layer3.19.conv2.pre_context.bias, layer3.19.conv2.post_context.weight, layer3.19.conv2.post_context.bias, layer3.19.conv3.weight_gamma, layer3.19.conv3.weight_beta, layer3.20.conv1.weight_gamma, layer3.20.conv1.weight_beta, layer3.20.conv2.weight_diff, layer3.20.conv2.weight_gamma, layer3.20.conv2.weight_beta, layer3.20.conv2.switch.weight, layer3.20.conv2.switch.bias, layer3.20.conv2.pre_context.weight, layer3.20.conv2.pre_context.bias, layer3.20.conv2.post_context.weight, layer3.20.conv2.post_context.bias, layer3.20.conv3.weight_gamma, layer3.20.conv3.weight_beta, layer3.21.conv1.weight_gamma, layer3.21.conv1.weight_beta, layer3.21.conv2.weight_diff, layer3.21.conv2.weight_gamma, layer3.21.conv2.weight_beta, layer3.21.conv2.switch.weight, layer3.21.conv2.switch.bias, layer3.21.conv2.pre_context.weight, layer3.21.conv2.pre_context.bias, layer3.21.conv2.post_context.weight, layer3.21.conv2.post_context.bias, layer3.21.conv3.weight_gamma, layer3.21.conv3.weight_beta, layer3.22.conv1.weight_gamma, layer3.22.conv1.weight_beta, layer3.22.conv2.weight_diff, layer3.22.conv2.weight_gamma, layer3.22.conv2.weight_beta, layer3.22.conv2.switch.weight, layer3.22.conv2.switch.bias, layer3.22.conv2.pre_context.weight, layer3.22.conv2.pre_context.bias, layer3.22.conv2.post_context.weight, layer3.22.conv2.post_context.bias, layer3.22.conv3.weight_gamma, layer3.22.conv3.weight_beta, layer4.0.conv1.weight_gamma, layer4.0.conv1.weight_beta, layer4.0.conv2.weight_diff, layer4.0.conv2.weight_gamma, layer4.0.conv2.weight_beta, layer4.0.conv2.switch.weight, layer4.0.conv2.switch.bias, layer4.0.conv2.pre_context.weight, layer4.0.conv2.pre_context.bias, layer4.0.conv2.post_context.weight, layer4.0.conv2.post_context.bias, layer4.0.conv3.weight_gamma, layer4.0.conv3.weight_beta, layer4.0.downsample.0.weight_gamma, layer4.0.downsample.0.weight_beta, layer4.0.rfp_conv.weight, layer4.0.rfp_conv.bias, layer4.1.conv1.weight_gamma, layer4.1.conv1.weight_beta, layer4.1.conv2.weight_diff, layer4.1.conv2.weight_gamma, layer4.1.conv2.weight_beta, layer4.1.conv2.switch.weight, layer4.1.conv2.switch.bias, layer4.1.conv2.pre_context.weight, layer4.1.conv2.pre_context.bias, layer4.1.conv2.post_context.weight, layer4.1.conv2.post_context.bias, layer4.1.conv3.weight_gamma, layer4.1.conv3.weight_beta, layer4.2.conv1.weight_gamma, layer4.2.conv1.weight_beta, layer4.2.conv2.weight_diff, layer4.2.conv2.weight_gamma, layer4.2.conv2.weight_beta, layer4.2.conv2.switch.weight, layer4.2.conv2.switch.bias, layer4.2.conv2.pre_context.weight, layer4.2.conv2.pre_context.bias, layer4.2.conv2.post_context.weight, layer4.2.conv2.post_context.bias, layer4.2.conv3.weight_gamma, layer4.2.conv3.weight_beta

loading annotations into memory...
Done (t=5.79s)
creating index...
index created!
2020-06-09 04:25:19,269 - mmdet - INFO - Start running, host: root@e04ecbefda9f, work_dir: /content/drive/My Drive/DetectoRS/work_dirs/DetectoRS_mstrain_400_1200_x101_32x4d_40e.py
2020-06-09 04:25:19,270 - mmdet - INFO - workflow: [('train', 1)], max: 40 epochs
tensor([-1, -1, -1,  ..., -1, -1, -1], device='cuda:0') before
tensor([0, 0, 0,  ..., 0, 0, 0], device='cuda:0') after
None
torch.Size([1, 1, 60, 80, 3]) before
torch.Size([1, 60, 80, 3]) after
torch.Size([1, 60, 80]) after modification
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [864,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [865,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [866,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [867,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [868,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [869,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [870,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [871,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [872,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [873,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [874,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [875,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [876,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [877,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [878,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [879,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [880,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [881,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [882,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [883,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [884,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [885,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [886,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [887,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [888,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [889,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [890,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [891,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [892,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [893,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [894,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [895,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [640,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [641,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [642,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [643,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [644,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [645,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [646,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [647,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [648,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [649,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [650,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [651,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [652,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [653,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [654,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [655,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [656,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [657,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [658,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [659,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [660,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [661,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [662,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [663,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [664,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [665,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [666,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [667,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [668,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [669,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [670,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [671,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [448,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [449,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [450,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [451,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [452,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [453,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [454,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [455,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [456,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [457,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [458,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [459,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [460,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [461,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [462,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [463,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [464,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [465,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [466,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [467,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [468,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [469,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [470,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [471,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [472,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [473,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [474,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [475,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [476,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [477,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [478,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [479,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [128,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [129,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [130,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [131,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [132,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [133,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [134,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [135,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [136,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [137,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [138,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [139,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [140,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [141,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [142,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [143,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [144,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [145,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [146,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [147,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [148,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [149,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [150,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [151,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [152,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [153,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [154,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [155,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [156,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [157,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [158,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:104: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [159,0,0] Assertion `t >= 0 && t < n_classes` failed.
Traceback (most recent call last):
  File "/content/drive/My Drive/DetectoRS/tools/train.py", line 142, in <module>
    main()
  File "/content/drive/My Drive/DetectoRS/tools/train.py", line 138, in main
    meta=meta)
  File "/content/drive/My Drive/DetectoRS/mmdet/apis/train.py", line 111, in train_detector
    meta=meta)
  File "/content/drive/My Drive/DetectoRS/mmdet/apis/train.py", line 242, in _non_dist_train
    runner.run(data_loaders, cfg.workflow, cfg.total_epochs)
  File "/usr/local/lib/python3.6/dist-packages/mmcv/runner/runner.py", line 384, in run
    epoch_runner(data_loaders[i], **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/mmcv/runner/runner.py", line 283, in train
    self.model, data_batch, train_mode=True, **kwargs)
  File "/content/drive/My Drive/DetectoRS/mmdet/apis/train.py", line 75, in batch_processor
    losses = model(**data)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/data_parallel.py", line 150, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/content/drive/My Drive/DetectoRS/mmdet/core/fp16/decorators.py", line 49, in new_func
    return old_func(*args, **kwargs)
  File "/content/drive/My Drive/DetectoRS/mmdet/models/detectors/base.py", line 147, in forward
    return self.forward_train(img, img_metas, **kwargs)
  File "/content/drive/My Drive/DetectoRS/mmdet/models/detectors/htc.py", line 252, in forward_train
    gt_labels[j])
  File "/content/drive/My Drive/DetectoRS/mmdet/core/bbox/assigners/max_iou_assigner.py", line 113, in assign
    assign_result = self.assign_wrt_overlaps(overlaps, gt_labels)
  File "/content/drive/My Drive/DetectoRS/mmdet/core/bbox/assigners/max_iou_assigner.py", line 165, in assign_wrt_overlaps
    print(assigned_gt_inds,'before')
  File "/usr/local/lib/python3.6/dist-packages/torch/tensor.py", line 159, in __repr__
    return torch._tensor_str._str(self)
  File "/usr/local/lib/python3.6/dist-packages/torch/_tensor_str.py", line 311, in _str
    tensor_str = _tensor_str(self, indent)
  File "/usr/local/lib/python3.6/dist-packages/torch/_tensor_str.py", line 209, in _tensor_str
    formatter = _Formatter(get_summarized_data(self) if summarize else self)
  File "/usr/local/lib/python3.6/dist-packages/torch/_tensor_str.py", line 83, in __init__
    value_str = '{}'.format(value)
  File "/usr/local/lib/python3.6/dist-packages/torch/tensor.py", line 409, in __format__
    return self.item().__format__(format_spec)
RuntimeError: CUDA error: device-side assert triggered
terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: device-side assert triggered (insert_events at /pytorch/c10/cuda/CUDACachingAllocator.cpp:764)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x33 (0x7ffa1b4d9193 in /usr/local/lib/python3.6/dist-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x17f66 (0x7ffa1b716f66 in /usr/local/lib/python3.6/dist-packages/torch/lib/libc10_cuda.so)
frame #2: <unknown function> + 0x19cbd (0x7ffa1b718cbd in /usr/local/lib/python3.6/dist-packages/torch/lib/libc10_cuda.so)
frame #3: c10::TensorImpl::release_resources() + 0x4d (0x7ffa1b4c963d in /usr/local/lib/python3.6/dist-packages/torch/lib/libc10.so)
frame #4: <unknown function> + 0x67a902 (0x7ffa64500902 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
frame #5: <unknown function> + 0x67a9a6 (0x7ffa645009a6 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
frame #6: python3() [0x54ef16]
frame #7: python3() [0x572890]
frame #8: python3() [0x54ec1b]
frame #9: python3() [0x588948]
frame #10: python3() [0x5ad438]
frame #11: python3() [0x5ad44e]
frame #12: python3() [0x5ad44e]
frame #13: python3() [0x5ad44e]
frame #14: python3() [0x5ad44e]
frame #15: python3() [0x5ad44e]
frame #16: python3() [0x5ad44e]
frame #17: python3() [0x5ad44e]
frame #18: python3() [0x5ad44e]
frame #19: python3() [0x56b276]
<omitting python frames>
frame #25: __libc_start_main + 0xe7 (0x7ffa6fb70b97 in /lib/x86_64-linux-gnu/libc.so.6)
joe-siyuan-qiao commented 4 years ago

Please take a look at the following issue and see if it helps.

https://github.com/open-mmlab/mmdetection/issues/2639

MotiBaadror commented 4 years ago

I have only one class, num_class variable is also set to 1, any suggestion

joe-siyuan-qiao commented 4 years ago

It seems many people have also tried one class training. You might want to take a look at the following issues:

https://github.com/open-mmlab/mmdetection/issues/1366 https://github.com/open-mmlab/mmdetection/issues/1300 https://github.com/open-mmlab/mmdetection/issues/2288