When I was ran python experiments/fcis/fcis_end2end_train_test.py --cfg experiments/fcis/cfgs/resnet_v1_101_coco_fcis_end2end_ohem.yaml, I've been waiting, how can I fix this? Thank you!

[xingbowei]

xbw@xbw-P65xRP:~/FCIS$ python experiments/fcis/fcis_end2end_train_test.py --cfg experiments/fcis/cfgs/resnet_v1_101_coco_fcis_end2end_ohem.yaml Called with argument: Namespace(cfg='experiments/fcis/cfgs/resnet_v1_101_coco_fcis_end2end_ohem.yaml', frequent=100) {'BINARY_THRESH': 0.4, 'CLASS_AGNOSTIC': True, 'MASK_SIZE': 21, 'MXNET_VERSION': 'mxnet', 'SCALES': [(600, 1000)], 'TEST': {'BATCH_IMAGES': 1, 'CXX_PROPOSAL': False, 'HAS_RPN': True, 'ITER': 2, 'MASK_MERGE_THRESH': 0.5, 'MIN_DROP_SIZE': 2, 'NMS': 0.3, 'PROPOSAL_MIN_SIZE': 2, 'PROPOSAL_NMS_THRESH': 0.7, 'PROPOSAL_POST_NMS_TOP_N': 2000, 'PROPOSAL_PRE_NMS_TOP_N': 20000, 'RPN_MIN_SIZE': 2, 'RPN_NMS_THRESH': 0.7, 'RPN_POST_NMS_TOP_N': 300, 'RPN_PRE_NMS_TOP_N': 6000, 'USE_GPU_MASK_MERGE': True, 'USE_MASK_MERGE': True, 'test_epoch': 8}, 'TRAIN': {'ASPECT_GROUPING': True, 'BATCH_IMAGES': 1, 'BATCH_ROIS': -1, 'BATCH_ROIS_OHEM': 128, 'BBOX_MEANS': [0.0, 0.0, 0.0, 0.0], 'BBOX_NORMALIZATION_PRECOMPUTED': True, 'BBOX_REGRESSION_THRESH': 0.5, 'BBOX_STDS': [0.2, 0.2, 0.5, 0.5], 'BBOX_WEIGHTS': array([ 1., 1., 1., 1.]), 'BG_THRESH_HI': 0.5, 'BG_THRESH_LO': 0, 'BINARY_THRESH': 0.4, 'CONVNEW3': True, 'CXX_PROPOSAL': False, 'ENABLE_OHEM': True, 'END2END': True, 'FG_FRACTION': 0.25, 'FG_THRESH': 0.5, 'FLIP': True, 'GAP_SELECT_FROM_ALL': False, 'IGNORE_GAP': False, 'LOSS_WEIGHT': [1.0, 10.0, 1.0], 'RESUME': False, 'RPN_ALLOWED_BORDER': 0, 'RPN_BATCH_SIZE': 256, 'RPN_BBOX_WEIGHTS': [1.0, 1.0, 1.0, 1.0], 'RPN_CLOBBER_POSITIVES': False, 'RPN_FG_FRACTION': 0.5, 'RPN_MIN_SIZE': 2, 'RPN_NEGATIVE_OVERLAP': 0.3, 'RPN_NMS_THRESH': 0.7, 'RPN_POSITIVE_OVERLAP': 0.7, 'RPN_POSITIVE_WEIGHT': -1.0, 'RPN_POST_NMS_TOP_N': 300, 'RPN_PRE_NMS_TOP_N': 6000, 'SHUFFLE': True, 'begin_epoch': 0, 'end_epoch': 8, 'lr': 0.0005, 'lr_step': '5.33', 'model_prefix': 'e2e', 'momentum': 0.9, 'warmup': True, 'warmup_lr': 5e-05, 'warmup_step': 250, 'wd': 0.0005}, 'dataset': {'NUM_CLASSES': 81, 'dataset': 'coco', 'dataset_path': './data/coco', 'image_set': 'train2014+valminusminival2014', 'proposal': 'rpn', 'root_path': './data', 'test_image_set': 'test-dev2015'}, 'default': {'frequent': 100, 'kvstore': 'device'}, 'gpus': '0,1,2,3,4,5,6,7', 'network': {'ANCHOR_RATIOS': [0.5, 1, 2], 'ANCHOR_SCALES': [4, 8, 16, 32], 'FIXED_PARAMS': ['conv1', 'bn_conv1', 'res2', 'bn2', 'gamma', 'beta'], 'FIXED_PARAMS_SHARED': ['conv1', 'bn_conv1', 'res2', 'bn2', 'res3', 'bn3', 'res4', 'bn4', 'gamma', 'beta'], 'IMAGE_STRIDE': 0, 'NUM_ANCHORS': 12, 'PIXEL_MEANS': array([ 103.06, 115.9 , 123.15]), 'RCNN_FEAT_STRIDE': 16, 'RPN_FEAT_STRIDE': 16, 'pretrained': './model/pretrained_model/resnet_v1_101', 'pretrained_epoch': 0}, 'output_path': './output/fcis/coco', 'symbol': 'resnet_v1_101_fcis'} loading annotations into memory... Done (t=12.27s) creating index... index created! num_images 82783 prepare gt_sdsdb using 28.8503608704 seconds

[xingbowei]

I waited and it didn't respond

[encodingwaddles]

@xingbowei were you able to get pass this stage? I think you might need to wait a bit longer. I gave it half hour or so and it started training.

[yxchen99]

i encounter the same issue. it does not respond after printing 'prepare gt_sdsdb using 61.031017065 seconds'. i wait more than one hour and it does not proceed. can someone help please...?

[xingbowei]

if you use the coco datasets,you need to wait a more long time.

[shiyang07]

it may be a fool coding