I got a problem with training. Help :)

[gtkim1828]

Hi. I did a test with demo.ipynb. And it worked. But I got a problem with coco.py. I want to train with coco dataset in 2014. I followed your procedure like :

Train a new model starting from pre-trained COCO weights

python3 samples/coco/coco.py train --dataset=/path/to/coco/ --model=coco

And I changed a variable, IMAGES_PER_GPU=1. But, It didn't work.... I don't know Why.... Help....

This is the console.

Using TensorFlow backend. Command: train Model: coco Dataset: ../../train2014 Year: 2014 Logs: D:\GT\workspace\Mask_RCNN-master\samples\coco\logs Auto Download: False configure printing start-----------------

Configurations: BACKBONE resnet101 BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] COMPUTE_BACKBONE_SHAPE None DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.7 DETECTION_NMS_THRESHOLD 0.3 FPN_CLASSIF_FC_LAYERS_SIZE 1024 GPU_COUNT 1 GRADIENT_CLIP_NORM 5.0 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 1024 IMAGE_META_SIZE 93 IMAGE_MIN_DIM 800 IMAGE_MIN_SCALE 0 IMAGE_RESIZE_MODE square IMAGE_SHAPE [1024 1024 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'rpn_class_loss': 1.0, 'mrcnn_class_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME coco NUM_CLASSES 81 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (32, 64, 128, 256, 512) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TOP_DOWN_PYRAMID_SIZE 256 TRAIN_BN False TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001

Loading weights D:\GT\workspace\Mask_RCNN-master\samples\coco\mask_rcnn_coco.h5 loading annotations into memory... Done (t=22.91s) creating index... index created! loading annotations into memory... Done (t=10.47s) creating index... index created! loading annotations into memory... Done (t=1.28s) creating index... index created! Training network heads test0 test1 test2 test3 test4

Starting at epoch 0. LR=0.001

Checkpoint Path: D:\GT\workspace\Mask_RCNN-master\samples\coco\logs\coco20181001T0901\mask_rcnncoco{epoch:04d}.h5 Selecting layers to train fpn_c5p5 (Conv2D) fpn_c4p4 (Conv2D) fpn_c3p3 (Conv2D) fpn_c2p2 (Conv2D) fpn_p5 (Conv2D) fpn_p2 (Conv2D) fpn_p3 (Conv2D) fpn_p4 (Conv2D) In model: rpn_model rpn_conv_shared (Conv2D) rpn_class_raw (Conv2D) rpn_bbox_pred (Conv2D) mrcnn_mask_conv1 (TimeDistributed) mrcnn_mask_bn1 (TimeDistributed) mrcnn_mask_conv2 (TimeDistributed) mrcnn_mask_bn2 (TimeDistributed) mrcnn_class_conv1 (TimeDistributed) mrcnn_class_bn1 (TimeDistributed) mrcnn_mask_conv3 (TimeDistributed) mrcnn_mask_bn3 (TimeDistributed) mrcnn_class_conv2 (TimeDistributed) mrcnn_class_bn2 (TimeDistributed) mrcnn_mask_conv4 (TimeDistributed) mrcnn_mask_bn4 (TimeDistributed) mrcnn_bbox_fc (TimeDistributed) mrcnn_mask_deconv (TimeDistributed) mrcnn_class_logits (TimeDistributed) mrcnn_mask (TimeDistributed) C:\Users\Ipcl\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\gradients_impl.py:108: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory. "Converting sparse IndexedSlices to a dense Tensor of unknown shape. " Epoch 1/40

Kernel died, restarting

---

PLEASE Help me...

[gtkim1828]

I use GTX1080Ti, 24GB RAM and i5-6600 CPU.

[kevinisbest]

I guess maybe your shortage of memory? Try to change IMAGE_MAX_DIM and IMAGE_MIN_DIM in your config.

[YalongLiu]

It is not about OOM(out of memory) Just Baidu/Google "Converting sparse IndexedSlices to a dense Tensor of unknown shape. " and you can get this: https://blog.csdn.net/LaineGates/article/details/78974660