search

matterport / Mask_RCNN

Mask R-CNN for object detection and instance segmentation on Keras and TensorFlow
Other
24.64k stars 11.7k forks source link

tensorflow error during training my own dataset #1509

Open dantingz opened 5 years ago

dantingz commented 5 years ago

My data set is coco-like and should have fit well, but then it generates error with tensor flow. I didn't find much info online. Does anyone have any ideas?

Starting at epoch 0. LR=0.001

Checkpoint Path: /content/drive/My Drive/newlar/logs/larvaes20190517T0434/mask_rcnnlarvaes{epoch:04d}.h5 Selecting layers to train fpn_c5p5 (Conv2D) fpn_c4p4 (Conv2D) fpn_c3p3 (Conv2D) fpn_c2p2 (Conv2D) fpn_p5 (Conv2D) fpn_p2 (Conv2D) fpn_p3 (Conv2D) fpn_p4 (Conv2D) In model: rpn_model rpn_conv_shared (Conv2D) rpn_class_raw (Conv2D) rpn_bbox_pred (Conv2D) mrcnn_mask_conv1 (TimeDistributed) mrcnn_mask_bn1 (TimeDistributed) mrcnn_mask_conv2 (TimeDistributed) mrcnn_mask_bn2 (TimeDistributed) mrcnn_class_conv1 (TimeDistributed) mrcnn_class_bn1 (TimeDistributed) mrcnn_mask_conv3 (TimeDistributed) mrcnn_mask_bn3 (TimeDistributed) mrcnn_class_conv2 (TimeDistributed) mrcnn_class_bn2 (TimeDistributed) mrcnn_mask_conv4 (TimeDistributed) mrcnn_mask_bn4 (TimeDistributed) mrcnn_bbox_fc (TimeDistributed) mrcnn_mask_deconv (TimeDistributed) mrcnn_class_logits (TimeDistributed) mrcnn_mask (TimeDistributed) /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gradients_impl.py:110: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory. "Converting sparse IndexedSlices to a dense Tensor of unknown shape. " /usr/local/lib/python3.6/dist-packages/keras/engine/training_generator.py:47: UserWarning: Using a generator with use_multiprocessing=True and multiple workers may duplicate your data. Please consider using thekeras.utils.Sequence class. UserWarning('Using a generator withuse_multiprocessing=True`'

KeyError Traceback (most recent call last)

in () 4 learning_rate=config.LEARNING_RATE, 5 epochs=4, ----> 6 layers='heads') 7 8 end_train = time.time() 10 frames /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py in _as_graph_def(self, from_version, add_shapes) 3116 if add_shapes: 3117 for node in graph.node: -> 3118 op = self._nodes_by_name[node.name] 3119 if op.outputs: 3120 node.attr["_output_shapes"].list.shape.extend( KeyError: 'training_9/SGD/gradients/zeros_like_16'
jackbartley8 commented 5 years ago

try this, it worked for me for a similar issue: import keras.backend as K K.clear_session()

bettasimousss commented 4 years ago

@jackbartley8 Thank you ! I've been struggling with this error for a while. It worked for me.