search

matterport / Mask_RCNN

Mask R-CNN for object detection and instance segmentation on Keras and TensorFlow
Other
24.53k stars 11.68k forks source link

Mask R-CNN Training Issue (Memory Allocation) #2450

Open pavpanda opened 3 years ago

pavpanda commented 3 years ago

I am trying to train a simple Mask R-CNN on a 2 image training set (and 1 image validation set). However, I have over 300 objects per image, so in actuality, there's are approximately 1000 objects in the dataset that I am working with. When I begin training, I encounter the following issue:

Epoch 1/1 tcmalloc: large alloc... tcmalloc: large alloc... tcmalloc: large alloc... tcmalloc: large alloc... ...

I left for approx. 30 minutes and these messages of "tcmalloc: large alloc" continued. Any suggestions as to how to fix this? I'm running the Mask R-CNN on Google Colab and the RAM usage just shoots up to its maximum when training begins. My classification model currently consists of 3 classes (4 classees including the background). Thanks in advance!

Mahi-Mai commented 3 years ago

I wonder if we're looking for the same solution? https://github.com/matterport/Mask_RCNN/issues/2343