search

matterport / Mask_RCNN

Mask R-CNN for object detection and instance segmentation on Keras and TensorFlow
Other
24.54k stars 11.68k forks source link

When I use two gpus to train Mask_Rcnn，it will appear error. #1318

Open BITQinYong opened 5 years ago

BITQinYong commented 5 years ago

When I use two gpus to train Mask_Rcnn，it will appear error.

InvalidArgumentError (see above for traceback): Cannot colocate nodes 'tower_0/mask_rcnn/Variable/anchors/Variable/read_tower_0/mask_rcnn/Variable_0' and 'anchors/Variable: Cannot merge devices with incompatible ids: '/device:GPU:1' and '/device:GPU:0' [[Node: tower_0/mask_rcnn/Variable/anchors/Variable/read_tower_0/mask_rcnn/Variable_0 = IdentityT=DT_FLOAT, _class=["loc:@anchors/Variable"], _device="/device:GPU:0"]]

xelmirage commented 5 years ago

are you using 

config = tf.ConfigProto()  
config.gpu_options.allow_growth = True

or

import os  
os.environ["CUDA_VISIBLE_DEVICES"] = "1"  
from keras.backend.tensorflow_backend import set_session  
config = tf.ConfigProto()  
config.gpu_options.per_process_gpu_memory_fraction = 0.3  
set_session(tf.Session(config=config))

to limit the memory?

I used them and encountered the same problem with you, and removing there limits solved the problem