Device Error while training custom dataset

fizyr / keras-maskrcnn

Keras implementation of MaskRCNN object detection.

Apache License 2.0

406 stars 131 forks source link

Device Error while training custom dataset #85

Closed Caliber2 closed 5 years ago

Caliber2 commented 5 years ago

InvalidArgumentError (see above for traceback): Cannot assign a device for operation 'training/Adam/gradients/filtered_detections/map/while/embedding_lookup_3_grad/Reshape_1/f_acc': Could not satisfy explicit device specification '' because the node was colocated with a group of nodes that required incompatible device '/job:localhost/replica:0/task:0/device:GPU:0'

hgaiser commented 5 years ago

I think I've seen this error before, I seem to remember updating tensorflow fixed this issue. Could you give that a shot? What version are you using at the moment?

Caliber2 commented 5 years ago

@hgaiser Thanks for the reply. Anyways, just after posting the issue, I realized the combination CUDA 9, CUDNN 7, tensorflow 1.4 and tensorflow-gpu 1.8 solves it.