search

Arturus / kaggle-web-traffic

1st place solution
MIT License
1.82k stars 667 forks source link

Cannot colocate nodes 'global_norm/L2Loss' and 'gradients/CudnnRNN_grad/CudnnRNNBackprop' #21

Closed sunfu2016 closed 5 years ago

sunfu2016 commented 6 years ago

InvalidArgumentError (see above for traceback): Cannot colocate nodes 'global_norm/L2Loss' and 'gradients/CudnnRNN_grad/CudnnRNNBackprop' because no device type supports both of those nodes and the other nodes colocated with them.

runzeer commented 6 years ago

I came across this issue.So could you tell me why?

changyunke commented 5 years ago

so am i

changyunke commented 5 years ago

this question is important,but no one can solve it,...................

Arturus commented 5 years ago

Model is fixed to work with TF 1.10

kingaza commented 5 years ago

I met the same problem, and I am using TF 1.10.0 ... @Arturus @sunfu2016 @changyunke , could you give me some help? thanks a lot.

tensorboard 1.10.0 tensorflow-gpu 1.10.0

tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot colocate nodes 'm_0/global_norm/L2Loss' and 'm_0/gradients/m_0/cudnn_gru/CudnnRNN_grad/CudnnRNNBackprop' because no device type supports both of those nodes and the other nodes colocated with them. Colocation Debug Info: Colocation group had the following types and devices: CudnnRNNBackprop: GPU Identity: GPU CPU L2Loss: CPU

Colocation members and user-requested devices: m_0/gradients/m_0/cudnn_gru/CudnnRNN_grad/CudnnRNNBackprop (CudnnRNNBackprop) /device:GPU:0 m_0/gradients/m_0/cudnn_gru/CudnnRNN_grad/tuple/control_dependency (Identity) /device:GPU:0 m_0/gradients/m_0/cudnn_gru/CudnnRNN_grad/tuple/control_dependency_1 (Identity) /device:GPU:0 m_0/gradients/m_0/cudnn_gru/CudnnRNN_grad/tuple/control_dependency_2 (Identity) /device:GPU:0 m_0/gradients/m_0/cudnn_gru/CudnnRNN_grad/tuple/control_dependency_3 (Identity) /device:GPU:0 m_0/global_norm/L2Loss (L2Loss) /device:GPU:0

     [[Node: m_0/global_norm/L2Loss = L2Loss[T=DT_FLOAT, _class=["loc:@m_0/gradients/m_0/cudnn_gru/CudnnRNN_grad/CudnnRNNBackprop"], _device="/device:GPU:0"](m_0/gradients/m_0/cudnn_gru/CudnnRNN_grad/tuple/control_dependency_3)]]
Clytzes commented 5 years ago

add this，it will solve the issue and work with GPU

flags = tf.flags flags.DEFINE_boolean("use_cudnn", False, "Whether to use cudnn (only for GPU)")

reference: https://github.com/HKUST-KnowComp/R-Net/issues/7

win10

tensorflow-gpu 1.11.0

CUDA 9

cudnn v7