Just documenting this in case anyone else encounters this issue.
Built the provided dockerfile and ran python3 inference_15parts.py --scale=1 --scale=0.5 --scale=0.75
Saw E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR (among other errors) when calling model.predict.
Resolved the issue by adding the following to inference_15parts.py
import tensorflow as tf
from keras.backend.tensorflow_backend import set_session
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
sess = tf.Session(config=config)
set_session(sess)
Just documenting this in case anyone else encounters this issue. Built the provided dockerfile and ran
python3 inference_15parts.py --scale=1 --scale=0.5 --scale=0.75
Saw
E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
(among other errors) when callingmodel.predict
.Resolved the issue by adding the following to
inference_15parts.py