fchollet / deep-learning-with-python-notebooks

Jupyter notebooks for the code samples of the book "Deep Learning with Python"
MIT License
18.17k stars 8.53k forks source link

Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR #108

Closed pgoldtho closed 5 years ago

pgoldtho commented 5 years ago

Getting internal errors while running 5.1-introduction-to-convnets.ipynb (and some other examples). I found a workaround. Posting it here to help others who may encounter the same issue. I have the following setup:

Ubuntu 18.04 RTX 2070 NVIDIA 410.104 CUDA 10.0 cuDNN 7.5 tensorflow-gpu 1.13.1

The model.fit(train_images, train_labels, epochs=5, batch_size=64) command fail and displays this in the console:

2019-04-11 14:19:56.948126: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6955 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2070, pci bus id: 0000:01:00.0, compute capability: 7.5)
2019-04-11 14:19:57.904577: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally
2019-04-11 14:19:59.264535: E tensorflow/stream_executor/cuda/cuda_dnn.cc:334] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2019-04-11 14:19:59.276447: E tensorflow/stream_executor/cuda/cuda_dnn.cc:334] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

This appears to be an issue with RTX graphics cards. The workaround is to edit the workbook adding a cell with

import tensorflow as tf
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
session = tf.Session(config=config)