Can not use cuDNN on context None: cannot compile with cuDNN

[yyf]

Trying to run ./run_demo.sh using GPU, but got a cuDNN related error. Checked related issues but still no clue what combination of the packages work. Below are my setup info and error log. Wondering if there is any more info on troubleshooting GPU setup for Merlin? Thanks.

Setup info

• Ubuntu: 16.04
• Conda: 4.6.7
• Python version : 2.7.15
• Theano: 1.0.4
• Nvidia Driver: 384.130
• Cuda: 7.5
• CUDNN_MAJOR: 7

Error log Running on GPU id=0 ... Can not use cuDNN on context None: cannot compile with cuDNN. We got this error: /tmp/try_flags__R11_b.c:4:10: fatal error: cudnn.h: No such file or directory

include

      ^~~~~~~~~

compilation terminated.

ERROR (theano.gpuarray): Could not initialize pygpu, support disabled Traceback (most recent call last): File "/root/anaconda2/lib/python2.7/site-packages/theano/gpuarray/init.py", line 227, in use(config.device) File "/root/anaconda2/lib/python2.7/site-packages/theano/gpuarray/init.py", line 214, in use init_dev(device, preallocate=preallocate) File "/root/anaconda2/lib/python2.7/site-packages/theano/gpuarray/init.py", line 159, in init_dev pygpu.blas.gemm(0, tmp, tmp, 0, tmp, overwrite_c=True) File "pygpu/blas.pyx", line 149, in pygpu.blas.gemm File "pygpu/blas.pyx", line 47, in pygpu.blas.pygpu_blas_rgemm GpuArrayException: ('cublasCreate: (cublas) Library not initialized. (Possibly because the driver version is too old for the cuda version)', 11) /home/.../.../merlin/src/logplot/logging_plotting.py:55: UserWarning: This call to matplotlib.use() has no effect because the backend has already been chosen; matplotlib.use() must be called before pylab, matplotlib.pyplot, or matplotlib.backends is imported for the first time.

[ZhaoZeqing]

My environment is:

• Theano: 0.9.0
• python: 2.7
• cuda: 8.0
• cudnn:5.0.5

I had also encountered this problem, the following may help you:

1. change line 20 in egs/xxx/script/submit.sh to "THEANO_FLAGS="mode=FAST_RUN,device=cuda$gpu_id,"$MERLIN_THEANO_FLAGS"
2. add "export LIBRARY_PATH=${CUDA}/lib64" to .bashrc

[yyf]

Thanks. I tried both but still got the same error. Any other thought?

[ZhaoZeqing]

emm... try this? rm -rf ~/.theano

[yyf]

Tried removing /.theano folder but the issue persists. Here is the full error log, do the PATHS look right? :

`(base) ubuntu@ip-address:~/.../merlin/egs/build_your_own_voice/LJ_16k$ ./05_train_duration_model.sh conf/global_settings.cfg Step 5: training duration model... Architecture: x86_64 Distribution: Ubuntu 16.04.6 LTS HOSTNAME=ip-address USER=ubuntu

PATH: /home/ubuntu/bin /home/ubuntu/.local/bin /home/ubuntu/anaconda2/bin /home/ubuntu/anaconda2/condabin /usr/local/cuda/bin /usr/local/bin /opt/aws/bin /home/ubuntu/src/cntk/bin /usr/local/mpi/bin /usr/local/sbin /usr/local/bin /usr/sbin /usr/bin /sbin /bin /usr/games /usr/local/games /snap/bin LD_LIBRARY_PATH: /home/ubuntu/src/cntk/bindings/python/cntk/libs /usr/local/cuda/lib64 /usr/local/lib /usr/lib /usr/local/cuda/extras/CUPTI/lib64 /usr/local/mpi/lib PYTHONPATH: /home/ubuntu/src/cntk/bindings/python PYTHONBIN: python MERLIN_THEANO_FLAGS: cuda.root=/usr/local/8.0 floatX=float32 on_unused_input=ignore

No GPU is available! Running on CPU... /home/ubuntu/.../merlin/src/logplot/logging_plotting.py:55: UserWarning: This call to matplotlib.use() has no effect because the backend has already been chosen; matplotlib.use() must be called before pylab, matplotlib.pyplot, or matplotlib.backends is imported for the first time.

The backend was originally set to 'agg' by the following ...: File "/home/ubuntu/.../merlin/src/run_merlin.py", line 74, in from models.deep_rnn import DeepRecurrentNetwork File "/home/ubuntu/.../merlin/src/models/deep_rnn.py", line 16, in from training_schemes.rprop import compile_RPROP_train_function File "/home/ubuntu/.../merlin/src/training_schemes/rprop.py", line 47, in import matplotlib.pyplot as plt File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/matplotlib/pyplot.py", line 71, in from matplotlib.backends import pylab_setup File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/matplotlib/backends/init.py", line 16, in line for line in traceback.format_stack()

matplotlib.use('PDF') 2019-03-04 19:33:58,904 CRITICAL configuration: error reading user configuration file /home/ubuntu/.../merlin/egs/build_your_own_voice/LJ_16k/conf/global_settings.cfg Traceback (most recent call last): File "/home/ubuntu/.../merlin/src/run_merlin.py", line 1244, in cfg.configure(config_file) File "/home/ubuntu/.../merlin/src/configuration/configuration.py", line 86, in configure self.user_configuration(configFile) File "/home/ubuntu/.../merlin/src/configuration/configuration.py", line 124, in user_configuration cfgparser.readfp(open(configFile)) File "/home/ubuntu/anaconda2/lib/python2.7/ConfigParser.py", line 324, in readfp self._read(fp, filename) File "/home/ubuntu/anaconda2/lib/python2.7/ConfigParser.py", line 512, in _read raise MissingSectionHeaderError(fpname, lineno, line) ConfigParser.MissingSectionHeaderError: File contains no section headers. file: /home/ubuntu/.../merlin/egs/build_your_own_voice/LJ_16k/conf/global_settings.cfg, line: 5 'MerlinDir=/home/ubuntu/.../merlin\n'`

[ZhaoZeqing]

try modify src/setup_env.sh line 13 MERLIN_THEANO_FLAGS="mode=FAST_RUN,device=cuda0,cuda.root=/usr/local/8.0,floatX=float32,on_unused_input=ignore"

[yyf]

This is what I have currently in setup_env.sh: MERLIN_THEANO_FLAGS="cuda.root=/usr/local/8.0,floatX=float32,on_unused_input=ignore"

Adding "mode=FAST_RUN,device=cuda0" to my setup_env.sh doesn't resolve the issue still?

[ZhaoZeqing]

change cuda.root to your cuda path, mine is: MERLIN_THEANO_FLAGS="cuda.root=/usr/local/cuda-8.0,floatX=float32,on_unused_input=ignore"

[yyf]

Started on a clean ubuntu machine and changed this line works for me now: "THEANO_FLAGS="mode=FAST_RUN,device=cuda$gpu_id,"$MERLIN_THEANO_FLAGS"

Thanks @ZhaoZeqing