BUG: Lc0 cannot detect cudNN

[Caleb-Kang]

Bug Report

Lc0 cannot detect the CudNN libraries.

I downloaded both the Runtime and Developer Libraries for cudNN into /usr/local/cuda-11.0/lib64. Then, in meson_options.txt, I added the folder where cudNN was installed into the "cudNN_libdirs" option.

This is the code:

option('cudnn_libdirs',
       type: 'array',
       value: ['/opt/cuda/lib64/', '/usr/local/cuda/lib64/', '/usr/lib/cuda/lib64/', '/usr/local/cuda-11.0/lib64/'],
       description: 'Paths to Cuda/cudnn libraries')

I also included the folder that holds the cudNN libraries in the "cudNN_include" option.

option('cudnn_include', 
       type: 'array',
       value: ['/opt/cuda/include/', '/usr/local/cuda/include/', '/usr/lib/cuda/include/', '/usr/local/cuda-11.0/lib64/'],
       description: 'Paths to cudnn include directory')

Lc0 version

Lc0 v0.26.1 / Ubuntu 18.04.4 LTS / CUDA + cudNN Backend

Hardware

• One NVIDIA® Tesla P100 GPU.
• 14.7 GB RAM
• Intel® Xeon(R) CPU @ 2.20GHz × 4

[Caleb-Kang]

Nevermind, lc0 found CudNN. But now there is another problem: When I run ./lc0 benchmark, this is what happens:

./lc0 benchmark
       _
|   _ | |
|_ |_ |_| v0.26.1 built Jul 18 2020
Found pb network file: ./256x20-t40-1541.pb.gz
Creating backend [cudnn-auto]...
Switching to [cudnn-fp16]...
CUDA Runtime version: 11.0.0
Cudnn version: 8.0.1
Latest version of CUDA supported by the driver: 11.0.0
GPU: Tesla P100-PCIE-16GB
GPU memory: 15.8993 Gb
GPU clock frequency: 1328.5 MHz
GPU compute capability: 6.0
CUDA error: no kernel image is available for execution on the device (../../src/neural/cuda/winograd_helper.inc:363)

[Caleb-Kang]

I am using Google Cloud Compute Engine and am running a Virtual Machine Box that does not use the P100 GPU, but generates its own display (llvmpipe (LLVM 10.0.0, 256 bits)). Is that why no kernel image is detected?

[Caleb-Kang]

Hmm, I tried --backend-opts=custom_winograd=false and lc0 works now. I will close this issue without any help. Lol

[borg323]

Actually you can do better than that. Change the line https://github.com/LeelaChessZero/lc0/blob/master/meson.build#L447 to files += cuda_gen.process(cuda_files_nvcc_fp16, extra_args: ['-arch=compute_60', '-code=sm_60']). You can do that with a simple sed -i "/compute_/ s/70/60/g" meson.build

[Caleb-Kang]

Oh, thank you @borg323. I will try that.