2020-05-10 18:38:43.349890: E tensorflow/stream_executor/cuda/cuda_dnn.cc:319] Loaded runtime CuDNN library: 7.4.1 but source was compiled with: 7.6.0. CuDNN library major and minor version needs to match or have higher minor version in case of CuDNN 7.0 or later version. If using a binary install, upgrade your CuDNN library. If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
2020-05-10 18:38:43.352466: E tensorflow/stream_executor/cuda/cuda_dnn.cc:319] Loaded runtime CuDNN library: 7.4.1 but source was compiled with: 7.6.0. CuDNN library major and minor version needs to match or have higher minor version in case of CuDNN 7.0 or later version. If using a binary install, upgrade your CuDNN library. If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
Traceback (most recent call last):
File "/home/developer/anaconda3/envs/tf115/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
return fn(*args)
File "/home/developer/anaconda3/envs/tf115/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn
target_list, run_metadata)
File "/home/developer/anaconda3/envs/tf115/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
(0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[{{node generator_A2B/h1_conv/Conv2D}}]]
[[truediv/_1819]]
(1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[{{node generator_A2B/h1_conv/Conv2D}}]]
0 successful operations.
0 derived errors ignored.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "train_relgan_vm.py", line 172, in <module>
main()
File "train_relgan_vm.py", line 106, in main
discriminator_learning_rate=discriminator_learning_rate)
File "/mnt/datadisk/wdf/Voice-morphing-RelGAN/models/relgan.py", line 300, in train
self.rnd: rand, self.alpha: alpha, self.generator_learning_rate: generator_learning_rate})
File "/home/developer/anaconda3/envs/tf115/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 956, in run
run_metadata_ptr)
File "/home/developer/anaconda3/envs/tf115/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1180, in _run
feed_dict_tensor, options, run_metadata)
File "/home/developer/anaconda3/envs/tf115/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
run_metadata)
File "/home/developer/anaconda3/envs/tf115/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
(0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[node generator_A2B/h1_conv/Conv2D (defined at /home/developer/anaconda3/envs/tf115/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]
[[truediv/_1819]]
(1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[node generator_A2B/h1_conv/Conv2D (defined at /home/developer/anaconda3/envs/tf115/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]
0 successful operations.
0 derived errors ignored.
環境 UBUNTU 16.04.6 LTS python 3.7 CUDA 10.0 cuDNN 7.4.1 NVIDIA Driver 418.87.01 tensorflow 1.15
上記の環境で、
python train_relgan_vm.pyを実行すると、以下のようなエラーがでます。 同様のエラーが発生したことがあったり、解決方法に心当たりがありましたら、ご教授頂きたいです。もしかすると、環境が異なることが原因（特にcuDNN）かもしれないので、itsuki8914さんの環境を教えていただけると助かります。