E tensorflow/stream_executor/cuda/cuda_dnn.cc:396] Loaded runtime CuDNN library: 7103 (compatibility version 7100) but source was compiled with 7005 (compatibility version 7000). If using a binary install, upgrade your CuDNN library to match. If building from sources, make sure the library loaded at runtime matches a compatible version specified during compile configuration. 2018-12-20 17:49:07.556965: F tensorflow/core/kernels/conv_ops.cc:712] Check failed: stream->parent()->GetConvolveAlgorithms( conv_parameters.ShouldIncludeWinogradNonfusedAlgo<T>(), &algorithms)

[TulipDi]

System information

• What is the top-level directory of the model you are using: models/research
• Have I written custom code (as opposed to using a stock example script provided in TensorFlow):no
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04):Ubuntu 14.04
• TensorFlow installed from (source or binary): conda install tensorflow-gpu
• TensorFlow version (use command below):1.7
• Bazel version (if compiling from source):
• CUDA/cuDNN version:8.0/7.0.5
• GPU model and memory:TITANX
• Exact command to reproduce:python deeplab/train.py --logtostderr --training_number_of_steps=300 --train_split="train" --model_variant="xception_65" --atrous_rates=6 --atrous_rates=12 --atrous_rates=18 --output_stride=16 --decoder_output_stride=4 --train_crop_size=513 --train_crop_size=513 --train_batch_size=2 --dataset="camvid" --tf_initial_checkpoint='./deeplab/backbone/deeplabv3_cityscapes_train/model.ckpt' --train_logdir='./deeplab/exp/camvid_train/train' --dataset_dir='./deeplab/datasets/CamVid/tfrecord'

Describe the problem

i try to update my cudnn from 7005 to 7103, but i got the same error.

Source code / logs

WARNING:tensorflow:From /home/hhd/anaconda3/envs/tf1.7/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/base.py:198: retry (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version. Instructions for updating: Use the retry module or similar alternatives. /home/hhd/anaconda3/envs/tf1.7/lib/python3.6/site-packages/absl/flags/_validators.py:358: UserWarning: Flag --train_logdir has a non-None default value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line! 'command line!' % flag_name) /home/hhd/anaconda3/envs/tf1.7/lib/python3.6/site-packages/absl/flags/_validators.py:358: UserWarning: Flag --tf_initial_checkpoint has a non-None default value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line! 'command line!' % flag_name) /home/hhd/anaconda3/envs/tf1.7/lib/python3.6/site-packages/absl/flags/_validators.py:358: UserWarning: Flag --dataset_dir has a non-None default value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line! 'command line!' % flag_name) INFO:tensorflow:Training on train set INFO:tensorflow:Ignoring initialization; other checkpoint exists WARNING:tensorflow:From /home/hhd/anaconda3/envs/tf1.7/lib/python3.6/site-packages/tensorflow/contrib/slim/python/slim/learning.py:736: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession 2018-12-20 17:47:36.440252: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX 2018-12-20 17:47:36.761742: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1344] Found device 0 with properties: name: GeForce GTX TITAN X major: 5 minor: 2 memoryClockRate(GHz): 1.076 pciBusID: 0000:05:00.0 totalMemory: 11.92GiB freeMemory: 1.92GiB 2018-12-20 17:47:36.761854: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0 2018-12-20 17:47:37.516797: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix: 2018-12-20 17:47:37.516883: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0 2018-12-20 17:47:37.516905: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N 2018-12-20 17:47:37.517219: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1695 MB memory) -> physical GPU (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:05:00.0, compute capability: 5.2) INFO:tensorflow:Restoring parameters from ./deeplab/exp/camvid_train/train/model.ckpt-0 INFO:tensorflow:Running local_init_op. INFO:tensorflow:Done running local_init_op. INFO:tensorflow:Starting Session. INFO:tensorflow:Saving checkpoint to path ./deeplab/exp/camvid_train/train/model.ckpt INFO:tensorflow:Starting Queues. INFO:tensorflow:global_step/sec: 0 2018-12-20 17:49:07.554923: E tensorflow/stream_executor/cuda/cuda_dnn.cc:396] Loaded runtime CuDNN library: 7103 (compatibility version 7100) but source was compiled with 7005 (compatibility version 7000). If using a binary install, upgrade your CuDNN library to match. If building from sources, make sure the library loaded at runtime matches a compatible version specified during compile configuration. 2018-12-20 17:49:07.556965: F tensorflow/core/kernels/conv_ops.cc:712] Check failed: stream->parent()->GetConvolveAlgorithms( conv_parameters.ShouldIncludeWinogradNonfusedAlgo(), &algorithms)

[rootkitchao]

I don't think the current Tensorflow version supports CUDA 8.0.Maybe you need to upgrade your CUDA.

[TulipDi]

@rootkitchao Thank you for the answers. I resolved it by upgrade CUDA 8.0 to CUDA 9.0.