search

arnab39 / FewShot_GAN-Unet3D

Tensorflow implementation of our paper: Few-shot 3D Multi-modal Medical Image Segmentation using Generative Adversarial Learning
MIT License
379 stars 83 forks source link

Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR #12

Open skyuuka opened 5 years ago

skyuuka commented 5 years ago

I got the following error when running python main_unset.py --training

2019-09-12 21:17:30.594167: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
2019-09-12 21:17:31.140393: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 0 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:1a:00.0
totalMemory: 10.73GiB freeMemory: 10.57GiB
2019-09-12 21:17:31.324298: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 1 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:1b:00.0
totalMemory: 10.73GiB freeMemory: 10.57GiB
2019-09-12 21:17:31.508353: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 2 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:3d:00.0
totalMemory: 10.73GiB freeMemory: 10.57GiB
2019-09-12 21:17:31.691047: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 3 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:3e:00.0
totalMemory: 10.73GiB freeMemory: 10.57GiB
2019-09-12 21:17:31.691543: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1484] Adding visible gpu devices: 0, 1, 2, 3
2019-09-12 21:17:33.385536: I tensorflow/core/common_runtime/gpu/gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-12 21:17:33.385595: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971]      0 1 2 3 
2019-09-12 21:17:33.385603: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 0:   N N N N 
2019-09-12 21:17:33.385608: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 1:   N N N N 
2019-09-12 21:17:33.385613: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 2:   N N N N 
2019-09-12 21:17:33.385618: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 3:   N N N N 
2019-09-12 21:17:33.386348: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10439 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:1a:00.0, compute capability: 7.5)
2019-09-12 21:17:33.589232: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 10439 MB memory) -> physical GPU (device: 1, name: GeForce RTX 2080 Ti, pci bus id: 0000:1b:00.0, compute capability: 7.5)
2019-09-12 21:17:33.799606: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 10439 MB memory) -> physical GPU (device: 2, name: GeForce RTX 2080 Ti, pci bus id: 0000:3d:00.0, compute capability: 7.5)
2019-09-12 21:17:33.995857: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 10439 MB memory) -> physical GPU (device: 3, name: GeForce RTX 2080 Ti, pci bus id: 0000:3e:00.0, compute capability: 7.5)
2019-09-12 21:18:10.689782: E tensorflow/stream_executor/cuda/cuda_dnn.cc:352] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
Fatal Python error: Segmentation fault

Thread 0x00007f1e85266740 (most recent call first):
  File "/home/skyuuka/anaconda2/envs/FewShot_GAN-Unet3D/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1350 in _call_tf_sessionrun
  File "/home/skyuuka/anaconda2/envs/FewShot_GAN-Unet3D/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1263 in _run_fn
  File "/home/skyuuka/anaconda2/envs/FewShot_GAN-Unet3D/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1278 in _do_call
  File "/home/skyuuka/anaconda2/envs/FewShot_GAN-Unet3D/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1272 in _do_run
  File "/home/skyuuka/anaconda2/envs/FewShot_GAN-Unet3D/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1100 in _run
  File "/home/skyuuka/anaconda2/envs/FewShot_GAN-Unet3D/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 877 in run
  File "/home/skyuuka/anaconda2/envs/FewShot_GAN-Unet3D/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 4986 in _run_using_default_session
  File "/home/skyuuka/anaconda2/envs/FewShot_GAN-Unet3D/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2241 in run
  File "/home/skyuuka/workspace/FewShot_GAN-Unet3D/unet3D/model_unet.py", line 213 in train
  File "main_unet.py", line 62 in main
  File "/home/skyuuka/anaconda2/envs/FewShot_GAN-Unet3D/lib/python3.5/site-packages/tensorflow/python/platform/app.py", line 125 in run
  File "main_unet.py", line 69 in <module>
little1TouwaErio commented 5 years ago

@ @skyuuka Hello, did you successfully run the original code？

skyuuka commented 5 years ago

@ @skyuuka Hello, did you successfully run the original code？

Yes. I tried newer version of TF > 1.7.0, and it works.

little1TouwaErio commented 5 years ago

我这边可以加你QQ 请教一下吗 非常感谢啊

发自我的iPhone

------------------ Original ------------------ From: Lin Chen <notifications@github.com> Date: Tue,Oct 15,2019 0:53 PM To: arnab39/FewShot_GAN-Unet3D <FewShot_GAN-Unet3D@noreply.github.com> Cc: Lililicg <416580793@qq.com>, Comment <comment@noreply.github.com> Subject: Re: [arnab39/FewShot_GAN-Unet3D] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR (#12)

little1TouwaErio commented 5 years ago

Hello, my registration on iseg  website has not passed. Could you please share the dataset with me？

------------------ 原始邮件 ------------------ 发件人: "Lin Chen"<notifications@github.com>; 发送时间: 2019年10月15日(星期二) 中午12:53 收件人: "arnab39/FewShot_GAN-Unet3D"<FewShot_GAN-Unet3D@noreply.github.com>; 抄送: "416580793"<416580793@qq.com>;"Comment"<comment@noreply.github.com>; 主题: Re: [arnab39/FewShot_GAN-Unet3D] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR (#12)

@ @skyuuka Hello, did you successfully run the original code？

Yes. I tried newer version of TF > 1.7.0, and it works.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

hkthirano commented 4 years ago

If the version of tensorflow is 1.7.0 and the version of cudnn is 7.0.5, the program run. However, If the version of cudnn is 7.3.0, the program does not run.