When I training the VoxResNet.
The error shows:
Namespace(display_step=1000, gpu=0, input_file='dataset_train.json', input_shape=[80, 80, 80], iteration=10000, learning_rate=0.001, n_batch=1, n_tiles=[5, 5, 5], out='vrn.npz', output_shape=[60, 60, 60], validation_file=None, weight_decay=0.0005)
Traceback (most recent call last):
File "train.py", line 113, in
logits = vrn(x_train, train=True)
File "/media/jingjingpc/0D291C800D291C80/tensorflow_practice/brain_segmentation-master/model.py", line 79, in call
h = self.conv1a(x)
File "/usr/local/lib/python2.7/dist-packages/chainer/links/connection/convolution_nd.py", line 88, in call
x, self.W, self.b, self.stride, self.pad, cover_all=self.cover_all)
File "/usr/local/lib/python2.7/dist-packages/chainer/functions/connection/convolution_nd.py", line 423, in convolution_nd
return func(x, W, b)
File "/usr/local/lib/python2.7/dist-packages/chainer/function.py", line 235, in call
ret = node.apply(inputs)
File "/usr/local/lib/python2.7/dist-packages/chainer/function_node.py", line 245, in apply
outputs = self.forward(in_data)
File "/usr/local/lib/python2.7/dist-packages/chainer/function.py", line 135, in forward
return self._function.forward(inputs)
File "/usr/local/lib/python2.7/dist-packages/chainer/functions/connection/convolution_nd.py", line 174, in forward
return self._forward_cudnn(x, W, b)
File "/usr/local/lib/python2.7/dist-packages/chainer/functions/connection/convolution_nd.py", line 111, in _forward_cudnn
handle = cudnn.get_handle()
File "cupy/cudnn.pyx", line 24, in cupy.cudnn.get_handle
File "cupy/cudnn.pyx", line 32, in cupy.cudnn.get_handle
File "cupy/cuda/cudnn.pyx", line 463, in cupy.cuda.cudnn.create
File "cupy/cuda/cudnn.pyx", line 444, in cupy.cuda.cudnn.check_status
cupy.cuda.cudnn.CuDNNError: CUDNN_STATUS_INTERNAL_ERROR: CUDNN_STATUS_INTERNAL_ERROR
When I training the VoxResNet. The error shows: Namespace(display_step=1000, gpu=0, input_file='dataset_train.json', input_shape=[80, 80, 80], iteration=10000, learning_rate=0.001, n_batch=1, n_tiles=[5, 5, 5], out='vrn.npz', output_shape=[60, 60, 60], validation_file=None, weight_decay=0.0005) Traceback (most recent call last): File "train.py", line 113, in logits = vrn(x_train, train=True) File "/media/jingjingpc/0D291C800D291C80/tensorflow_practice/brain_segmentation-master/model.py", line 79, in call h = self.conv1a(x) File "/usr/local/lib/python2.7/dist-packages/chainer/links/connection/convolution_nd.py", line 88, in call x, self.W, self.b, self.stride, self.pad, cover_all=self.cover_all) File "/usr/local/lib/python2.7/dist-packages/chainer/functions/connection/convolution_nd.py", line 423, in convolution_nd return func(x, W, b) File "/usr/local/lib/python2.7/dist-packages/chainer/function.py", line 235, in call ret = node.apply(inputs) File "/usr/local/lib/python2.7/dist-packages/chainer/function_node.py", line 245, in apply outputs = self.forward(in_data) File "/usr/local/lib/python2.7/dist-packages/chainer/function.py", line 135, in forward return self._function.forward(inputs) File "/usr/local/lib/python2.7/dist-packages/chainer/functions/connection/convolution_nd.py", line 174, in forward return self._forward_cudnn(x, W, b) File "/usr/local/lib/python2.7/dist-packages/chainer/functions/connection/convolution_nd.py", line 111, in _forward_cudnn handle = cudnn.get_handle() File "cupy/cudnn.pyx", line 24, in cupy.cudnn.get_handle File "cupy/cudnn.pyx", line 32, in cupy.cudnn.get_handle File "cupy/cuda/cudnn.pyx", line 463, in cupy.cuda.cudnn.create File "cupy/cuda/cudnn.pyx", line 444, in cupy.cuda.cudnn.check_status cupy.cuda.cudnn.CuDNNError: CUDNN_STATUS_INTERNAL_ERROR: CUDNN_STATUS_INTERNAL_ERROR