Errors in vl_testnn

[JoeCool90]

Windows 10, CUDA v10.1.168, VS 2017, MATLAB 2018a, matconvnet-1.0-beta25

Managed to get it compiled.

vl_testnn without cudnn passes all. vl_testnn with cudnn fails with 61 errors.

Every function that has an error, only fails sometimes. An example of a fail from each function that generates an error:

Error occurred in nnbnorm[dataType=single,device=gpu]/regression(rows=value1,cols=value1,numDims=value1,batchSize=value1) and it did not run to completion.

---------
Error ID:
---------
''

--------------
Error Details:
--------------
Error using <a href="matlab:matlab.internal.language.introspective.errorDocCallback('vl_nnbnorm')" style="font-weight:bold">vl_nnbnorm</a>
vl::impl::dispatch_cudnn<C, CU>::operator(): BatchNormForwardCudnn<dataType>::operator(): cuDNN error [cudnn: "e:\\projects\\dataprocessing\\matconvnet-1.0-beta25\\matlab\\src\\bits\\nnbnorm_cudnn.cu":241 (CUDNN_STATUS_EXECUTION_FAILED)]

Error in <a href="matlab:matlab.internal.language.introspective.errorDocCallback('nnbnorm/regression', 'E:\projects\DataProcessing\matconvnet-1.0-beta25\matlab\xtest\suite\nnbnorm.m', 19)" style="font-weight:bold">nnbnorm/regression</a> (<a href="matlab: opentoline('E:\projects\DataProcessing\matconvnet-1.0-beta25\matlab\xtest\suite\nnbnorm.m',19,0)">line 19</a>)
      [y,m] = vl_nnbnorm(x,g,b,'epsilon',epsilon) ;

Error occurred in nnconv[dataType=single,device=gpu]/filter_shapes(bias=value1,fw=value2,fh=value2) and it did not run to completion.

---------
Error ID:
---------
'VLMX:InvalidArgument'

--------------
Error Details:
--------------
Error using <a href="matlab:matlab.internal.language.introspective.errorDocCallback('vl_nnconv')" style="font-weight:bold">vl_nnconv</a>
vl::impl::dispatch_cudnn<C, CU>::operator(): ConvolutionForwardCudnn<dataType>::operator(): cuDNN error [cudnn: "e:\\projects\\dataprocessing\\matconvnet-1.0-beta25\\matlab\\src\\bits\\nnconv_cudnn.cu":209 (CUDNN_STATUS_BAD_PARAM)]

Error in <a href="matlab:matlab.internal.language.introspective.errorDocCallback('nnconv/filter_shapes', 'E:\projects\DataProcessing\matconvnet-1.0-beta25\matlab\xtest\suite\nnconv.m', 49)" style="font-weight:bold">nnconv/filter_shapes</a> (<a href="matlab: opentoline('E:\projects\DataProcessing\matconvnet-1.0-beta25\matlab\xtest\suite\nnconv.m',49,0)">line 49</a>)
      y = vl_nnconv(x,w,b) ;

Error occurred in nnconvt[dataType=single,device=gpu]/basic(depth=value1,numImages=value1,numFilters=value1) and it did not run to completion.

---------
Error ID:
---------
''

--------------
Error Details:
--------------
Error using <a href="matlab:matlab.internal.language.introspective.errorDocCallback('vl_nnconvt')" style="font-weight:bold">vl_nnconvt</a>
vl::impl::dispatch<C>::operator(): vl::impl::dispatch_cudnn<C, CU>::operator(): ConvolutionBackwardCudnn<dataType>::operator(): cuDNN error [cudnn: "e:\\projects\\dataprocessing\\matconvnet-1.0-beta25\\matlab\\src\\bits\\nnconv_cudnn.cu":517
(CUDNN_STATUS_EXECUTION_FAILED)]

Error in <a href="matlab:matlab.internal.language.introspective.errorDocCallback('nnconvt/basic', 'E:\projects\DataProcessing\matconvnet-1.0-beta25\matlab\xtest\suite\nnconvt.m', 26)" style="font-weight:bold">nnconvt/basic</a> (<a href="matlab: opentoline('E:\projects\DataProcessing\matconvnet-1.0-beta25\matlab\xtest\suite\nnconvt.m',26,0)">line 26</a>)
      y = vl_nnconvt(x,f,b) ;

Error occurred in nndagnn[dataType=single,device=gpu]/simpleRun and it did not run to completion.

---------
Error ID:
---------
''

--------------
Error Details:
--------------
Error using <a href="matlab:matlab.internal.language.introspective.errorDocCallback('vl_nnpool')" style="font-weight:bold">vl_nnpool</a>
vl::impl::dispatch_cudnn<C, CU>::operator(): nnpooling_cudnn::forward: cuDNN error [cudnn: 112:"e:\\projects\\dataprocessing\\matconvnet-1.0-beta25\\matlab\\src\\bits\\nnpooling_cudnn.cu" (CUDNN_STATUS_EXECUTION_FAILED)]

Error in <a href="matlab:matlab.internal.language.introspective.errorDocCallback('dagnn.Pooling/forward', 'E:\projects\DataProcessing\matconvnet-1.0-beta25\matlab\+dagnn\Pooling.m', 10)" style="font-weight:bold">dagnn.Pooling/forward</a> (<a href="matlab: opentoline('E:\projects\DataProcessing\matconvnet-1.0-beta25\matlab\+dagnn\Pooling.m',10,0)">line 10</a>)
      outputs{1} = vl_nnpool(inputs{1}, self.poolSize, ...

Error in <a href="matlab:matlab.internal.language.introspective.errorDocCallback('dagnn.Layer/forwardAdvanced', 'E:\projects\DataProcessing\matconvnet-1.0-beta25\matlab\+dagnn\Layer.m', 85)" style="font-weight:bold">dagnn.Layer/forwardAdvanced</a> (<a href="matlab: opentoline('E:\projects\DataProcessing\matconvnet-1.0-beta25\matlab\+dagnn\Layer.m',85,0)">line 85</a>)
      outputs = obj.forward(inputs, {net.params(par).value}) ;

Error in <a href="matlab:matlab.internal.language.introspective.errorDocCallback('dagnn.DagNN/eval', 'E:\projects\DataProcessing\matconvnet-1.0-beta25\matlab\+dagnn\@DagNN\eval.m', 91)" style="font-weight:bold">dagnn.DagNN/eval</a> (<a href="matlab: opentoline('E:\projects\DataProcessing\matconvnet-1.0-beta25\matlab\+dagnn\@DagNN\eval.m',91,0)">line 91</a>)
  obj.layers(l).block.forwardAdvanced(obj.layers(l)) ;

Error in <a href="matlab:matlab.internal.language.introspective.errorDocCallback('nndagnn/forward', 'E:\projects\DataProcessing\matconvnet-1.0-beta25\matlab\xtest\suite\nndagnn.m', 125)" style="font-weight:bold">nndagnn/forward</a> (<a href="matlab: opentoline('E:\projects\DataProcessing\matconvnet-1.0-beta25\matlab\xtest\suite\nndagnn.m',125,0)">line 125</a>)
      test.net.eval({'x0', test.x, 'label', test.class});

Error in <a href="matlab:matlab.internal.language.introspective.errorDocCallback('nndagnn/simpleRun', 'E:\projects\DataProcessing\matconvnet-1.0-beta25\matlab\xtest\suite\nndagnn.m', 58)" style="font-weight:bold">nndagnn/simpleRun</a> (<a href="matlab: opentoline('E:\projects\DataProcessing\matconvnet-1.0-beta25\matlab\xtest\suite\nndagnn.m',58,0)">line 58</a>)
      test.forward();

Error occurred in nnmnist[dataType=single,device=gpu]/valErrorRate(networkType=simplenn) and it did not run to completion.

---------
Error ID:
---------
'VLMX:InvalidArgument'

--------------
Error Details:
--------------
Error using <a href="matlab:matlab.internal.language.introspective.errorDocCallback('vl_nnconv')" style="font-weight:bold">vl_nnconv</a>
vl::impl::dispatch_cudnn<C, CU>::operator(): ConvolutionForwardCudnn<dataType>::operator(): cuDNN error [cudnn: "e:\\projects\\dataprocessing\\matconvnet-1.0-beta25\\matlab\\src\\bits\\nnconv_cudnn.cu":209 (CUDNN_STATUS_EXECUTION_FAILED)]

Error in <a href="matlab:matlab.internal.language.introspective.errorDocCallback('vl_simplenn', 'E:\projects\DataProcessing\matconvnet-1.0-beta25\matlab\simplenn\vl_simplenn.m', 300)" style="font-weight:bold">vl_simplenn</a> (<a href="matlab: opentoline('E:\projects\DataProcessing\matconvnet-1.0-beta25\matlab\simplenn\vl_simplenn.m',300,0)">line 300</a>)
      res(i+1).x = vl_nnconv(res(i).x, l.weights{1}, l.weights{2}, ...

Error in <a href="matlab:matlab.internal.language.introspective.errorDocCallback('cnn_train>processEpoch', 'E:\projects\DataProcessing\matconvnet-1.0-beta25\examples\cnn_train.m', 350)" style="font-weight:bold">cnn_train>processEpoch</a> (<a href="matlab: opentoline('E:\projects\DataProcessing\matconvnet-1.0-beta25\examples\cnn_train.m',350,0)">line 350</a>)
    res = vl_simplenn(net, im, dzdy, res, ...

Error in <a href="matlab:matlab.internal.language.introspective.errorDocCallback('cnn_train', 'E:\projects\DataProcessing\matconvnet-1.0-beta25\examples\cnn_train.m', 150)" style="font-weight:bold">cnn_train</a> (<a href="matlab: opentoline('E:\projects\DataProcessing\matconvnet-1.0-beta25\examples\cnn_train.m',150,0)">line 150</a>)
    [net, state] = processEpoch(net, state, params, 'train') ;

Error in <a href="matlab:matlab.internal.language.introspective.errorDocCallback('cnn_mnist', 'E:\projects\DataProcessing\matconvnet-1.0-beta25\examples\mnist\cnn_mnist.m', 54)" style="font-weight:bold">cnn_mnist</a> (<a href="matlab: opentoline('E:\projects\DataProcessing\matconvnet-1.0-beta25\examples\mnist\cnn_mnist.m',54,0)">line 54</a>)
[net, info] = trainfn(net, imdb, getBatch(opts), ...

Error in <a href="matlab:matlab.internal.language.introspective.errorDocCallback('nnmnist/valErrorRate', 'E:\projects\DataProcessing\matconvnet-1.0-beta25\matlab\xtest\suite\nnmnist.m', 29)" style="font-weight:bold">nnmnist/valErrorRate</a> (<a href="matlab: opentoline('E:\projects\DataProcessing\matconvnet-1.0-beta25\matlab\xtest\suite\nnmnist.m',29,0)">line 29</a>)
      [~, info] = cnn_mnist('train', trainOpts, 'networkType', networkType);

Error occurred in nnpool[dataType=single,device=gpu]/basic(poolx=value1,pooly=value1) and it did not run to completion.

---------
Error ID:
---------
''

--------------
Error Details:
--------------
Error using <a href="matlab:matlab.internal.language.introspective.errorDocCallback('vl_nnpool')" style="font-weight:bold">vl_nnpool</a>
vl::impl::dispatch_cudnn<C, CU>::operator(): nnpooling_cudnn::forward: cuDNN error [cudnn: 112:"e:\\projects\\dataprocessing\\matconvnet-1.0-beta25\\matlab\\src\\bits\\nnpooling_cudnn.cu" (CUDNN_STATUS_EXECUTION_FAILED)]

Error in <a href="matlab:matlab.internal.language.introspective.errorDocCallback('nnpool/basic', 'E:\projects\DataProcessing\matconvnet-1.0-beta25\matlab\xtest\suite\nnpool.m', 42)" style="font-weight:bold">nnpool/basic</a> (<a href="matlab: opentoline('E:\projects\DataProcessing\matconvnet-1.0-beta25\matlab\xtest\suite\nnpool.m',42,0)">line 42</a>)
      y = vl_nnpool(x,pool,args{:}) ;

Any ideas? Thanks

[JohnsonZ-microe]

I have exactly the same issue, waiting for a good answer.