Follow installation with nvidia-docker at tutorial, after pulling latest image (GPU support, CUDA 8.0, cuDNN 7) I ran test:
nvidia-docker run -it caffe2ai/caffe2:latest python -m caffe2.python.operator_test.relu_op_test
But error came in:
Trying example: test_relu(self=<__main__.TestRelu testMethod=test_relu>, X=array([-0.98402572, -0.10023266, 0.5741834 , -0.5871824 ], dtype=float32), gc=, dc=[, device_type: 1], engine=u'CUDNN')
Trying example: test_relu(self=<__main__.TestRelu testMethod=test_relu>, X=array([ 0.], dtype=float32), gc=device_type: 1, dc=[, device_type: 1], engine=u'')
Failure in checking device option 1 and output Y. The outputs are:
[ 0.]
[ 0.02]
0.02
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/hypothesis/core.py", line 523, in evaluate_test_data
self.search_strategy, self.test,
File "/usr/local/lib/python2.7/dist-packages/hypothesis/executors.py", line 58, in default_new_style_executor
return function(data)
File "/usr/local/lib/python2.7/dist-packages/hypothesis/core.py", line 115, in run
return test(*args, **kwargs)
File "/usr/local/caffe2/python/operator_test/relu_op_test.py", line 26, in test_relu
self.assertDeviceChecks(dc, op, [X], [0])
File "/usr/local/caffe2/python/hypothesis_test_util.py", line 325, in assertDeviceChecks
dc.CheckSimple(op, inputs, outputs_to_check, input_device_options)
File "/usr/lib/python2.7/unittest/case.py", line 422, in assertTrue
raise self.failureException(msg)
AssertionError: False is not true
Trying example: test_relu(self=<__main__.TestRelu testMethod=test_relu>, X=array([ 0.], dtype=float32), gc=device_type: 1, dc=[, device_type: 1], engine=u'')
Failure in checking device option 1 and output Y. The outputs are:
[ 0.]
[ 0.02]
0.02
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/hypothesis/core.py", line 523, in evaluate_test_data
self.search_strategy, self.test,
File "/usr/local/lib/python2.7/dist-packages/hypothesis/executors.py", line 58, in default_new_style_executor
return function(data)
File "/usr/local/lib/python2.7/dist-packages/hypothesis/core.py", line 115, in run
return test(*args, **kwargs)
File "/usr/local/caffe2/python/operator_test/relu_op_test.py", line 26, in test_relu
self.assertDeviceChecks(dc, op, [X], [0])
File "/usr/local/caffe2/python/hypothesis_test_util.py", line 325, in assertDeviceChecks
dc.CheckSimple(op, inputs, outputs_to_check, input_device_options)
File "/usr/lib/python2.7/unittest/case.py", line 422, in assertTrue
raise self.failureException(msg)
AssertionError: False is not true
Trying example: test_relu(self=<__main__.TestRelu testMethod=test_relu>, X=array([ 0.], dtype=float32), gc=, dc=[, device_type: 1], engine=u'')
Failure in checking device option 1 and output Y. The outputs are:
[ 0.]
[ 0.02]
0.02
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/hypothesis/core.py", line 523, in evaluate_test_data
self.search_strategy, self.test,
File "/usr/local/lib/python2.7/dist-packages/hypothesis/executors.py", line 58, in default_new_style_executor
return function(data)
File "/usr/local/lib/python2.7/dist-packages/hypothesis/core.py", line 115, in run
return test(*args, **kwargs)
File "/usr/local/caffe2/python/operator_test/relu_op_test.py", line 26, in test_relu
self.assertDeviceChecks(dc, op, [X], [0])
File "/usr/local/caffe2/python/hypothesis_test_util.py", line 325, in assertDeviceChecks
dc.CheckSimple(op, inputs, outputs_to_check, input_device_options)
File "/usr/lib/python2.7/unittest/case.py", line 422, in assertTrue
raise self.failureException(msg)
AssertionError: False is not true
Falsifying example: test_relu(self=<__main__.TestRelu testMethod=test_relu>, X=array([ 0.], dtype=float32), gc=, dc=[, device_type: 1], engine=u'')
Failure in checking device option 1 and output Y. The outputs are:
[ 0.]
[ 0.02]
0.02
F
======================================================================
FAIL: test_relu (__main__.TestRelu)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/caffe2/python/operator_test/relu_op_test.py", line 19, in test_relu
engine=st.sampled_from(["", "CUDNN"]),
File "/usr/local/lib/python2.7/dist-packages/hypothesis/core.py", line 721, in wrapped_test
state.run()
File "/usr/local/lib/python2.7/dist-packages/hypothesis/core.py", line 614, in run
print_example=True, is_final=True
File "/usr/local/lib/python2.7/dist-packages/hypothesis/executors.py", line 58, in default_new_style_executor
return function(data)
File "/usr/local/lib/python2.7/dist-packages/hypothesis/core.py", line 115, in run
return test(*args, **kwargs)
File "/usr/local/caffe2/python/operator_test/relu_op_test.py", line 26, in test_relu
self.assertDeviceChecks(dc, op, [X], [0])
File "/usr/local/caffe2/python/hypothesis_test_util.py", line 325, in assertDeviceChecks
dc.CheckSimple(op, inputs, outputs_to_check, input_device_options)
AssertionError: False is not true
----------------------------------------------------------------------
Ran 1 test in 6.212s
Ignore test, try to run char_rnn.py example with:
nvidia-docker run -it caffe2ai/caffe2:latest /bin/bash
# after getting data & main py file
python char_rnn.py --train_data shakespeare.txt --gpu
Error as expected:
Input has 62 characters. Total input size: 99993
DEBUG:char_rnn:Start training
DEBUG:char_rnn:Training model
Traceback for operator 10 in network char_rnn
Entering interactive debugger. Type "bt" to print the full stacktrace. Type "help" to see command listing.
[enforce fail at reshape_op.h:96] total_size == size. 1550 vs 0. Argument `shape` does not agree with the input data. (1550 != 0) Error from operator:
input: "softmax_reshaped_grad" input: "_" output: "softmax_grad" output: "_softmax_grad_dims" name: "" type: "Reshape" device_option { device_type: 1 cuda_gpu_id: 0 } is_gradient_op: true
> /usr/local/caffe2/python/workspace.py(166)CallWithExceptionIntercept()
-> return func(*args, **kwargs)
(Pdb) bt
/usr/local/caffe2/python/utils.py(258)run()
-> return func()
/usr/local/caffe2/python/utils.py(290)func()
-> return f(*args, **kwargs)
/home/char_rnn.py(286)main()
-> model.TrainModel()
/home/char_rnn.py(197)TrainModel()
-> workspace.RunNet(self.model.net.Name())
/usr/local/caffe2/python/workspace.py(201)RunNet()
-> StringifyNetName(name), num_iter, allow_fail,
> /usr/local/caffe2/python/workspace.py(166)CallWithExceptionIntercept()
-> return func(*args, **kwargs)
Follow installation with
nvidia-docker
at tutorial, after pulling latest image (GPU support, CUDA 8.0, cuDNN 7) I ran test:But error came in:
Ignore test, try to run
char_rnn.py
example with:Error as expected:
What went wrong?