Closed psyhtest closed 8 years ago
CL_INVALID_VALUE actually can mean two things:
Since "Convolution_Spatial" is a convolution implementation by Intel, I will pass this on to them. @gongzg would you like to have a look at this?
@naibaf7 I would like to check it. As I don't have the environment to reproduce this issue, @psyhtest could you please provide more information on it. Could you rebuild a debug version, and run the case under gdb then share the back trace stack to me. That will make things clearer. Thanks.
@gongzg With pleasure.
As the failures appear identical, I'm debugging:
LD_LIBRARY_PATH=/data/install/lib-openblas-v0.2.18/lib:$LD_LIBRARY_PATH \
/data/caffe-naibaf7-issue#31/build/test/test_all.testbin \
--gtest_filter=ConvolutionLayerTest_Spatial/1.TestSimpleConvolution_Spatial
as follows:
$ gdb
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
...
(gdb) file /data/caffe-naibaf7-issue#31/build/test/test_all.testbin
(gdb) set args --gtest_filter=ConvolutionLayerTest_Spatial/1.TestSimpleConvolution_Spatial
(gdb) set env LD_LIBRARY_PATH=/data/install/lib-openblas-v0.2.18/lib:$LD_LIBRARY_PATH
(gdb) catch throw
Catchpoint 1 (throw)
(gdb) r
Starting program: /data/caffe-naibaf7-issue#31/.build_debug/test/test_all.testbin --gtest_filter=ConvolutionLayerTest_Spatial/1.TestSimpleConvolution_Spatial
...
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from ConvolutionLayerTest_Spatial/1, where TypeParam = caffe::GPUDevice<float>
[ RUN ] ConvolutionLayerTest_Spatial/1.TestSimpleConvolution_Spatial
Catchpoint 1 (exception thrown), 0xb689b292 in __cxa_throw () from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
(gdb) bt
#0 0xb689b292 in __cxa_throw () from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
#1 0x00143de4 in viennacl::ocl::error_checker<void>::raise_exception (err=-30)
at /data/install/lib-viennacl-release-1.7.1/viennacl/ocl/error.hpp:610
#2 0x00141e8e in viennacl::ocl::error_checker<void>::checkError (err=-30) at /data/install/lib-viennacl-release-1.7.1/viennacl/ocl/error.hpp:675
#3 0xb5345a32 in viennacl::ocl::context::create_memory_without_smart_handle (this=0x8c5720, flags=40, size=4, ptr=0xbeffeb60)
at /data/install/lib-viennacl-release-1.7.1/viennacl/ocl/context.hpp:205
#4 0xb53d0330 in viennacl::ocl::context::create_memory (this=0x8c5720, flags=8, size=4, ptr=0xbeffeb60)
at /data/install/lib-viennacl-release-1.7.1/viennacl/ocl/context.hpp:218
#5 0xb53cd5f0 in caffe::ConvolutionLayerSpatial<float>::verify_result (this=0x10526e0, bottom=std::vector of length 2, capacity 2 = {...},
top=std::vector of length 2, capacity 2 = {...}, index=0, numImages=2, config=0x18685c8) at src/caffe/layers/conv_layer_spatial.cu:688
#6 0xb53cedd0 in caffe::ConvolutionLayerSpatial<float>::setup_convolution (this=0x10526e0, bottom=std::vector of length 2, capacity 2 = {...},
top=std::vector of length 2, capacity 2 = {...}) at src/caffe/layers/conv_layer_spatial.cu:1004
#7 0xb53cf732 in caffe::ConvolutionLayerSpatial<float>::Forward_gpu (this=0x10526e0, bottom=std::vector of length 2, capacity 2 = {...},
top=std::vector of length 2, capacity 2 = {...}) at src/caffe/layers/conv_layer_spatial.cu:1109
#8 0x00169aee in caffe::Layer<float>::Forward (this=0x10526e0, bottom=std::vector of length 2, capacity 2 = {...},
top=std::vector of length 2, capacity 2 = {...}) at ./include/caffe/layer.hpp:522
#9 0x0046eda0 in caffe::ConvolutionLayerTest_Spatial_TestSimpleConvolution_Spatial_Test<caffe::GPUDevice<float> >::TestBody_Impl (this=0x12eda88)
at src/caffe/test/test_convolution_layer_spatial.cpp:218
#10 0x0046ec5e in caffe::ConvolutionLayerTest_Spatial_TestSimpleConvolution_Spatial_Test<caffe::GPUDevice<float> >::TestBody (this=0x12eda88)
at src/caffe/test/test_convolution_layer_spatial.cpp:202
#11 0x004e0272 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (object=0x12eda88,
method=&virtual testing::Test::TestBody(), location=0x6b4e3c "the test body") at src/gtest/gtest-all.cpp:3393
#12 0x004dcdc0 in testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=0x12eda88,
method=&virtual testing::Test::TestBody(), location=0x6b4e3c "the test body") at src/gtest/gtest-all.cpp:3429
#13 0x004cdf2a in testing::Test::Run (this=0x12eda88) at src/gtest/gtest-all.cpp:3465
#14 0x004ce494 in testing::TestInfo::Run (this=0x88c828) at src/gtest/gtest-all.cpp:3641
#15 0x004ce8e4 in testing::TestCase::Run (this=0x88c608) at src/gtest/gtest-all.cpp:3748
#16 0x004d2346 in testing::internal::UnitTestImpl::RunAllTests (this=0x80d9f8) at src/gtest/gtest-all.cpp:5540
#17 0x004e0bae in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x80d9f8,
method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x4d2185 <testing::internal::UnitTestImpl::RunAllTests()>, location=0x6b5908 "auxiliary test code (environments or event listeners)") at src/gtest/gtest-all.cpp:3393
#18 0x004dd684 in testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x80d9f8,
method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x4d2185 <testing::internal::UnitTestImpl::RunAllTests()>, location=0x6b5908 "auxiliary test code (environments or event listeners)") at src/gtest/gtest-all.cpp:3429
#19 0x004d177a in testing::UnitTest::Run (this=0x78e584 <_ZZN7testing8UnitTest11GetInstanceEvE8instance>) at src/gtest/gtest-all.cpp:5177
#20 0x0014165a in main (argc=1, argv=0xbefff464) at src/caffe/test/test_caffe_main.cpp:96
(gdb) c
unknown file: Failure
C++ exception with description "ViennaCL: FATAL ERROR: CL_INVALID_VALUE.
If you think that this is a bug in ViennaCL, please report it at viennacl-support@lists.sourceforge.net and supply at least the following information:
* Operating System
* Which OpenCL implementation (AMD, NVIDIA, etc.)
* ViennaCL version
Many thanks in advance!" thrown in the test body.
[ FAILED ] ConvolutionLayerTest_Spatial/1.TestSimpleConvolution_Spatial, where TypeParam = caffe::GPUDevice<float> (78621 ms)
[----------] 1 test from ConvolutionLayerTest_Spatial/1 (78621 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (78622 ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] ConvolutionLayerTest_Spatial/1.TestSimpleConvolution_Spatial, where TypeParam = caffe::GPUDevice<float>
1 FAILED TEST
Any ideas?
@psyhtest It looks like an issue in Viennacl which I has fixed a few weeks ago. Could you check your viennacl version? I put a pointer in the readme as below:
"Please use the latest git master viennacl which has the patch: https://github.com/viennacl/viennacl-dev/pull/181"
@gongzg Unfortunately, using the latest ViennaCL master doesn't seem to have helped on the Odroid XU3. I'll double check on the Chromebook 2.
@psyhtest did you rebuild caffe after upgrade viennacl? and could you check the back trace with the latest viennacl, is there any difference? thanks.
@gongzg I thought I had made a clean build, but I rebuilt everything from scratch and it worked both on the Chromebook 2 and Odroid XU3. I saw a couple of weird messages "Verification of kernel was not successful,fallback to basic kernel" on the Chromebook on the first execution but not any more:
$ LD_LIBRARY_PATH=/data/install/lib-openblas-v0.2.18/lib:$LD_LIBRARY_PATH \
/data/caffe-naibaf7/build/test/test_all.testbin \
--gtest_filter=ConvolutionLayerTest_Spatial/1.TestSimpleConvolution_Spatial*
Setting to use device 0
Note: Google Test filter = ConvolutionLayerTest_Spatial/1.TestSimpleConvolution_Spatial*
[==========] Running 9 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 9 tests from ConvolutionLayerTest_Spatial/1, where TypeParam = caffe::GPUDevice<float>
[ RUN ] ConvolutionLayerTest_Spatial/1.TestSimpleConvolution_Spatial
[ OK ] ConvolutionLayerTest_Spatial/1.TestSimpleConvolution_Spatial (488 ms)
[ RUN ] ConvolutionLayerTest_Spatial/1.TestSimpleConvolution_Spatial3x3
Verification of kernel was not successful,fallback to basic kernel
[ OK ] ConvolutionLayerTest_Spatial/1.TestSimpleConvolution_Spatial3x3 (8233 ms)
[ RUN ] ConvolutionLayerTest_Spatial/1.TestSimpleConvolution_Spatial3x3xPad1
Verification of kernel was not successful,fallback to basic kernel
[ OK ] ConvolutionLayerTest_Spatial/1.TestSimpleConvolution_Spatial3x3xPad1 (2278 ms)
[ RUN ] ConvolutionLayerTest_Spatial/1.TestSimpleConvolution_Spatial11x11x1x2_caffenet_Conv1
Verification of kernel was not successful,fallback to basic kernel
[ OK ] ConvolutionLayerTest_Spatial/1.TestSimpleConvolution_Spatial11x11x1x2_caffenet_Conv1 (2462 ms)
[ RUN ] ConvolutionLayerTest_Spatial/1.TestSimpleConvolution_Spatial5x5x1x2_caffenet_Conv2
Verification of kernel was not successful,fallback to basic kernel
[ OK ] ConvolutionLayerTest_Spatial/1.TestSimpleConvolution_Spatial5x5x1x2_caffenet_Conv2 (10895 ms)
[ RUN ] ConvolutionLayerTest_Spatial/1.TestSimpleConvolution_Spatial3x3x1_caffenet_Conv3
Verification of kernel was not successful,fallback to basic kernel
[ OK ] ConvolutionLayerTest_Spatial/1.TestSimpleConvolution_Spatial3x3x1_caffenet_Conv3 (7701 ms)
[ RUN ] ConvolutionLayerTest_Spatial/1.TestSimpleConvolution_Spatial3x3x1_caffenet_Conv4
Verification of kernel was not successful,fallback to basic kernel
[ OK ] ConvolutionLayerTest_Spatial/1.TestSimpleConvolution_Spatial3x3x1_caffenet_Conv4 (5803 ms)
[ RUN ] ConvolutionLayerTest_Spatial/1.TestSimpleConvolution_Spatial3x3x2_caffenet_Conv5
[ OK ] ConvolutionLayerTest_Spatial/1.TestSimpleConvolution_Spatial3x3x2_caffenet_Conv5 (6683 ms)
[ RUN ] ConvolutionLayerTest_Spatial/1.TestSimpleConvolution_Spatial5x5
[ OK ] ConvolutionLayerTest_Spatial/1.TestSimpleConvolution_Spatial5x5 (21090 ms)
[----------] 9 tests from ConvolutionLayerTest_Spatial/1 (65636 ms total)
[----------] Global test environment tear-down
[==========] 9 tests from 1 test case ran. (65637 ms total)
[ PASSED ] 9 tests.
Many thanks for your help!
3 out of 9 tests in
ConvolutionLayerTest_Spatial/1.TestSimpleConvolution_Spatial*
fail on at least two ARM platforms (Samsung Chromebook 2 and Odroid XU3) with e.g.It's clear that the problem is in ViennaCL (1.7.1), however, it's worth inspecting the Caffe tests to ensure that nothing obvious makes them fail.
caffe-naibaf7.6c0fbdc.odroid-xu3.Convolution_Spatial.txt caffe-naibaf7.6c0fbdc.chromebook-2.Convolution_Spatial.txt