search

weiliu89 / caffe

Caffe: a fast open framework for deep learning.
http://caffe.berkeleyvision.org/
Other
4.77k stars 1.67k forks source link

Check failed: error == cudaSuccess (48 vs. 0) #952

Open ToPP13 opened 5 years ago

ToPP13 commented 5 years ago

Issue summary

I've used JetPack 4.1 to install OS, drivers, CUDA, CUDNN, OpenCV Other prerequisites I have installed manually (glfags, protobuf, boost and so on) I've compiled current Caffe fork successfully on NVIDIA Jetson AGX Xavier, howerver I've got errors in testing both "make runtest" and .build//tools/caffe -time .... Whenever gpu is used i got such errors: F1105 14:38:07.009965 11364 im2col.cu:231] Check failed: error == cudaSuccess (48 vs. 0) no kernel image is available for execution on the device Check failure stack trace: @ 0x7fafbcc128 google::LogMessage::Fail() @ 0x7fafbcdf98 google::LogMessage::SendToLog() @ 0x7fafbcbc90 google::LogMessage::Flush() @ 0x7fafbce83c google::LogMessageFatal::~LogMessageFatal() @ 0x7fadebc690 caffe::im2col_nd_gpu<>() @ 0x7fadd9f674 caffe::BaseConvolutionLayer<>::conv_im2col_gpu() @ 0x7fadd9f848 caffe::BaseConvolutionLayer<>::forward_gpu_gemm() @ 0x7fade71328 caffe::ConvolutionLayer<>::Forward_gpu() @ 0x5572240fa0 caffe::Layer<>::Forward() @ 0x557224313c caffe::GradientChecker<>::CheckGradientSingle() @ 0x5572244254 caffe::GradientChecker<>::CheckGradientExhaustive() @ 0x55723015e8 caffe::ConvolutionLayerTest_TestGradient3D_Test<>::TestBody() @ 0x55727777ec testing::internal::HandleExceptionsInMethodIfSupported<>() @ 0x55727709d4 testing::Test::Run() @ 0x5572770a9c testing::TestInfo::Run() @ 0x5572770bcc testing::TestCase::Run() @ 0x5572771098 testing::internal::UnitTestImpl::RunAllTests() @ 0x55727711f8 testing::UnitTest::Run() @ 0x5572233ca8 main @ 0x7fad7fc6e0 __libc_start_main @ 0x557223d854 (unknown) Makefile:526: recipe for target 'runtest' failed make: *** [runtest] Aborted (core dumped)

or

... caffe pooling_layer.cu:212] Check failed: error == cudaSuccess (48 vs. 0) no kernel image is available for execution on the device ...

Nvidia has hints aboutthis code: https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html cudaErrorNoKernelImageForDevice = 48 This indicates that there is no kernel image available that is suitable for the device. This can occur when a user specifies code generation options for a particular CUDA source file that do not include the corresponding device configuration.

./build/tools/caffe with cpu works fine with no error.

So i suspect the problem i cannot run on cuda 10 caffe, or i've missed something in building the caffe with the CUDA 10. May be i should somehow change build keys, such as CUDA_ARCH?

I've built successfully caffe and tested ssd in 2017 with CUDA 8.0 and it works fine. No i got really frustrated. I can try to build everything for CUDA 8.0 but it will take time. Does anyone knows what is the solution for such problem?

Steps to reproduce

Ive built caffe

Your system configuration

Operating system: Ubuntu 18.04 for Jetson AGX Xavier Compiler: gcc version 7.3.0 (Ubuntu/Linaro 7.3.0-27ubuntu1~18.04) CUDA version (if applicable): 10.0 CUDNN version (if applicable): 7.3 BLAS: OpenBLAS Python or MATLAB version (for pycaffe and matcaffe respectively):

yvanwy commented 5 years ago

I meet the same problem. My CUDA version is 9.0. How can I modify the file of "Makefile.config"?

yiboliu31 commented 5 years ago

Have you fixed this problem?

ToPP13 commented 5 years ago

I haven't, unfortunately. 

Четверг, 21 февраля 2019, 9:33 +03:00 от yiboliu31 notifications@github.com:

Have you fixed this problem? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub , or mute the thread .

-- Кирилл Мартынович