BVLC / caffe

Caffe: a fast open framework for deep learning.
http://caffe.berkeleyvision.org/
Other
34.09k stars 18.7k forks source link

RTX2080 + cuda10 runtest error #6768

Open iiiHunter opened 5 years ago

iiiHunter commented 5 years ago

When I run make runtest, I met the problem below: [ RUN ] AdaGradSolverTest/2.TestAdaGradLeastSquaresUpdateWithWeightDecay I0521 09:34:12.293684 18124 solver.cpp:391] Solving TestNetwork I0521 09:34:12.293690 18124 solver.cpp:392] Learning Rate Policy: fixed I0521 09:34:12.293702 18124 solver.cpp:434] Optimization Done. F0521 09:34:12.294821 18124 math_functions.cu:26] Check failed: status == CUBLAS_STATUS_SUCCESS (13 vs. 0) CUBLAS_STATUS_EXECUTION_FAILED Check failure stack trace: @ 0x7f49347330cd google::LogMessage::Fail() @ 0x7f4934734f33 google::LogMessage::SendToLog() @ 0x7f4934732c28 google::LogMessage::Flush() @ 0x7f4934735999 google::LogMessageFatal::~LogMessageFatal() @ 0x7f492de327ae caffe::caffe_gpu_gemm<>() @ 0x7f492de73c8d caffe::InnerProductLayer<>::Forward_gpu() @ 0x7f492dc8d8e2 caffe::Net<>::ForwardFromTo() @ 0x7f492dc8d9e7 caffe::Net<>::Forward() @ 0x55fbd8ca0362 caffe::GradientBasedSolverTest<>::ComputeLeastSquaresUpdate() @ 0x55fbd8cad1f3 caffe::GradientBasedSolverTest<>::TestLeastSquaresUpdate() @ 0x55fbd911199a testing::internal::HandleExceptionsInMethodIfSupported<>() @ 0x55fbd910ac0a testing::Test::Run() @ 0x55fbd910acec testing::TestInfo::Run() @ 0x55fbd910ae25 testing::TestCase::Run() @ 0x55fbd910b2e0 testing::internal::UnitTestImpl::RunAllTests() @ 0x55fbd910b427 testing::UnitTest::Run() @ 0x55fbd8c64601 main @ 0x7f492ce87b97 __libc_start_main @ 0x55fbd8c6c26a _start Makefile:533: recipe for target 'runtest' failed make: *** [runtest] Aborted (core dumped)

My GPU is RTX2080, cuda10.1, thanks!

heyonghong commented 5 years ago

我也遇到了相似问题。去cuda官网安装相应补丁https://developer.nvidia.com/cuda-90-download-archive解决了

matheusdefarias commented 5 years ago

When I run make runtest, I met the problem below: [ RUN ] AdaGradSolverTest/2.TestAdaGradLeastSquaresUpdateWithWeightDecay I0521 09:34:12.293684 18124 solver.cpp:391] Solving TestNetwork I0521 09:34:12.293690 18124 solver.cpp:392] Learning Rate Policy: fixed I0521 09:34:12.293702 18124 solver.cpp:434] Optimization Done. F0521 09:34:12.294821 18124 math_functions.cu:26] Check failed: status == CUBLAS_STATUS_SUCCESS (13 vs. 0) CUBLAS_STATUS_EXECUTION_FAILED Check failure stack trace: @ 0x7f49347330cd google::LogMessage::Fail() @ 0x7f4934734f33 google::LogMessage::SendToLog() @ 0x7f4934732c28 google::LogMessage::Flush() @ 0x7f4934735999 google::LogMessageFatal::~LogMessageFatal() @ 0x7f492de327ae caffe::caffe_gpu_gemm<>() @ 0x7f492de73c8d caffe::InnerProductLayer<>::Forward_gpu() @ 0x7f492dc8d8e2 caffe::Net<>::ForwardFromTo() @ 0x7f492dc8d9e7 caffe::Net<>::Forward() @ 0x55fbd8ca0362 caffe::GradientBasedSolverTest<>::ComputeLeastSquaresUpdate() @ 0x55fbd8cad1f3 caffe::GradientBasedSolverTest<>::TestLeastSquaresUpdate() @ 0x55fbd911199a testing::internal::HandleExceptionsInMethodIfSupported<>() @ 0x55fbd910ac0a testing::Test::Run() @ 0x55fbd910acec testing::TestInfo::Run() @ 0x55fbd910ae25 testing::TestCase::Run() @ 0x55fbd910b2e0 testing::internal::UnitTestImpl::RunAllTests() @ 0x55fbd910b427 testing::UnitTest::Run() @ 0x55fbd8c64601 main @ 0x7f492ce87b97 __libc_start_main @ 0x55fbd8c6c26a _start Makefile:533: recipe for target 'runtest' failed make: *** [runtest] Aborted (core dumped)

My GPU is RTX2080, cuda10.1, thanks!

Hi, I am working with Caffe too. My graphics card is a RTX 2070 and I am facing the same issue. Did you solve the problem?

andyzj2621 commented 3 years ago

I have the same problem ,math_functions.cu:26] Check failed: status == CUBLAS_STATUS_SUCCESS (13 vs. 0) CUBLAS_STATUS_EXECUTION_FAILED. when i use caffe via matlab. I slove the problem now .The reason is that my gpu_0 is busy (100% used). However ,when i set gup =1,because gup_1 is availvbale ,the same problem occurs. So ,i run the matlab code ----main.m ,when gpu_0 is available . I want to know ,Why does it use gup_0 inspire i set gup =1