search

zhaoweicai / mscnn

Caffe implementation of our multi-scale object detection framework
404 stars 211 forks source link

Check failed: error == cudaSuccess (2 vs. 0) out of memory #94

Open Hub-Tian opened 6 years ago

Hub-Tian commented 6 years ago

Check failed: error == cudaSuccess (2 vs. 0) out of memory Check failure stack trace: @ 0x7f6d6f7dce6d (unknown) @ 0x7f6d6f7deced (unknown) @ 0x7f6d6f7dca5c (unknown) @ 0x7f6d6f7df63e (unknown) @ 0x7f6d7699a2c1 caffe::SyncedMemory::to_gpu() @ 0x7f6d76999629 caffe::SyncedMemory::mutable_gpu_data() @ 0x7f6d7680e1a2 caffe::Blob<>::mutable_gpu_data() @ 0x7f6d76842448 caffe::BaseConvolutionLayer<>::forward_gpu_gemm() @ 0x7f6d769ce826 caffe::ConvolutionLayer<>::Forward_gpu() @ 0x7f6d76963215 caffe::Net<>::ForwardFromTo() @ 0x7f6d76963587 caffe::Net<>::Forward() @ 0x7f6d76981b67 caffe::Solver<>::Step() @ 0x7f6d76982429 caffe::Solver<>::Solve() @ 0x408adb train() @ 0x405f8c main @ 0x7f6d6b59baf5 __libc_start_main @ 0x4067fd (unknown) I0221 13:19:16.887039 26191 caffe.cpp:217] Using GPUs 0 I0221 13:19:17.773277 26191 caffe.cpp:222] GPU 0: Tesla K80 I0221 13:19:18.569195 26191 solver.cpp:48] Initializing solver from parameters:

I am using the original code downloaded from this repository to train a car detector on KITTI. My GPU is Tesla K80. I ran train_mscnn.sh under /kitti_car/mscnn-7s-576-2x/ but the error message suggested that the memory is not enough. And it can not be solved by using more GPUs, for example 4. I wonder what I should change to handle this problem.

zhaoweicai commented 6 years ago

A single K80 is enough for all model training. Please see https://github.com/zhaoweicai/mscnn/issues/7