search

shurans / sscnet

Semantic Scene Completion from a Single Depth Image
http://sscnet.cs.princeton.edu/
339 stars 91 forks source link

'out of memory'problems #12

Closed CJohnny closed 7 years ago

CJohnny commented 7 years ago

Hello! I'm working on the Training step , and , both the '0.finetuning on NYU' and '1.Training from scratch' report the problem below: ' F0424 12:35:52.323909 28444 syncedmem.cpp:56] Check failed: error == cudaSuccess (2 vs. 0) out of memory ' Check failure stack trace: @ 0x7f58d9a4e5cd google::LogMessage::Fail() @ 0x7f58d9a50433 google::LogMessage::SendToLog() @ 0x7f58d9a4e15b google::LogMessage::Flush() @ 0x7f58d9a50e1e google::LogMessageFatal::~LogMessageFatal() @ 0x7f58da07e180 caffe::SyncedMemory::to_gpu() @ 0x7f58da07d1a9 caffe::SyncedMemory::mutable_gpu_data() @ 0x7f58da1e3b63 caffe::Blob<>::mutable_gpu_diff() @ 0x7f58da24a58b caffe::ConvolutionLayer<>::Backward_gpu() @ 0x7f58da08526b caffe::Net<>::BackwardFromTo() @ 0x7f58da0852bf caffe::Net<>::Backward() @ 0x7f58da072d3c caffe::Solver<>::Step() @ 0x7f58da073769 caffe::Solver<>::Solve() @ 0x40bdaf train() @ 0x4078d4 main @ 0x7f58d833c830 __libc_start_main @ 0x4081e9 _start @ (nil) (unknown) I'm working on a gtx1080Ti GPU. I do see a peak of memory usage in nvidia-smi which is 11154MiB/11169MiB, and then the problem is reported. I'm wondering if it is a coding problem and where should I fix . Thanks a lot!

Fromandto commented 7 years ago

According to my experience, during training the peak memory use is nearly 12GB. So 1080Ti is not a valid choice.

CJohnny commented 7 years ago

@Fromandto Wow. I see. Do you mind telling me details about the GPU you use? like type&number

Fromandto commented 7 years ago

@CJohnny well, actually you can just comment 'test_iter' and 'test_intervel' lines in your solver. The provided solver can only be trained on a 12gb memory gpu

CJohnny commented 7 years ago

@Fromandto All right! Knowing this is already important to me. Thank you!