search

martinkersner / train-CRF-RNN

Train CRF-RNN for Semantic Image Segmentation
Other
199 stars 92 forks source link

iteration 0, test infinitely #33

Open maohule opened 7 years ago

maohule commented 7 years ago

Hi @martinkersner I use the TVG_CRFRNN.sh which at caffe/examples/segmentationcrfasrnn/ to train crfasrnn.But the training iterate 0 and then test infinitely as follows. What should I do?Thanks!

I0102 21:24:32.579792 5267 upgrade_proto.cpp:620] Attempting to upgrade input file specified using deprecated V1LayerParameter: TVG_CRFRNN_COCO_VOC.caffemodel I0102 21:24:33.525709 5267 upgrade_proto.cpp:628] Successfully upgraded file specified using deprecated V1LayerParameter [libprotobuf WARNING google/protobuf/io/coded_stream.cc:537] Reading dangerously large protocol message. If the message turns out to be larger than 2147483647 bytes, parsing will be halted for security reasons. To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h. [libprotobuf WARNING google/protobuf/io/coded_stream.cc:78] The total number of bytes read was 537968303 I0102 21:24:34.088544 5267 upgrade_proto.cpp:620] Attempting to upgrade input file specified using deprecated V1LayerParameter: TVG_CRFRNN_COCO_VOC.caffemodel I0102 21:24:34.936173 5267 upgrade_proto.cpp:628] Successfully upgraded file specified using deprecated V1LayerParameter I0102 21:24:35.053526 5267 caffe.cpp:211] Starting Optimization I0102 21:24:35.053546 5267 solver.cpp:293] Solving CRFRNN-VOC I0102 21:24:35.053550 5267 solver.cpp:294] Learning Rate Policy: fixed I0102 21:24:35.057199 5267 solver.cpp:346] Iteration 0, Testing net (#0) I0102 21:35:39.176551 5267 solver.cpp:414] Test net output #0: label = 0.32 I0102 21:35:39.180371 5267 solver.cpp:414] Test net output #1: label = 0.32 I0102 21:35:39.180466 5267 solver.cpp:414] Test net output #2: label = 0.32 I0102 21:35:39.180511 5267 solver.cpp:414] Test net output #3: label = 0.32 I0102 21:35:39.180546 5267 solver.cpp:414] Test net output #4: label = 0.32 I0102 21:35:39.180575 5267 solver.cpp:414] Test net output #5: label = 0.32 I0102 21:35:39.180606 5267 solver.cpp:414] Test net output #6: label = 0.32 I0102 21:35:39.180635 5267 solver.cpp:414] Test net output #7: label = 0.32 I0102 21:35:39.180665 5267 solver.cpp:414] Test net output #8: label = 0.32 I0102 21:35:39.180696 5267 solver.cpp:414] Test net output #9: label = 0.32 . . . I0106 17:52:28.314189 12092 solver.cpp:414] Test net output #5499997: pred = 0.379582 I0106 17:52:28.314199 12092 solver.cpp:414] Test net output #5499998: pred = 0.369715 I0106 17:52:28.314213 12092 solver.cpp:414] Test net output #5499999: pred = 0.359848 F0106 17:52:28.376559 12092 syncedmem.cpp:58] Check failed: error == cudaSuccess (2 vs. 0) out of memory Check failure stack trace: @ 0x7f3b8c70295d google::LogMessage::Fail() @ 0x7f3b8c7046e0 google::LogMessage::SendToLog() @ 0x7f3b8c702543 google::LogMessage::Flush() @ 0x7f3b8c7050ae google::LogMessageFatal::~LogMessageFatal() @ 0x7f3b8ce43590 caffe::SyncedMemory::to_gpu() @ 0x7f3b8ce42579 caffe::SyncedMemory::mutable_gpu_data() @ 0x7f3b8cd58c82 caffe::Blob<>::mutable_gpu_data() @ 0x7f3b8cdb4238 caffe::BaseConvolutionLayer<>::forward_gpu_gemm() @ 0x7f3b8ce726e8 caffe::ConvolutionLayer<>::Forward_gpu() @ 0x7f3b8cd42462 caffe::Net<>::ForwardFromTo() @ 0x7f3b8cd42587 caffe::Net<>::ForwardPrefilled() @ 0x7f3b8cd322ad caffe::Solver<>::Step() @ 0x7f3b8cd32e3a caffe::Solver<>::Solve() @ 0x40c0cb train() @ 0x408830 main @ 0x7f3b8b54f830 (unknown) @ 0x408f89 _start Aborted (core dumped) Thank you very much!

fanguu commented 7 years ago

what's your GPU type? I face the problem too, my GPU is gtx1080 and the memory is 8G.

maohule commented 7 years ago

@fanguu My GPU is 1070. It is ralated to the size of your picture.You can try to use the piture which is smaller than 320*320.

KleinYuan commented 7 years ago

Similar case here