sanghoon / pva-faster-rcnn

Demo code for PVANet
https://arxiv.org/abs/1611.08588
Other
651 stars 241 forks source link

crashed when i training my own data..., please help... #16

Closed xiaoxiongli closed 7 years ago

xiaoxiongli commented 7 years ago

When i train my own data(229+1 category), when i run to here: ' self.solver = caffe.SGDSolver(solver_prototxt) in train.py line 43, crash happened. and i can not find where the caffe's log is. How can i enable the caffe's log in this python project?

I1020 17:44:25.036016 9109 net.cpp:228] data_input-data_0_split does not need backward computation. I1020 17:44:25.036031 9109 net.cpp:228] input-data does not need backward computation. I1020 17:44:25.036041 9109 net.cpp:270] This network produces output cls_loss I1020 17:44:25.036052 9109 net.cpp:270] This network produces output loss_bbox I1020 17:44:25.036065 9109 net.cpp:270] This network produces output rpn_cls_loss I1020 17:44:25.036077 9109 net.cpp:270] This network produces output rpn_loss_bbox I1020 17:44:25.036396 9109 net.cpp:283] Network initialization done. I1020 17:44:25.037508 9109 solver.cpp:60] Solver scaffolding done. Loading pretrained model weights from models/pvanet/imagenet/original.model I1020 17:44:26.811261 9109 net.cpp:761] Ignoring source layer data I1020 17:44:26.811317 9109 net.cpp:761] Ignoring source layer label_data_1_split I1020 17:44:26.814769 9109 net.cpp:761] Ignoring source layer pool5 I1020 17:44:26.879279 9109 net.cpp:761] Ignoring source layer fc8 I1020 17:44:26.879341 9109 net.cpp:761] Ignoring source layer fc8_fc8_0_split I1020 17:44:26.879362 9109 net.cpp:761] Ignoring source layer loss I1020 17:44:26.879374 9109 net.cpp:761] Ignoring source layer accuracy I1020 17:44:26.879390 9109 net.cpp:761] Ignoring source layer accuracy_top5 Traceback (most recent call last): File "tools/train_net.py", line 112, in max_iters=args.max_iters) File "/disk/SX/pva-faster-rcnn/tools/../lib/fast_rcnn/train.py", line 157, in train_net pretrained_model=pretrained_model) File "/disk/SX/pva-faster-rcnn/tools/../lib/fast_rcnn/train.py", line 53, in init self.solver.net.layers[0].set_roidb(roidb) File "/disk/SX/pva-faster-rcnn/tools/../lib/roi_data_layer/layer.py", line 68, in set_roidb self._shuffle_roidb_inds() File "/disk/SX/pva-faster-rcnn/tools/../lib/roi_data_layer/layer.py", line 35, in _shuffle_roidb_inds inds = np.reshape(inds, (-1, 2)) File "/usr/local/lib/python2.7/dist-packages/numpy/core/fromnumeric.py", line 225, in reshape return reshape(newshape, order=order) ValueError: total size of new array must be unchanged

xiaoxiongli commented 7 years ago

it is a issue of VSCode, when crash in self.solver = caffe.SGDSolver(solver_prototxt), the VSCode give no tips about the crash reason.

then i install a PyCharm, it print like below, so i find out the crash reason.

[ 0.1 0.1 0.2 0.2] [ 0.1 0.1 0.2 0.2] [ 0.1 0.1 0.2 0.2]] [ 0.1 0.1 0.2 0.2] Normalizing targets done F1021 13:51:14.622349 1611 io.cpp:36] Check failed: fd != -1 (-1 vs. -1) File not found: models/pvanet/example_train_384_logo229/train.prototxt * Check failure stack trace: * @ 0x7f7ec6389ddd google::LogMessage::Fail() @ 0x7f7ec638bc90 google::LogMessage::SendToLog() @ 0x7f7ec63899a2 google::LogMessage::Flush() @ 0x7f7ec638c6ae google::LogMessageFatal::~LogMessageFatal() @ 0x7f7ec6781508 caffe::ReadProtoFromTextFile() @ 0x7f7ec677d7f6 caffe::ReadNetParamsFromTextFileOrDie() @ 0x7f7ec68dc40b caffe::Solver<>::InitTrainNet() @ 0x7f7ec68dd891 caffe::Solver<>::Init() @ 0x7f7ec68ddd76 caffe::Solver<>::Solver() @ 0x7f7ec715341d boost::python::objects::make_holder<>::apply<>::execute() @ 0x7f7ec714e543 boost::python::objects::caller_py_function_impl<>::operator()() @ 0x7f7ec591878d (unknown) @ 0x7f7ec59189a8 (unknown) @ 0x7f7ec5922433 (unknown) @ 0x7f7ec5917189 (unknown) @ 0x50b968 (unknown) @ 0x50aa03 (unknown) @ 0x4bd28b (unknown) @ 0x4cd9ab (unknown) @ 0x4e7cc8 (unknown) @ 0x50b968 (unknown) @ 0x50aa03 (unknown) @ 0x4bd28b (unknown) @ 0x4cd9ab (unknown) @ 0x4cb6b1 (unknown) @ 0x4cd217 (unknown) @ 0x4cb6b1 (unknown) @ 0x50481f (unknown) @ 0x4fc182 (unknown) @ 0x575644 (unknown) @ 0x4d075a (unknown) @ 0x4cb6b1 (unknown)

Process finished with exit code 134 (interrupted by signal 6: SIGABRT)