segmentation fault in training

[jjzhu]

I1008 16:19:57.421929 26865 net.cpp:165] Memory required for data: 285605888 I1008 16:19:57.421936 26865 layer_factory.hpp:77] Creating layer bn1_0_0 I1008 16:19:57.421943 26865 net.cpp:100] Creating Layer bn1_0_0 I1008 16:19:57.421947 26865 net.cpp:434] bn1_0_0 <- conv1_0_0 I1008 16:19:57.421952 26865 net.cpp:408] bn1_0_0 -> bn1_0_0 I1008 16:19:57.422133 26865 net.cpp:150] Setting up bn1_0_0 I1008 16:19:57.422152 26865 net.cpp:157] Top shape: 4 16 128 256 (2097152) I1008 16:19:57.422157 26865 net.cpp:165] Memory required for data: 293994496 I1008 16:19:57.422164 26865 layer_factory.hpp:77] Creating layer prelu1_0_0 I1008 16:19:57.422173 26865 net.cpp:100] Creating Layer prelu1_0_0 I1008 16:19:57.422175 26865 net.cpp:434] prelu1_0_0 <- bn1_0_0 I1008 16:19:57.422181 26865 net.cpp:408] prelu1_0_0 -> prelu1_0_0 I1008 16:19:57.422938 26865 net.cpp:150] Setting up prelu1_0_0 I1008 16:19:57.422947 26865 net.cpp:157] Top shape: 4 16 128 256 (2097152) I1008 16:19:57.422968 26865 net.cpp:165] Memory required for data: 302383104 I1008 16:19:57.422976 26865 layer_factory.hpp:77] Creating layer conv1_0_1 I1008 16:19:57.422982 26865 net.cpp:100] Creating Layer conv1_0_1 I1008 16:19:57.422986 26865 net.cpp:434] conv1_0_1 <- prelu1_0_0 I1008 16:19:57.422991 26865 net.cpp:408] conv1_0_1 -> conv1_0_1 I1008 16:19:57.424350 26865 net.cpp:150] Setting up conv1_0_1 I1008 16:19:57.424381 26865 net.cpp:157] Top shape: 4 16 128 256 (2097152) I1008 16:19:57.424382 26865 net.cpp:165] Memory required for data: 310771712 I1008 16:19:57.424389 26865 layer_factory.hpp:77] Creating layer bn1_0_1 I1008 16:19:57.424397 26865 net.cpp:100] Creating Layer bn1_0_1 I1008 16:19:57.424401 26865 net.cpp:434] bn1_0_1 <- conv1_0_1 I1008 16:19:57.424418 26865 net.cpp:408] bn1_0_1 -> bn1_0_1 I1008 16:19:57.424620 26865 net.cpp:150] Setting up bn1_0_1 I1008 16:19:57.424625 26865 net.cpp:157] Top shape: 4 16 128 256 (2097152) I1008 16:19:57.424645 26865 net.cpp:165] Memory required for data: 319160320 I1008 16:19:57.424649 26865 layer_factory.hpp:77] Creating layer prelu1_0_1 I1008 16:19:57.424654 26865 net.cpp:100] Creating Layer prelu1_0_1 I1008 16:19:57.424656 26865 net.cpp:434] prelu1_0_1 <- bn1_0_1 I1008 16:19:57.424660 26865 net.cpp:408] prelu1_0_1 -> prelu1_0_1 I1008 16:19:57.425477 26865 net.cpp:150] Setting up prelu1_0_1 I1008 16:19:57.425487 26865 net.cpp:157] Top shape: 4 16 128 256 (2097152) I1008 16:19:57.425489 26865 net.cpp:165] Memory required for data: 327548928 I1008 16:19:57.425493 26865 layer_factory.hpp:77] Creating layer conv1_0_2 I1008 16:19:57.425501 26865 net.cpp:100] Creating Layer conv1_0_2 I1008 16:19:57.425504 26865 net.cpp:434] conv1_0_2 <- prelu1_0_1 I1008 16:19:57.425509 26865 net.cpp:408] conv1_0_2 -> conv1_0_2 I1008 16:19:57.426268 26865 net.cpp:150] Setting up conv1_0_2 I1008 16:19:57.426277 26865 net.cpp:157] Top shape: 4 64 128 256 (8388608) I1008 16:19:57.426280 26865 net.cpp:165] Memory required for data: 361103360 I1008 16:19:57.426285 26865 layer_factory.hpp:77] Creating layer bn1_0_2 I1008 16:19:57.426319 26865 net.cpp:100] Creating Layer bn1_0_2 I1008 16:19:57.426323 26865 net.cpp:434] bn1_0_2 <- conv1_0_2 I1008 16:19:57.426327 26865 net.cpp:408] bn1_0_2 -> bn1_0_2 I1008 16:19:57.426498 26865 net.cpp:150] Setting up bn1_0_2 I1008 16:19:57.426502 26865 net.cpp:157] Top shape: 4 64 128 256 (8388608) I1008 16:19:57.426506 26865 net.cpp:165] Memory required for data: 394657792 I1008 16:19:57.426512 26865 layer_factory.hpp:77] Creating layer drop1_0_3 Aborted at 1507504798 (unix time) try "date -d @1507504798" if you are using GNU date PC: @ 0x7fcff5b417a3 std::_Hashtable<>::clear() SIGSEGV (@0x9) received by PID 26865 (TID 0x7fd0429d0ac0) from PID 9; stack trace: @ 0x7fd03fffc4b0 (unknown) @ 0x7fcff5b417a3 std::_Hashtable<>::clear() @ 0x7fcff5b3a116 google::protobuf::DescriptorPool::FindFileByName() @ 0x7fcff5af3f98 google::protobuf::python::cdescriptor_pool::AddSerializedFile() @ 0x7fd040c347d0 PyEval_EvalFrameEx @ 0x7fd040d5d01c PyEval_EvalCodeEx @ 0x7fd040cb33dd (unknown) @ 0x7fd040c861e3 PyObject_Call @ 0x7fd040ca6ae5 (unknown) @ 0x7fd040c3d123 (unknown) @ 0x7fd040c861e3 PyObject_Call @ 0x7fd040c3113c PyEval_EvalFrameEx @ 0x7fd040d5d01c PyEval_EvalCodeEx @ 0x7fd040c2bb89 PyEval_EvalCode @ 0x7fd040cc01b4 PyImport_ExecCodeModuleEx @ 0x7fd040cc0b8f (unknown) @ 0x7fd040cc2300 (unknown) @ 0x7fd040cc25c8 (unknown) @ 0x7fd040cc36db PyImport_ImportModuleLevel @ 0x7fd040c3a698 (unknown) @ 0x7fd040c861e3 PyObject_Call @ 0x7fd040d5c447 PyEval_CallObjectWithKeywords @ 0x7fd040c2f5c6 PyEval_EvalFrameEx @ 0x7fd040d5d01c PyEval_EvalCodeEx @ 0x7fd040c2bb89 PyEval_EvalCode @ 0x7fd040cc01b4 PyImport_ExecCodeModuleEx @ 0x7fd040cc0b8f (unknown) @ 0x7fd040cc2300 (unknown) @ 0x7fd040cc25c8 (unknown) @ 0x7fd040cc36db PyImport_ImportModuleLevel @ 0x7fd040c3a698 (unknown) @ 0x7fd040c861e3 PyObject_Call Segmentation fault (core dumped)

any idea?

Thank you, -Victor

[jjzhu]

resolved by downgrading protobuf from 3.4.0 to 3.1 sudo -H pip install --upgrade protobuf==3.1.0.post1

[An-Pan]

@jjzhu I got the same error. I am using protobuf 3.5.1,So Only 3.1.0 can work?

[lucasjinreal]

however another error occured:

", line 3, in <module>
    import caffe, json
  File "/media/jintian/53e40f02-ade2-4d49-97c4-93fb0f872981/Netac/ai/career/driveless/uisee/lab/ENet/caffe-enet/python/caffe/__init__.py", line 1, in <module>
    from .pycaffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, RMSPropSolver, AdaDeltaSolver, AdamSolver
  File "/media/jintian/53e40f02-ade2-4d49-97c4-93fb0f872981/Netac/ai/career/driveless/uisee/lab/ENet/caffe-enet/python/caffe/pycaffe.py", line 15, in <module>
    import caffe.io
  File "/media/jintian/53e40f02-ade2-4d49-97c4-93fb0f872981/Netac/ai/career/driveless/uisee/lab/ENet/caffe-enet/python/caffe/io.py", line 8, in <module>
    from caffe.proto import caffe_pb2
  File "/media/jintian/53e40f02-ade2-4d49-97c4-93fb0f872981/Netac/ai/career/driveless/uisee/lab/ENet/caffe-enet/python/caffe/proto/caffe_pb2.py", line 853, in <module>
    options=_descriptor._ParseOptions(descriptor_pb2.FieldOptions(), _b('\020\001')), file=DESCRIPTOR),
TypeError: __init__() got an unexpected keyword argument 'file'

[mingx9527]

The Error is caused by data. you can check your cityscapes data, no have groundtruth: xxx_xxx_xxx_gtFine_labelTrainIds.png you must: git clone https://github.com/mcordts/cityscapesScripts.git and then run the /preparation/createTrainIdLabelImags.py to generate xxx_xxx_xxx_gtFine_labelTrainIds.png As shown in the Tutorial