search

lucasjinreal / kitti-ssd

Train your own data using SSD in a more clear and simple way(not include source code)
102 stars 44 forks source link

Check failed: sampler.min_scale() > 0. (0 vs. 0) #15

Open Tahirhan opened 6 years ago

Tahirhan commented 6 years ago

when i run "train_ssd_kitti.py" code begins to create layers. When Creating layer conv3_2 i get "Check failed: sampler.min_scale() > 0. (0 vs. 0)" . Despite this error code continues to creating layers but when Creating Layer con4_1 i get "Aborted (core dumped)" error and code stops. Can you help me ?

i get this after running "train_ssd_kitti.py" ;

I0220 00:55:57.175670 30617 layer_factory.hpp:77] Creating layer data I0220 00:55:57.175789 30617 net.cpp:100] Creating Layer data I0220 00:55:57.175801 30617 net.cpp:408] data -> data I0220 00:55:57.175834 30617 net.cpp:408] data -> label I0220 00:55:57.176373 30630 db_lmdb.cpp:35] Opened lmdb /home/taho/PycharmProjects/Deep_Learning/mobilenet_ssd_car_human_v3/data/KITTI/lmdb/KITTI_trainval_lmdb I0220 00:55:57.187115 30617 annotated_data_layer.cpp:62] output data size: 12,3,125,414 I0220 00:55:57.196645 30617 net.cpp:150] Setting up data I0220 00:55:57.196676 30617 net.cpp:157] Top shape: 12 3 125 414 (1863000) I0220 00:55:57.196681 30617 net.cpp:157] Top shape: 1 1 1 8 (8) I0220 00:55:57.196683 30617 net.cpp:165] Memory required for data: 7452032 I0220 00:55:57.196692 30617 layer_factory.hpp:77] Creating layer data_data_0_split I0220 00:55:57.196707 30617 net.cpp:100] Creating Layer data_data_0_split I0220 00:55:57.196712 30617 net.cpp:434] data_data_0_split <- data I0220 00:55:57.196723 30617 net.cpp:408] data_data_0_split -> data_data_0_split_0 I0220 00:55:57.196732 30617 net.cpp:408] data_data_0_split -> data_data_0_split_1 I0220 00:55:57.196738 30617 net.cpp:408] data_data_0_split -> data_data_0_split_2 I0220 00:55:57.196743 30617 net.cpp:408] data_data_0_split -> data_data_0_split_3 I0220 00:55:57.196746 30617 net.cpp:408] data_data_0_split -> data_data_0_split_4 I0220 00:55:57.196768 30617 net.cpp:408] data_data_0_split -> data_data_0_split_5 I0220 00:55:57.196774 30617 net.cpp:408] data_data_0_split -> data_data_0_split_6 I0220 00:55:57.196843 30617 net.cpp:150] Setting up data_data_0_split I0220 00:55:57.196849 30617 net.cpp:157] Top shape: 12 3 125 414 (1863000) I0220 00:55:57.196852 30617 net.cpp:157] Top shape: 12 3 125 414 (1863000) I0220 00:55:57.196856 30617 net.cpp:157] Top shape: 12 3 125 414 (1863000) I0220 00:55:57.196859 30617 net.cpp:157] Top shape: 12 3 125 414 (1863000) I0220 00:55:57.196862 30617 net.cpp:157] Top shape: 12 3 125 414 (1863000) I0220 00:55:57.196866 30617 net.cpp:157] Top shape: 12 3 125 414 (1863000) I0220 00:55:57.196868 30617 net.cpp:157] Top shape: 12 3 125 414 (1863000) I0220 00:55:57.196871 30617 net.cpp:165] Memory required for data: 59616032 I0220 00:55:57.196873 30617 layer_factory.hpp:77] Creating layer conv1_1 I0220 00:55:57.196887 30617 net.cpp:100] Creating Layer conv1_1 I0220 00:55:57.196890 30617 net.cpp:434] conv1_1 <- data_data_0_split_0 I0220 00:55:57.196894 30617 net.cpp:408] conv1_1 -> conv1_1 I0220 00:55:57.197880 30617 net.cpp:150] Setting up conv1_1 I0220 00:55:57.197891 30617 net.cpp:157] Top shape: 12 64 125 414 (39744000) I0220 00:55:57.197893 30617 net.cpp:165] Memory required for data: 218592032 I0220 00:55:57.197906 30617 layer_factory.hpp:77] Creating layer relu1_1 I0220 00:55:57.197911 30617 net.cpp:100] Creating Layer relu1_1 I0220 00:55:57.197914 30617 net.cpp:434] relu1_1 <- conv1_1 I0220 00:55:57.197918 30617 net.cpp:395] relu1_1 -> conv1_1 (in-place) I0220 00:55:57.197923 30617 net.cpp:150] Setting up relu1_1 I0220 00:55:57.197927 30617 net.cpp:157] Top shape: 12 64 125 414 (39744000) I0220 00:55:57.197929 30617 net.cpp:165] Memory required for data: 377568032 I0220 00:55:57.197932 30617 layer_factory.hpp:77] Creating layer conv1_2 I0220 00:55:57.197938 30617 net.cpp:100] Creating Layer conv1_2 I0220 00:55:57.197942 30617 net.cpp:434] conv1_2 <- conv1_1 I0220 00:55:57.197944 30617 net.cpp:408] conv1_2 -> conv1_2 I0220 00:55:57.198299 30617 net.cpp:150] Setting up conv1_2 I0220 00:55:57.198307 30617 net.cpp:157] Top shape: 12 64 125 414 (39744000) I0220 00:55:57.198308 30617 net.cpp:165] Memory required for data: 536544032 I0220 00:55:57.198313 30617 layer_factory.hpp:77] Creating layer relu1_2 I0220 00:55:57.198318 30617 net.cpp:100] Creating Layer relu1_2 I0220 00:55:57.198320 30617 net.cpp:434] relu1_2 <- conv1_2 I0220 00:55:57.198324 30617 net.cpp:395] relu1_2 -> conv1_2 (in-place) I0220 00:55:57.198328 30617 net.cpp:150] Setting up relu1_2 I0220 00:55:57.198331 30617 net.cpp:157] Top shape: 12 64 125 414 (39744000) I0220 00:55:57.198333 30617 net.cpp:165] Memory required for data: 695520032 I0220 00:55:57.198335 30617 layer_factory.hpp:77] Creating layer pool1 I0220 00:55:57.198340 30617 net.cpp:100] Creating Layer pool1 I0220 00:55:57.198343 30617 net.cpp:434] pool1 <- conv1_2 I0220 00:55:57.198345 30617 net.cpp:408] pool1 -> pool1 I0220 00:55:57.198374 30617 net.cpp:150] Setting up pool1 I0220 00:55:57.198379 30617 net.cpp:157] Top shape: 12 64 63 207 (10015488) I0220 00:55:57.198382 30617 net.cpp:165] Memory required for data: 735581984 I0220 00:55:57.198385 30617 layer_factory.hpp:77] Creating layer conv2_1 I0220 00:55:57.198390 30617 net.cpp:100] Creating Layer conv2_1 I0220 00:55:57.198392 30617 net.cpp:434] conv2_1 <- pool1 I0220 00:55:57.198395 30617 net.cpp:408] conv2_1 -> conv2_1 I0220 00:55:57.199576 30617 net.cpp:150] Setting up conv2_1 I0220 00:55:57.199587 30617 net.cpp:157] Top shape: 12 128 63 207 (20030976) I0220 00:55:57.199590 30617 net.cpp:165] Memory required for data: 815705888 I0220 00:55:57.199597 30617 layer_factory.hpp:77] Creating layer relu2_1 I0220 00:55:57.199604 30617 net.cpp:100] Creating Layer relu2_1 I0220 00:55:57.199606 30617 net.cpp:434] relu2_1 <- conv2_1 I0220 00:55:57.199609 30617 net.cpp:395] relu2_1 -> conv2_1 (in-place) I0220 00:55:57.199615 30617 net.cpp:150] Setting up relu2_1 I0220 00:55:57.199618 30617 net.cpp:157] Top shape: 12 128 63 207 (20030976) I0220 00:55:57.199632 30617 net.cpp:165] Memory required for data: 895829792 I0220 00:55:57.199635 30617 layer_factory.hpp:77] Creating layer conv2_2 I0220 00:55:57.199647 30617 net.cpp:100] Creating Layer conv2_2 I0220 00:55:57.199651 30617 net.cpp:434] conv2_2 <- conv2_1 I0220 00:55:57.199654 30617 net.cpp:408] conv2_2 -> conv2_2 I0220 00:55:57.200532 30617 net.cpp:150] Setting up conv2_2 I0220 00:55:57.200541 30617 net.cpp:157] Top shape: 12 128 63 207 (20030976) I0220 00:55:57.200543 30617 net.cpp:165] Memory required for data: 975953696 I0220 00:55:57.200548 30617 layer_factory.hpp:77] Creating layer relu2_2 I0220 00:55:57.200552 30617 net.cpp:100] Creating Layer relu2_2 I0220 00:55:57.200556 30617 net.cpp:434] relu2_2 <- conv2_2 I0220 00:55:57.200558 30617 net.cpp:395] relu2_2 -> conv2_2 (in-place) I0220 00:55:57.200562 30617 net.cpp:150] Setting up relu2_2 I0220 00:55:57.200565 30617 net.cpp:157] Top shape: 12 128 63 207 (20030976) I0220 00:55:57.200567 30617 net.cpp:165] Memory required for data: 1056077600 I0220 00:55:57.200570 30617 layer_factory.hpp:77] Creating layer pool2 I0220 00:55:57.200574 30617 net.cpp:100] Creating Layer pool2 I0220 00:55:57.200577 30617 net.cpp:434] pool2 <- conv2_2 I0220 00:55:57.200579 30617 net.cpp:408] pool2 -> pool2 I0220 00:55:57.200603 30617 net.cpp:150] Setting up pool2 I0220 00:55:57.200608 30617 net.cpp:157] Top shape: 12 128 32 104 (5111808) I0220 00:55:57.200609 30617 net.cpp:165] Memory required for data: 1076524832 I0220 00:55:57.200613 30617 layer_factory.hpp:77] Creating layer conv3_1 I0220 00:55:57.200618 30617 net.cpp:100] Creating Layer conv3_1 I0220 00:55:57.200619 30617 net.cpp:434] conv3_1 <- pool2 I0220 00:55:57.200623 30617 net.cpp:408] conv3_1 -> conv3_1 I0220 00:55:57.202555 30617 net.cpp:150] Setting up conv3_1 I0220 00:55:57.202571 30617 net.cpp:157] Top shape: 12 256 32 104 (10223616) I0220 00:55:57.202574 30617 net.cpp:165] Memory required for data: 1117419296 I0220 00:55:57.202582 30617 layer_factory.hpp:77] Creating layer relu3_1 I0220 00:55:57.202589 30617 net.cpp:100] Creating Layer relu3_1 I0220 00:55:57.202591 30617 net.cpp:434] relu3_1 <- conv3_1 I0220 00:55:57.202596 30617 net.cpp:395] relu3_1 -> conv3_1 (in-place) I0220 00:55:57.202601 30617 net.cpp:150] Setting up relu3_1 I0220 00:55:57.202605 30617 net.cpp:157] Top shape: 12 256 32 104 (10223616) I0220 00:55:57.202607 30617 net.cpp:165] Memory required for data: 1158313760 I0220 00:55:57.202610 30617 layer_factory.hpp:77] Creating layer conv3_2 I0220 00:55:57.202616 30617 net.cpp:100] Creating Layer conv3_2 I0220 00:55:57.202618 30617 net.cpp:434] conv3_2 <- conv3_1 I0220 00:55:57.202621 30617 net.cpp:408] conv3_2 -> conv3_2 F0220 00:55:57.204839 30631 sampler.cpp:89] Check failed: sampler.min_scale() > 0. (0 vs. 0) Check failure stack trace: @ 0x7f7f03fa65cd google::LogMessage::Fail() I0220 00:55:57.206298 30617 net.cpp:150] Setting up conv3_2 I0220 00:55:57.206313 30617 net.cpp:157] Top shape: 12 256 32 104 (10223616) I0220 00:55:57.206316 30617 net.cpp:165] Memory required for data: 1199208224 I0220 00:55:57.206323 30617 layer_factory.hpp:77] Creating layer relu3_2 I0220 00:55:57.206331 30617 net.cpp:100] Creating Layer relu3_2 I0220 00:55:57.206338 30617 net.cpp:434] relu3_2 <- conv3_2 I0220 00:55:57.206346 30617 net.cpp:395] relu3_2 -> conv3_2 (in-place) I0220 00:55:57.206357 30617 net.cpp:150] Setting up relu3_2 I0220 00:55:57.206362 30617 net.cpp:157] Top shape: 12 256 32 104 (10223616) I0220 00:55:57.206365 30617 net.cpp:165] Memory required for data: 1240102688 I0220 00:55:57.206369 30617 layer_factory.hpp:77] Creating layer conv3_3 I0220 00:55:57.206382 30617 net.cpp:100] Creating Layer conv3_3 I0220 00:55:57.206388 30617 net.cpp:434] conv3_3 <- conv3_2 I0220 00:55:57.206394 30617 net.cpp:408] conv3_3 -> conv3_3 @ 0x7f7f03fa8433 google::LogMessage::SendToLog() @ 0x7f7f03fa615b google::LogMessage::Flush() @ 0x7f7f03fa8e1e google::LogMessageFatal::~LogMessageFatal() @ 0x7f7f0468c6f1 caffe::SampleBBox() I0220 00:55:57.210090 30617 net.cpp:150] Setting up conv3_3 I0220 00:55:57.210106 30617 net.cpp:157] Top shape: 12 256 32 104 (10223616) I0220 00:55:57.210108 30617 net.cpp:165] Memory required for data: 1280997152 I0220 00:55:57.210117 30617 layer_factory.hpp:77] Creating layer relu3_3 I0220 00:55:57.210137 30617 net.cpp:100] Creating Layer relu3_3 I0220 00:55:57.210144 30617 net.cpp:434] relu3_3 <- conv3_3 I0220 00:55:57.210150 30617 net.cpp:395] relu3_3 -> conv3_3 (in-place) I0220 00:55:57.210160 30617 net.cpp:150] Setting up relu3_3 I0220 00:55:57.210165 30617 net.cpp:157] Top shape: 12 256 32 104 (10223616) I0220 00:55:57.210170 30617 net.cpp:165] Memory required for data: 1321891616 I0220 00:55:57.210175 30617 layer_factory.hpp:77] Creating layer pool3 I0220 00:55:57.210181 30617 net.cpp:100] Creating Layer pool3 I0220 00:55:57.210186 30617 net.cpp:434] pool3 <- conv3_3 I0220 00:55:57.210192 30617 net.cpp:408] pool3 -> pool3 I0220 00:55:57.210227 30617 net.cpp:150] Setting up pool3 I0220 00:55:57.210234 30617 net.cpp:157] Top shape: 12 256 16 52 (2555904) I0220 00:55:57.210237 30617 net.cpp:165] Memory required for data: 1332115232 I0220 00:55:57.210239 30617 layer_factory.hpp:77] Creating layer conv4_1 I0220 00:55:57.210247 30617 net.cpp:100] Creating Layer conv4_1 I0220 00:55:57.210254 30617 net.cpp:434] conv4_1 <- pool3 I0220 00:55:57.210260 30617 net.cpp:408] conv4_1 -> conv4_1 @ 0x7f7f0468ca90 caffe::GenerateSamples() @ 0x7f7f0468cccf caffe::GenerateBatchSamples() @ 0x7f7f048050d8 caffe::AnnotatedDataLayer<>::load_batch() @ 0x7f7f047ea55f caffe::BasePrefetchingDataLayer<>::InternalThreadEntry() @ 0x7f7f04890ae5 caffe::InternalThread::entry() @ 0x7f7ef484c5d5 (unknown) @ 0x7f7f02ce06ba start_thread @ 0x7f7f02ffd41d clone @ (nil) (unknown) Aborted (core dumped)

lucasjinreal commented 6 years ago

@Tahirhan After a long time, I got this error too......................

Ram-Godavarthi commented 4 years ago

Hi,

I got this error.

What could be the problem.

I0623 21:52:11.409765 4680 layer_factory.hpp:77] Creating layer data I0623 21:52:11.409915 4680 net.cpp:100] Creating Layer data I0623 21:52:11.409930 4680 net.cpp:408] data -> data I0623 21:52:11.409967 4680 net.cpp:408] data -> label I0623 21:52:11.410069 4682 db_lmdb.cpp:35] Opened lmdb /home/ram/data/KITTI/lmdb/KITTI_trainval_lmdb I0623 21:52:11.411000 4680 annotated_data_layer.cpp:62] output data size: 2,3,300,300 I0623 21:52:11.411047 4680 net.cpp:150] Setting up data I0623 21:52:11.411054 4680 net.cpp:157] Top shape: 2 3 300 300 (540000) I0623 21:52:11.411063 4680 net.cpp:157] Top shape: 1 1 9 8 (72) I0623 21:52:11.411070 4680 net.cpp:165] Memory required for data: 2160288 I0623 21:52:11.411078 4680 layer_factory.hpp:77] Creating layer data_data_0_split I0623 21:52:11.411087 4680 net.cpp:100] Creating Layer data_data_0_split I0623 21:52:11.411094 4680 net.cpp:434] data_data_0_split <- data I0623 21:52:11.411105 4680 net.cpp:408] data_data_0_split -> data_data_0_split_0 I0623 21:52:11.411115 4680 net.cpp:408] data_data_0_split -> data_data_0_split_1 I0623 21:52:11.411124 4680 net.cpp:408] data_data_0_split -> data_data_0_split_2 I0623 21:52:11.411134 4680 net.cpp:408] data_data_0_split -> data_data_0_split_3 I0623 21:52:11.411141 4680 net.cpp:408] data_data_0_split -> data_data_0_split_4 I0623 21:52:11.411149 4680 net.cpp:408] data_data_0_split -> data_data_0_split_5 I0623 21:52:11.411157 4680 net.cpp:408] data_data_0_split -> data_data_0_split_6 I0623 21:52:11.411170 4680 net.cpp:150] Setting up data_data_0_split I0623 21:52:11.411187 4680 net.cpp:157] Top shape: 2 3 300 300 (540000) I0623 21:52:11.411195 4680 net.cpp:157] Top shape: 2 3 300 300 (540000) I0623 21:52:11.411202 4680 net.cpp:157] Top shape: 2 3 300 300 (540000) I0623 21:52:11.411209 4680 net.cpp:157] Top shape: 2 3 300 300 (540000) I0623 21:52:11.411216 4680 net.cpp:157] Top shape: 2 3 300 300 (540000) I0623 21:52:11.411223 4680 net.cpp:157] Top shape: 2 3 300 300 (540000) I0623 21:52:11.411231 4680 net.cpp:157] Top shape: 2 3 300 300 (540000) I0623 21:52:11.411237 4680 net.cpp:165] Memory required for data: 17280288 I0623 21:52:11.411242 4680 layer_factory.hpp:77] Creating layer conv1_1 I0623 21:52:11.411255 4680 net.cpp:100] Creating Layer conv1_1 I0623 21:52:11.411262 4680 net.cpp:434] conv1_1 <- data_data_0_split_0 I0623 21:52:11.411268 4680 net.cpp:408] conv1_1 -> conv1_1 I0623 21:52:11.411443 4680 net.cpp:150] Setting up conv1_1 I0623 21:52:11.411449 4680 net.cpp:157] Top shape: 2 64 300 300 (11520000) I0623 21:52:11.411458 4680 net.cpp:165] Memory required for data: 63360288 I0623 21:52:11.411473 4680 layer_factory.hpp:77] Creating layer relu1_1 I0623 21:52:11.411480 4680 net.cpp:100] Creating Layer relu1_1 I0623 21:52:11.411485 4680 net.cpp:434] relu1_1 <- conv1_1 I0623 21:52:11.411492 4680 net.cpp:395] relu1_1 -> conv1_1 (in-place) I0623 21:52:11.411499 4680 net.cpp:150] Setting up relu1_1 I0623 21:52:11.411504 4680 net.cpp:157] Top shape: 2 64 300 300 (11520000) I0623 21:52:11.411511 4680 net.cpp:165] Memory required for data: 109440288 I0623 21:52:11.411516 4680 layer_factory.hpp:77] Creating layer conv1_2 I0623 21:52:11.411525 4680 net.cpp:100] Creating Layer conv1_2 I0623 21:52:11.411530 4680 net.cpp:434] conv1_2 <- conv1_1 I0623 21:52:11.411536 4680 net.cpp:408] conv1_2 -> conv1_2 I0623 21:52:11.411883 4680 net.cpp:150] Setting up conv1_2 I0623 21:52:11.411891 4680 net.cpp:157] Top shape: 2 64 300 300 (11520000) I0623 21:52:11.411898 4680 net.cpp:165] Memory required for data: 155520288 I0623 21:52:11.411908 4680 layer_factory.hpp:77] Creating layer relu1_2 I0623 21:52:11.411916 4680 net.cpp:100] Creating Layer relu1_2 I0623 21:52:11.411921 4680 net.cpp:434] relu1_2 <- conv1_2 I0623 21:52:11.411926 4680 net.cpp:395] relu1_2 -> conv1_2 (in-place) I0623 21:52:11.411933 4680 net.cpp:150] Setting up relu1_2 I0623 21:52:11.411938 4680 net.cpp:157] Top shape: 2 64 300 300 (11520000) I0623 21:52:11.411945 4680 net.cpp:165] Memory required for data: 201600288 I0623 21:52:11.411950 4680 layer_factory.hpp:77] Creating layer pool1 I0623 21:52:11.411957 4680 net.cpp:100] Creating Layer pool1 I0623 21:52:11.411962 4680 net.cpp:434] pool1 <- conv1_2 I0623 21:52:11.411968 4680 net.cpp:408] pool1 -> pool1 I0623 21:52:11.411984 4680 net.cpp:150] Setting up pool1 I0623 21:52:11.411989 4680 net.cpp:157] Top shape: 2 64 150 150 (2880000) I0623 21:52:11.411998 4680 net.cpp:165] Memory required for data: 213120288 I0623 21:52:11.412003 4680 layer_factory.hpp:77] Creating layer conv2_1 I0623 21:52:11.412010 4680 net.cpp:100] Creating Layer conv2_1 I0623 21:52:11.412015 4680 net.cpp:434] conv2_1 <- pool1 I0623 21:52:11.412024 4680 net.cpp:408] conv2_1 -> conv2_1 I0623 21:52:11.417014 4680 net.cpp:150] Setting up conv2_1 I0623 21:52:11.417093 4680 net.cpp:157] Top shape: 2 128 150 150 (5760000) I0623 21:52:11.417165 4680 net.cpp:165] Memory required for data: 236160288 I0623 21:52:11.417241 4680 layer_factory.hpp:77] Creating layer relu2_1 I0623 21:52:11.417313 4680 net.cpp:100] Creating Layer relu2_1 I0623 21:52:11.417382 4680 net.cpp:434] relu2_1 <- conv2_1 I0623 21:52:11.417470 4680 net.cpp:395] relu2_1 -> conv2_1 (in-place) I0623 21:52:11.417542 4680 net.cpp:150] Setting up relu2_1 I0623 21:52:11.417610 4680 net.cpp:157] Top shape: 2 128 150 150 (5760000) I0623 21:52:11.417680 4680 net.cpp:165] Memory required for data: 259200288 I0623 21:52:11.417758 4680 layer_factory.hpp:77] Creating layer conv2_2 I0623 21:52:11.417835 4680 net.cpp:100] Creating Layer conv2_2 I0623 21:52:11.417907 4680 net.cpp:434] conv2_2 <- conv2_1 I0623 21:52:11.428017 4680 net.cpp:408] conv2_2 -> conv2_2 I0623 21:52:11.428951 4680 net.cpp:150] Setting up conv2_2 I0623 21:52:11.433679 4680 net.cpp:157] Top shape: 2 128 150 150 (5760000) I0623 21:52:11.433717 4680 net.cpp:165] Memory required for data: 282240288 I0623 21:52:11.433742 4680 layer_factory.hpp:77] Creating layer relu2_2 I0623 21:52:11.433773 4680 net.cpp:100] Creating Layer relu2_2 I0623 21:52:11.437012 4680 net.cpp:434] relu2_2 <- conv2_2 I0623 21:52:11.437048 4680 net.cpp:395] relu2_2 -> conv2_2 (in-place) I0623 21:52:11.437072 4680 net.cpp:150] Setting up relu2_2 I0623 21:52:11.437088 4680 net.cpp:157] Top shape: 2 128 150 150 (5760000) I0623 21:52:11.437108 4680 net.cpp:165] Memory required for data: 305280288 I0623 21:52:11.437124 4680 layer_factory.hpp:77] Creating layer pool2 I0623 21:52:11.437144 4680 net.cpp:100] Creating Layer pool2 I0623 21:52:11.437160 4680 net.cpp:434] pool2 <- conv2_2 I0623 21:52:11.437177 4680 net.cpp:408] pool2 -> pool2 I0623 21:52:11.437198 4680 net.cpp:150] Setting up pool2 I0623 21:52:11.437216 4680 net.cpp:157] Top shape: 2 128 75 75 (1440000) I0623 21:52:11.437233 4680 net.cpp:165] Memory required for data: 311040288 I0623 21:52:11.437249 4680 layer_factory.hpp:77] Creating layer conv3_1 I0623 21:52:11.437278 4680 net.cpp:100] Creating Layer conv3_1 I0623 21:52:11.437294 4680 net.cpp:434] conv3_1 <- pool2 I0623 21:52:11.437312 4680 net.cpp:408] conv3_1 -> conv3_1 I0623 21:52:11.438838 4680 net.cpp:150] Setting up conv3_1 I0623 21:52:11.438995 4680 net.cpp:157] Top shape: 2 256 75 75 (2880000) I0623 21:52:11.439098 4680 net.cpp:165] Memory required for data: 322560288 I0623 21:52:11.439173 4680 layer_factory.hpp:77] Creating layer relu3_1 I0623 21:52:11.439231 4680 net.cpp:100] Creating Layer relu3_1 I0623 21:52:11.439286 4680 net.cpp:434] relu3_1 <- conv3_1 I0623 21:52:11.439342 4680 net.cpp:395] relu3_1 -> conv3_1 (in-place) I0623 21:52:11.439465 4680 net.cpp:150] Setting up relu3_1 I0623 21:52:11.439519 4680 net.cpp:157] Top shape: 2 256 75 75 (2880000) I0623 21:52:11.439579 4680 net.cpp:165] Memory required for data: 334080288 I0623 21:52:11.439635 4680 layer_factory.hpp:77] Creating layer conv3_2 I0623 21:52:11.439693 4680 net.cpp:100] Creating Layer conv3_2 I0623 21:52:11.439749 4680 net.cpp:434] conv3_2 <- conv3_1 I0623 21:52:11.439805 4680 net.cpp:408] conv3_2 -> conv3_2 I0623 21:52:11.442930 4680 net.cpp:150] Setting up conv3_2 I0623 21:52:11.451547 4680 net.cpp:157] Top shape: 2 256 75 75 (2880000) I0623 21:52:11.453277 4680 net.cpp:165] Memory required for data: 345600288 I0623 21:52:11.453344 4680 layer_factory.hpp:77] Creating layer relu3_2 I0623 21:52:11.453404 4680 net.cpp:100] Creating Layer relu3_2 I0623 21:52:11.453459 4680 net.cpp:434] relu3_2 <- conv3_2 I0623 21:52:11.453517 4680 net.cpp:395] relu3_2 -> conv3_2 (in-place) I0623 21:52:11.453593 4680 net.cpp:150] Setting up relu3_2 I0623 21:52:11.453647 4680 net.cpp:157] Top shape: 2 256 75 75 (2880000) I0623 21:52:11.453704 4680 net.cpp:165] Memory required for data: 357120288 I0623 21:52:11.453786 4680 layer_factory.hpp:77] Creating layer conv3_3 I0623 21:52:11.453850 4680 net.cpp:100] Creating Layer conv3_3 I0623 21:52:11.453905 4680 net.cpp:434] conv3_3 <- conv3_2 I0623 21:52:11.453966 4680 net.cpp:408] conv3_3 -> conv3_3 F0623 21:52:11.463140 4683 math_functions.cpp:250] Check failed: a <= b (0 vs. -1.19209e-07) Check failure stack trace: I0623 21:52:11.463245 4680 net.cpp:150] Setting up conv3_3 I0623 21:52:11.469085 4680 net.cpp:157] Top shape: 2 256 75 75 (2880000) I0623 21:52:11.469219 4680 net.cpp:165] Memory required for data: 368640288 I0623 21:52:11.469288 4680 layer_factory.hpp:77] Creating layer relu3_3 I0623 21:52:11.469363 4680 net.cpp:100] Creating Layer relu3_3 I0623 21:52:11.469419 4680 net.cpp:434] relu3_3 <- conv3_3 I0623 21:52:11.469477 4680 net.cpp:395] relu3_3 -> conv3_3 (in-place) I0623 21:52:11.472312 4680 net.cpp:150] Setting up relu3_3 I0623 21:52:11.472393 4680 net.cpp:157] Top shape: 2 256 75 75 (2880000) I0623 21:52:11.472455 4680 net.cpp:165] Memory required for data: 380160288 I0623 21:52:11.472510 4680 layer_factory.hpp:77] Creating layer pool3 I0623 21:52:11.472575 4680 net.cpp:100] Creating Layer pool3 I0623 21:52:11.472633 4680 net.cpp:434] pool3 <- conv3_3 I0623 21:52:11.472694 4680 net.cpp:408] pool3 -> pool3 I0623 21:52:11.472760 4680 net.cpp:150] Setting up pool3 I0623 21:52:11.472816 4680 net.cpp:157] Top shape: 2 256 38 38 (739328) I0623 21:52:11.472875 4680 net.cpp:165] Memory required for data: 383117600 I0623 21:52:11.472931 4680 layer_factory.hpp:77] Creating layer conv4_1 I0623 21:52:11.472998 4680 net.cpp:100] Creating Layer conv4_1 I0623 21:52:11.473170 4680 net.cpp:434] conv4_1 <- pool3 I0623 21:52:11.473261 4680 net.cpp:408] conv4_1 -> conv4_1 @ 0x7f56ec0670cd google::LogMessage::Fail() @ 0x7f56ec068f33 google::LogMessage::SendToLog() @ 0x7f56ec066c28 google::LogMessage::Flush() @ 0x7f56ec069999 google::LogMessageFatal::~LogMessageFatal() @ 0x7f56ec634667 caffe::caffe_rng_uniform<>() @ 0x7f56ec636508 caffe::SampleBBox() @ 0x7f56ec636860 caffe::GenerateSamples() @ 0x7f56ec636ab0 caffe::GenerateBatchSamples() I0623 21:52:11.504565 4680 net.cpp:150] Setting up conv4_1 I0623 21:52:11.504828 4680 net.cpp:157] Top shape: 2 512 38 38 (1478656) I0623 21:52:11.504954 4680 net.cpp:165] Memory required for data: 389032224 I0623 21:52:11.505061 4680 layer_factory.hpp:77] Creating layer relu4_1 I0623 21:52:11.505151 4680 net.cpp:100] Creating Layer relu4_1 I0623 21:52:11.505201 4680 net.cpp:434] relu4_1 <- conv4_1 I0623 21:52:11.505316 4680 net.cpp:395] relu4_1 -> conv4_1 (in-place) I0623 21:52:11.505375 4680 net.cpp:150] Setting up relu4_1 I0623 21:52:11.505427 4680 net.cpp:157] Top shape: 2 512 38 38 (1478656) I0623 21:52:11.505481 4680 net.cpp:165] Memory required for data: 394946848 I0623 21:52:11.505532 4680 layer_factory.hpp:77] Creating layer conv4_2 I0623 21:52:11.505591 4680 net.cpp:100] Creating Layer conv4_2 I0623 21:52:11.505643 4680 net.cpp:434] conv4_2 <- conv4_1 I0623 21:52:11.505705 4680 net.cpp:408] conv4_2 -> conv4_2 @ 0x7f56ec4e37b2 caffe::AnnotatedDataLayer<>::load_batch() @ 0x7f56ec4eb39a caffe::BasePrefetchingDataLayer<>::InternalThreadEntry() @ 0x7f56e59acbcd (unknown) @ 0x7f56e313f6db start_thread @ 0x7f56ea67d88f clone Aborted (core dumped)