search

chuanqi305 / MobileNet-SSD

Caffe implementation of Google MobileNet SSD detection network, with pretrained weights on VOC0712 and mAP=0.727.
MIT License
2k stars 1.18k forks source link

caffe::BasePrefetchingDataLayer<>::InternalThreadEntry() @ 0x7fc72d160185 caffe::InternalThread::entry() @ 0x7fc722caf5d5 (unknown) #164

Closed AbhimanyuAryan closed 4 years ago

AbhimanyuAryan commented 4 years ago

I'm training my ssd-mobilenetv1 with caffe from here: https://github.com/chuanqi305/MobileNet-SSD

while running ./train.sh I get the below errors. A lot of them or opencv errors

....

I0724 17:04:40.733958  8375 layer_factory.hpp:77] Creating layer data
I0724 17:04:40.746536  8375 net.cpp:100] Creating Layer data
I0724 17:04:40.746582  8375 net.cpp:408] data -> data
I0724 17:04:40.746664  8375 net.cpp:408] data -> label
I0724 17:04:40.747843  8384 db_lmdb.cpp:35] Opened lmdb trainval_lmdb/
I0724 17:04:41.177836  8375 annotated_data_layer.cpp:62] output data size: 24,1,300,300
I0724 17:04:41.198844  8375 net.cpp:150] Setting up data
I0724 17:04:41.198904  8375 net.cpp:157] Top shape: 24 1 300 300 (2160000)
I0724 17:04:41.198917  8375 net.cpp:157] Top shape: 1 1 6 8 (48)
I0724 17:04:41.198926  8375 net.cpp:165] Memory required for data: 8640192
I0724 17:04:41.198951  8375 layer_factory.hpp:77] Creating layer data_data_0_split
I0724 17:04:41.198982  8375 net.cpp:100] Creating Layer data_data_0_split
I0724 17:04:41.198998  8375 net.cpp:434] data_data_0_split <- data
I0724 17:04:41.199023  8375 net.cpp:408] data_data_0_split -> data_data_0_split_0
I0724 17:04:41.199048  8375 net.cpp:408] data_data_0_split -> data_data_0_split_1
I0724 17:04:41.199065  8375 net.cpp:408] data_data_0_split -> data_data_0_split_2
I0724 17:04:41.199085  8375 net.cpp:408] data_data_0_split -> data_data_0_split_3
I0724 17:04:41.199102  8375 net.cpp:408] data_data_0_split -> data_data_0_split_4
I0724 17:04:41.199120  8375 net.cpp:408] data_data_0_split -> data_data_0_split_5
I0724 17:04:41.199136  8375 net.cpp:408] data_data_0_split -> data_data_0_split_6
I0724 17:04:41.199270  8375 net.cpp:150] Setting up data_data_0_split
I0724 17:04:41.199288  8375 net.cpp:157] Top shape: 24 1 300 300 (2160000)
I0724 17:04:41.199301  8375 net.cpp:157] Top shape: 24 1 300 300 (2160000)
I0724 17:04:41.199350  8375 net.cpp:157] Top shape: 24 1 300 300 (2160000)
I0724 17:04:41.199363  8375 net.cpp:157] Top shape: 24 1 300 300 (2160000)
I0724 17:04:41.199375  8375 net.cpp:157] Top shape: 24 1 300 300 (2160000)
I0724 17:04:41.199386  8375 net.cpp:157] Top shape: 24 1 300 300 (2160000)
I0724 17:04:41.199398  8375 net.cpp:157] Top shape: 24 1 300 300 (2160000)
I0724 17:04:41.199406  8375 net.cpp:165] Memory required for data: 69120192
I0724 17:04:41.199417  8375 layer_factory.hpp:77] Creating layer conv0
I0724 17:04:41.199450  8375 net.cpp:100] Creating Layer conv0
I0724 17:04:41.199461  8375 net.cpp:434] conv0 <- data_data_0_split_0
I0724 17:04:41.199477  8375 net.cpp:408] conv0 -> conv0
I0724 17:04:41.199724  8375 net.cpp:150] Setting up conv0
I0724 17:04:41.199743  8375 net.cpp:157] Top shape: 24 32 150 150 (17280000)
I0724 17:04:41.199753  8375 net.cpp:165] Memory required for data: 138240192
I0724 17:04:41.199777  8375 layer_factory.hpp:77] Creating layer conv0/bn
I0724 17:04:41.199795  8375 net.cpp:100] Creating Layer conv0/bn
I0724 17:04:41.199805  8375 net.cpp:434] conv0/bn <- conv0
I0724 17:04:41.199820  8375 net.cpp:395] conv0/bn -> conv0 (in-place)
I0724 17:04:41.200137  8375 net.cpp:150] Setting up conv0/bn
I0724 17:04:41.200165  8375 net.cpp:157] Top shape: 24 32 150 150 (17280000)
I0724 17:04:41.200179  8375 net.cpp:165] Memory required for data: 207360192
I0724 17:04:41.200208  8375 layer_factory.hpp:77] Creating layer conv0/scale
I0724 17:04:41.200237  8375 net.cpp:100] Creating Layer conv0/scale
I0724 17:04:41.200254  8375 net.cpp:434] conv0/scale <- conv0
I0724 17:04:41.200271  8375 net.cpp:395] conv0/scale -> conv0 (in-place)
I0724 17:04:41.200340  8375 layer_factory.hpp:77] Creating layer conv0/scale
I0724 17:04:41.200556  8375 net.cpp:150] Setting up conv0/scale
I0724 17:04:41.200577  8375 net.cpp:157] Top shape: 24 32 150 150 (17280000)
I0724 17:04:41.200587  8375 net.cpp:165] Memory required for data: 276480192
I0724 17:04:41.200605  8375 layer_factory.hpp:77] Creating layer conv0/relu
I0724 17:04:41.200621  8375 net.cpp:100] Creating Layer conv0/relu
I0724 17:04:41.200632  8375 net.cpp:434] conv0/relu <- conv0
...
lot of logs
...
I0724 17:04:41.220937  8375 net.cpp:408] conv7 -> conv7
OpenCV Error: Assertion failed ((scn == 3 || scn == 4) && (depth == CV_8U || depth == CV_32F)) in cvtColor, file /build/opencv-ys8xiq/opencv-2.4.9.1+dfsg/modules/imgproc/src/color.cpp, line 3959
I0724 17:04:41.225272  8375 net.cpp:150] Setting up conv7
I0724 17:04:41.225296  8375 net.cpp:157] Top shape: 24 512 19 19 (4435968)
I0724 17:04:41.225306  8375 net.cpp:165] Memory required for data: 2834731200
....
lot of logs
....
I0724 17:04:41.247714  8375 net.cpp:150] Setting up conv11/dw/relu
I0724 17:04:41.247725  8375 net.cpp:157] Top shape: 24 512 19 19 (4435968)
I0724 17:04:41.247735  8375 net.cpp:165] Memory required for data: 3384791232
I0724 17:04:41.247745  8375 layer_factory.hpp:77] Creating layer conv11
I0724 17:04:41.247764  8375 net.cpp:100] Creating Layer conv11
I0724 17:04:41.247776  8375 net.cpp:434] conv11 <- conv11/dw
I0724 17:04:41.247792  8375 net.cpp:408] conv11 -> conv11
  what():  /build/opencv-ys8xiq/opencv-2.4.9.1+dfsg/modules/imgproc/src/color.cpp:3959: error: (-215) (scn == 3 || scn == 4) && (depth == CV_8U || depth == CV_32F) in function cvtColor

*** Aborted at 1563968081 (unix time) try "date -d @1563968081" if you are using GNU date ***
PC: @     0x7fc72ba07428 gsignal
*** SIGABRT (@0x3e8000020b7) received by PID 8375 (TID 0x7fc6f1fff700) from PID 8375; stack trace: ***
I0724 17:04:41.252117  8375 net.cpp:150] Setting up conv11
......
I0724 17:04:41.252916  8375 net.cpp:408] conv11_conv11/relu_0_split -> conv11_conv11/relu_0_split_3
    @     0x7fc72ba074b0 (unknown)
I0724 17:04:41.252998  8375 net.cpp:150] Setting up conv11_conv11/relu_0_split
I0724 17:04:41.253015  8375 net.cpp:157] Top shape: 24 512 19 19 (4435968)
I0724 17:04:41.253023  8375 net.cpp:157] Top shape: 24 512 19 19 (4435968)
I0724 17:04:41.253033  8375 net.cpp:157] Top shape: 24 512 19 19 (4435968)
I0724 17:04:41.253059  8375 net.cpp:157] Top shape: 24 512 19 19 (4435968)
I0724 17:04:41.253068  8375 net.cpp:165] Memory required for data: 3526742208
......
I0724 17:04:41.254135  8375 layer_factory.hpp:77] Creating layer conv12
I0724 17:04:41.254153  8375 net.cpp:100] Creating Layer conv12
I0724 17:04:41.254163  8375 net.cpp:434] conv12 <- conv12/dw
I0724 17:04:41.254179  8375 net.cpp:408] conv12 -> conv12
    @     0x7fc72ba07428 gsignal
    @     0x7fc72ba0902a abort
    @     0x7fc72c04184d __gnu_cxx::__verbose_terminate_handler()
    @     0x7fc72c03f6b6 (unknown)
    @     0x7fc72c03f701 std::terminate()
    @     0x7fc72c03f919 __cxa_throw
    @     0x7fc723611896 cv::error()
I0724 17:04:41.263516  8375 net.cpp:150] Setting up conv12
I0724 17:04:41.263537  8375 net.cpp:157] Top shape: 24 1024 10 10 (2457600)
I0724 17:04:41.263550  8375 net.cpp:165] Memory required for data: 3556233408
.....
I0724 17:04:41.284293  8375 net.cpp:165] Memory required for data: 3703689408
I0724 17:04:41.284322  8375 layer_factory.hpp:77] Creating layer conv14_1
I0724 17:04:41.284341  8375 net.cpp:100] Creating Layer conv14_1
I0724 17:04:41.284353  8375 net.cpp:434] conv14_1 <- conv13_conv13/relu_0_split_0
I0724 17:04:41.284368  8375 net.cpp:408] conv14_1 -> conv14_1
    @     0x7fc72d166acb caffe::RandomHue()
I0724 17:04:41.289675  8375 net.cpp:150] Setting up conv14_1
I0724 17:04:41.289695  8375 net.cpp:157] Top shape: 24 256 10 10 (614400)
I0724 17:04:41.289705  8375 net.cpp:165] Memory required for data: 3706147008
I0724 17:04:41.289718  8375 layer_factory.hpp:77] Creating layer conv14_1/bn
I0724 17:04:41.289738  8375 net.cpp:100] Creating Layer conv14_1/bn
I0724 17:04:41.289750  8375 net.cpp:434] conv14_1/bn <- conv14_1
I0724 17:04:41.289762  8375 net.cpp:395] conv14_1/bn -> conv14_1 (in-place)
I0724 17:04:41.289995  8375 net.cpp:150] Setting up conv14_1/bn
I0724 17:04:41.290014  8375 net.cpp:157] Top shape: 24 256 10 10 (614400)
I0724 17:04:41.290024  8375 net.cpp:165] Memory required for data: 3708604608
I0724 17:04:41.290040  8375 layer_factory.hpp:77] Creating layer conv14_1/scale
I0724 17:04:41.290055  8375 net.cpp:100] Creating Layer conv14_1/scale
I0724 17:04:41.290066  8375 net.cpp:434] conv14_1/scale <- conv14_1
I0724 17:04:41.290078  8375 net.cpp:395] conv14_1/scale -> conv14_1 (in-place)
I0724 17:04:41.290138  8375 layer_factory.hpp:77] Creating layer conv14_1/scale
I0724 17:04:41.290282  8375 net.cpp:150] Setting up conv14_1/scale
I0724 17:04:41.290300  8375 net.cpp:157] Top shape: 24 256 10 10 (614400)
I0724 17:04:41.290309  8375 net.cpp:165] Memory required for data: 3711062208
I0724 17:04:41.290334  8375 layer_factory.hpp:77] Creating layer conv14_1/relu
I0724 17:04:41.290349  8375 net.cpp:100] Creating Layer conv14_1/relu
I0724 17:04:41.290359  8375 net.cpp:434] conv14_1/relu <- conv14_1
I0724 17:04:41.290371  8375 net.cpp:395] conv14_1/relu -> conv14_1 (in-place)
I0724 17:04:41.290390  8375 net.cpp:150] Setting up conv14_1/relu
I0724 17:04:41.290401  8375 net.cpp:157] Top shape: 24 256 10 10 (614400)
I0724 17:04:41.290410  8375 net.cpp:165] Memory required for data: 3713519808
I0724 17:04:41.290419  8375 layer_factory.hpp:77] Creating layer conv14_2
I0724 17:04:41.290436  8375 net.cpp:100] Creating Layer conv14_2
I0724 17:04:41.290446  8375 net.cpp:434] conv14_2 <- conv14_1
I0724 17:04:41.290462  8375 net.cpp:408] conv14_2 -> conv14_2
    @     0x7fc72d167694 caffe::ApplyDistort()
    @     0x7fc72d1d2072 caffe::DataTransformer<>::DistortImage()
    @     0x7fc72d21b426 caffe::AnnotatedDataLayer<>::load_batch()
    @     0x7fc72d2c2fcf caffe::BasePrefetchingDataLayer<>::InternalThreadEntry()
    @     0x7fc72d160185 caffe::InternalThread::entry()
    @     0x7fc722caf5d5 (unknown)
    @     0x7fc72255d6ba start_thread
    @     0x7fc72bad941d clone
    @                0x0 (unknown)
Aborted (core dumped)
AbhimanyuAryan commented 4 years ago

Fixed it...