search

GPUOpen-ProfessionalCompute-Libraries / amdovx-modules

AMD OpenVX modules: such as, neural network inference, 360 video stitching, etc.
110 stars 50 forks source link

caffe2openvx : Segmentation fault (core dumped) #226

Closed srohit0 closed 5 years ago

srohit0 commented 5 years ago

caffe2openvx gives Segmentation fault (core dumped) on VGG16 RCNN Model.

Caffe Model

Faster RCNN Caffe model was downloaded from BVLC/Caffe Model Zoo using a download script 

[faster_rcnn_models]$ fetch_faster_rcnn_models.sh 
Downloading Faster R-CNN demo models (695M)...
--2018-12-15 12:58:44--  https://dl.dropboxusercontent.com/s/o6ii098bu51d139/faster_rcnn_models.tgz?dl=0
Resolving dl.dropboxusercontent.com (dl.dropboxusercontent.com)... 162.125.7.6, 2620:100:601a:6::a27d:706
Connecting to dl.dropboxusercontent.com (dl.dropboxusercontent.com)|162.125.7.6|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 728560422 (695M) [application/x-gtar]
Saving to: faster_rcnn_models.tgz

faster_rcnn_models.tgz            100%[============================================================>] 694.81M  7.74MB/s    in 89s     

2018-12-15 13:00:14 (7.77 MB/s) - faster_rcnn_models.tgz saved [728560422/728560422]

Unzipping...
faster_rcnn_models/
faster_rcnn_models/ZF_faster_rcnn_final.caffemodel
faster_rcnn_models/VGG16_faster_rcnn_final.caffemodel
Done. Please run this command again to verify that checksum = ac116844f66aefe29587214272054668.
[faster_rcnn_models]$ fetch_faster_rcnn_models.sh
File already exists. Checking md5...
Checksum is correct. No need to download.
[faster_rcnn_models]$ tar xvfz faster_rcnn_models.tgz 
faster_rcnn_models/
faster_rcnn_models/ZF_faster_rcnn_final.caffemodel
faster_rcnn_models/VGG16_faster_rcnn_final.caffemodel

Command used

[openvx]$ caffe2openvx \ 
               /home/amd/models/caffe/faster_rcnn_models/VGG16_faster_rcnn_final.caffemodel \
               --output-dir VGG16_faster_rcnn_final

Output

Reading the binary file from : /home/amd/models/caffe/faster_rcnn_models/VGG16_faster_rcnn_final.caffemodel CaffeModel Read Successful Fetching the weights for : VGG_ILSVRC_16_layers Segmentation fault (core dumped)

Debugger Stack Trace

[openvx]$ gdb /opt/rocm/bin/caffe2openvx
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
...
...
Reading symbols from /opt/rocm/bin/caffe2openvx...(no debugging symbols found)...done.
(gdb) run /home/amd/models/caffe/faster_rcnn_models/VGG16_faster_rcnn_final.caffemodel --output-dir VGG16_faster_rcnn_final
Starting program: /opt/rocm/bin/caffe2openvx /home/amd/models/caffe/faster_rcnn_models/VGG16_faster_rcnn_final.caffemodel --output-dir VGG16_faster_rcnn_final
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Reading the binary file from : /home/amd/models/caffe/faster_rcnn_models/VGG16_faster_rcnn_final.caffemodel
CaffeModel Read Successful
Fetching the weights for : VGG_ILSVRC_16_layers
bt
Program received signal SIGSEGV, Segmentation fault.
__memcpy_avx_unaligned () at ../sysdeps/x86_64/multiarch/memcpy-avx-unaligned.S:51
51      ../sysdeps/x86_64/multiarch/memcpy-avx-unaligned.S: No such file or directory.
(gdb) bt
#0  __memcpy_avx_unaligned () at ../sysdeps/x86_64/multiarch/memcpy-avx-unaligned.S:51
#1  0x00000000004ba878 in void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char*>(char*, char*, std::forward_iterator_tag) ()
#2  0x00007ffff74d19cf in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3  0x00000000004be256 in std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::vector<int, std::allocator<int> > >::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, 0ul>(std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>&, std::tuple<>&, std::_Index_tuple<0ul>, std::_Index_tuple<>) ()
#4  0x00000000004bdd48 in std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::vector<int, std::allocator<int> > >::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>(std::piecewise_construct_t, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>, std::tuple<>) ()
#5  0x00000000004bd4db in void __gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::vector<int, std::allocator<int> > > > >::construct<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::vector<int, std::allocator<int> > >, std::piecewise_construct_t const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>, std::tuple<> >(std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::vector<int, std::allocator<int> > >*, std::piecewise_construct_t const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>&&, std::tuple<>&&) ()
#6  0x00000000004bc902 in void std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::vector<int, std::allocator<int> > > > > >::construct<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::vector<int, std::allocator<int> > >, std::piecewise_construct_t const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>, std::tuple<> >(std::allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::vector<int, std::allocator<int> > > > >&, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::vector<int, std::allocator<int> > >*, std::piecewise_construct_t const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>&&, std::tuple<>&&) ()
#7  0x00000000004bb357 in void std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::vector<int, std::allocator<int> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::vector<int, std::allocator<int> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::vector<int, std::allocator<int> > > > >::_M_construct_node<std::piecewise_construct_t const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>, std::tuple<> >(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::vector<int, std::allocator<int> > > >*, std::piecewise_construct_t const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>&&, std::tuple<>&&) ()
#8  0x00000000004b8fb4 in std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::vector<int, std::allocator<int> > > >* std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::vector<int, std::allocator<int> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::vector<int, std::allocator<int> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::vector<int, std::allocator<int> > > > >::_M_create_node<std::piecewise_construct_t const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>, std::tuple<> >(std::piecewise_construct_t const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>&&, std::tuple<>&&) ()
#9  0x00000000004b6833 in std::_Rb_tree_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::vector<int, std::allocator<int> > > > std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::vector<int, std::allocator<int> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::vector<int, std::allocator<int> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::vector<int, std::allocator<int> > > > >::_M_emplace_hint_unique<std::piecewise_construct_t const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>, std::tuple<> >(std::_Rb_tree_const_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::vector<int, std::allocator<int> > > >, std::piecewise_construct_t const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>&&, std::tuple<>&&) ()
#10 0x00000000004b4199 in std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::vector<int, std::allocator<int> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::vector<int, std::allocator<int> > > > >::operator[](std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()
#11 0x0000000000494309 in calculateTensorDim(std::vector<std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::allocator<std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >&, int*, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::vector<int, std::allocator<int> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::vector<int, std::allocator<int> > > > >&) ()
#12 0x00000000004b1290 in main ()
(gdb)
kiritigowda commented 5 years ago

R-CNN model is not supported by MIVisionX yet.

kiritigowda commented 5 years ago

Closing this issue as its the same as issue 6 in MIVisionX.

srohit0 commented 5 years ago

You mean the message "caffe2nnir.py ERROR: caffe operation Python is not supported yet" from script is causing segmentation fault?

Does the executable caffe2openvx use this script internally?

Stack trace shows seg-fault in STD::map operator, not in script.

So, it looks different at MIVision's issue#6 from usage, source and symptom point of view.

kiritigowda commented 5 years ago

You are trying to run VGG16 R-CNN Model, which is not supported yet by neural network extensions. If the underneath layers are not implemented, it does not matter if you call caffe2nnir or caffe2openvx, they would fail. The solution for both these issues is to extend the support to the missing layers.