tensorflow / ngraph-bridge

TensorFlow-nGraph bridge
Other
137 stars 64 forks source link

With OV-iGPU-plugin, ~Backend() destructor crashes #724

Closed bani-intelaipg closed 3 years ago

bani-intelaipg commented 3 years ago

(venv-tf-py3) banikuma@aipg-ra-igpu-08:/localdisk/bani/MODELS/tensorflow_openvino_models_public/demo$ gdb --args python -u infer_frozen.py -m ../frozen/resnet_50.pb -i input:0 -o predict:0 --image ./images/bike.jpg --num-iterations 1 --inf-time --device GPU --ngtf-enabled GNU gdb (Ubuntu 8.1.1-0ubuntu1) 8.1.1 Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from python...(no debugging symbols found)...done. (gdb) r Starting program: /localdisk/bani/NGTF_OPVIE/build_cmake/venv-tf-py3/bin/python -u infer_frozen.py -m ../frozen/resnet_50.pb -i input:0 -o predict:0 --image ./images/bike.jpg --num-iterations 1 --inf-time --device GPU --ngtf-enabled [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". [New Thread 0x7ffff394e700 (LWP 21822)] [New Thread 0x7ffff314d700 (LWP 21823)] [New Thread 0x7ffff094c700 (LWP 21824)] [New Thread 0x7fffec14b700 (LWP 21825)] [New Thread 0x7fffeb94a700 (LWP 21826)] [New Thread 0x7fffe7149700 (LWP 21827)] [New Thread 0x7fffe4948700 (LWP 21828)] [New Thread 0x7fffe4147700 (LWP 21829)] [New Thread 0x7fffdf946700 (LWP 21830)] [New Thread 0x7fffdd145700 (LWP 21831)] [New Thread 0x7fffda944700 (LWP 21832)] [New Thread 0x7fffd8143700 (LWP 21833)] [New Thread 0x7fffd5942700 (LWP 21834)] [New Thread 0x7fffd3141700 (LWP 21835)] [New Thread 0x7fffd0940700 (LWP 21836)] [Thread 0x7fffdd145700 (LWP 21831) exited] [Thread 0x7fffd0940700 (LWP 21836) exited] [Thread 0x7fffd3141700 (LWP 21835) exited] [Thread 0x7fffd5942700 (LWP 21834) exited] [Thread 0x7fffd8143700 (LWP 21833) exited] [Thread 0x7fffda944700 (LWP 21832) exited] [Thread 0x7fffdf946700 (LWP 21830) exited] [Thread 0x7fffe4147700 (LWP 21829) exited] [Thread 0x7fffe4948700 (LWP 21828) exited] [Thread 0x7fffe7149700 (LWP 21827) exited] [Thread 0x7fffeb94a700 (LWP 21826) exited] [Thread 0x7fffec14b700 (LWP 21825) exited] [Thread 0x7ffff094c700 (LWP 21824) exited] [Thread 0x7ffff314d700 (LWP 21823) exited] [Thread 0x7ffff394e700 (LWP 21822) exited] Input model: ../frozen/resnet_50.pb Input name: input:0 Output name: predict:0 Input image: ./images/bike.jpg Number of Iterations: 1 Measue Inf Time: True Device: GPU NGraph-Bridge Enabled [New Thread 0x7fffd0940700 (LWP 22366)] Device: CPU Device: GPU CREATE MODEL - BEGIN [New Thread 0x7fffd3141700 (LWP 22368)] CREATE MODEL - END PREDICTION - BEGIN [New Thread 0x7fffd5942700 (LWP 22369)] [New Thread 0x7fffd8143700 (LWP 22370)] [New Thread 0x7fff8cabe700 (LWP 22371)] [New Thread 0x7fff8c2bd700 (LWP 22372)] [New Thread 0x7fff8babc700 (LWP 22373)] [New Thread 0x7fff8b2bb700 (LWP 22374)] [New Thread 0x7fff8a440700 (LWP 22375)] [New Thread 0x7fff88f3c700 (LWP 22376)] [New Thread 0x7fff83fff700 (LWP 22377)] [New Thread 0x7fff837fe700 (LWP 22378)] [New Thread 0x7fff82ffd700 (LWP 22379)] [New Thread 0x7fff827fc700 (LWP 22380)] [New Thread 0x7fff81ffb700 (LWP 22381)] [New Thread 0x7fff817fa700 (LWP 22382)] [New Thread 0x7fff80ff9700 (LWP 22383)] [New Thread 0x7fff7bfff700 (LWP 22384)] 2020-12-03 11:37:43.661534: I tensorflow/core/common_runtime/process_util.cc:147] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance. [New Thread 0x7fff7b7fe700 (LWP 22385)] [New Thread 0x7fff7affd700 (LWP 22386)] [New Thread 0x7fff79d5c700 (LWP 22387)] [New Thread 0x7fff7955b700 (LWP 22388)] Inf Execution Time [ 0 ]: 15.472449541091919 PREDICTION - END Predictions for image 0: mountain bike, all-terrain bike, off-roader ( 0.27964649 ) bicycle-built-for-two, tandem bicycle, tandem ( 0.24313934 ) unicycle, monocycle ( 0.10700702 ) terminate called without an active exception

Thread 1 "python" received signal SIGABRT, Aborted. __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51 51 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory. (gdb) bt

0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51

1 0x00007ffff7a22921 in __GI_abort () at abort.c:79

2 0x00007fffbe37a957 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6

3 0x00007fffbe380ab6 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6

4 0x00007fffbe380af1 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6

5 0x00007fff79fa944a in ?? () from /localdisk/bani/NGTF_OPVIE/build_cmake/venv-tf-py3/lib/python3.6/site-packages/ngraph_bridge/libclDNNPlugin.so

6 0x00007fff79f10ffa in ?? () from /localdisk/bani/NGTF_OPVIE/build_cmake/venv-tf-py3/lib/python3.6/site-packages/ngraph_bridge/libclDNNPlugin.so

7 0x00007fff79fa8458 in ?? () from /localdisk/bani/NGTF_OPVIE/build_cmake/venv-tf-py3/lib/python3.6/site-packages/ngraph_bridge/libclDNNPlugin.so

8 0x00007fff79fa8477 in ?? () from /localdisk/bani/NGTF_OPVIE/build_cmake/venv-tf-py3/lib/python3.6/site-packages/ngraph_bridge/libclDNNPlugin.so

9 0x00007fff79e7ab09 in ?? () from /localdisk/bani/NGTF_OPVIE/build_cmake/venv-tf-py3/lib/python3.6/site-packages/ngraph_bridge/libclDNNPlugin.so

10 0x00007fff79e0ab06 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() ()

from /localdisk/bani/NGTF_OPVIE/build_cmake/venv-tf-py3/lib/python3.6/site-packages/ngraph_bridge/libclDNNPlugin.so

11 0x00007fff79e0b43a in ?? () from /localdisk/bani/NGTF_OPVIE/build_cmake/venv-tf-py3/lib/python3.6/site-packages/ngraph_bridge/libclDNNPlugin.so

12 0x00007fff79e0ab06 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() ()

from /localdisk/bani/NGTF_OPVIE/build_cmake/venv-tf-py3/lib/python3.6/site-packages/ngraph_bridge/libclDNNPlugin.so

13 0x00007fff79e4cca0 in ?? () from /localdisk/bani/NGTF_OPVIE/build_cmake/venv-tf-py3/lib/python3.6/site-packages/ngraph_bridge/libclDNNPlugin.so

14 0x00007fff8e3c4cf9 in std::_Sp_counted_ptr_inplace<tensorflow::ngraph_bridge::Executable, std::allocator, (__gnu_cxx::_Lock_policy)2>::_M_dispose() ()

from /localdisk/bani/NGTF_OPVIE/build_cmake/venv-tf-py3/lib/python3.6/site-packages/ngraph_bridge/libngraph_bridge.so

15 0x00007fff8e3c6ca2 in std::_Hashtable<std::shared_ptr, std::pair<std::shared_ptr const, std::shared_ptr >, std::allocator<std::pair<std::shared_ptr const, std::shared_ptr > >, std::detail::_Select1st, std::equal_to<std::shared_ptr >, std::hash<std::shared_ptr >, std::detail::_Mod_range_hashing, std::detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::detail::_Hashtable_traits<false, false, true> >::clear() ()

from /localdisk/bani/NGTF_OPVIE/build_cmake/venv-tf-py3/lib/python3.6/site-packages/ngraph_bridge/libngraph_bridge.so

16 0x00007fff8e3c36c9 in tensorflow::ngraph_bridge::Backend::~Backend() ()

from /localdisk/bani/NGTF_OPVIE/build_cmake/venv-tf-py3/lib/python3.6/site-packages/ngraph_bridge/libngraph_bridge.so

17 0x00007fff8e4458b1 in std::shared_ptr::~shared_ptr() ()

from /localdisk/bani/NGTF_OPVIE/build_cmake/venv-tf-py3/lib/python3.6/site-packages/ngraph_bridge/libngraph_bridge.so

18 0x00007ffff7a25161 in run_exit_handlers (status=0, listp=0x7ffff7dcd718 <exit_funcs>, run_list_atexit=run_list_atexit@entry=true,

run_dtors=run_dtors@entry=true) at exit.c:108

19 0x00007ffff7a2525a in __GI_exit (status=) at exit.c:139

20 0x00007ffff7a03bfe in __libc_start_main (main=0x4b0d00
, argc=17, argv=0x7fffffffe4f8, init=, fini=,

rtld_fini=<optimized out>, stack_end=0x7fffffffe4e8) at ../csu/libc-start.c:344

21 0x00000000005b1d5a in _start ()

(gdb) q A debugging session is active.

Inferior 1 [process 21789] will be killed.

Quit anyway? (y or n) y (venv-tf-py3) banikuma@aipg-ra-igpu-08:/localdisk/bani/MODELS/tensorflow_openvino_models_public/demo$

adk9 commented 3 years ago

Duplicate of #675.