Open wanghaoshuang opened 6 years ago
一个解决方案是:编译的时候关闭mkldnn相关的选项,因为mkldnn相关的库也会占用大量static TLS
建议将mkldnn放在platform/dynload下。
Duplicate of #3007
可以用readelf –l XXX.so 查看一下1.5 release so里program header的TLS:
./third_party/install/mkldnn/lib/libmkldnn.so
Program Headers:
Type Offset VirtAddr PhysAddr
FileSiz MemSiz Flags Align
TLS 0x0000000000c38928 0x0000000000e38928 0x0000000000e38928
0x0000000000000000 0x0000000000000020 R 8
./third_party/install/mklml/lib/libmklml_intel.so
Program Headers:
Type Offset VirtAddr PhysAddr
FileSiz MemSiz Flags Align
TLS 0x0000000007a043e8 0x0000000007c043e8 0x0000000007c043e8
0x0000000000000000 0x0000000000000020 R 8
./paddle/fluid/inference/libpaddle_fluid.so
Program Headers:
Type Offset VirtAddr PhysAddr
FileSiz MemSiz Flags Align
TLS 0x0000000001e89020 0x0000000002089020 0x0000000002089020
0x0000000000000010 0x00000000000014b0 R 20
Mkldnn/mklml动态库里都只有32字节的TLS;Paddle inference库则有0x14b0=5296字节的TLS。
求问这个是咋解决的?
任务10连跪。。
目前唯一的解决办法是编译源码安装paddle,编译时把MKL选项关掉:
https://github.com/PaddlePaddle/Paddle/blob/develop/CMakeLists.txt#L65
继续探索更科学的解决办法...
Failed to find dynamic library: libwarpctc.so ( dlopen: cannot load any more object with static TLS )
Find lbwarpctc.so
in python site-packages:
find ${PythonPath}/python2.7/site-packages -name libwarpctc.so
Use readelf -l libwarpctc.so
check memory size of TLS:
Elf file type is DYN (Shared object file)
Entry point 0x78f0
There are 6 program headers, starting at offset 64
Program Headers:
Type Offset VirtAddr PhysAddr
FileSiz MemSiz Flags Align
LOAD 0x0000000000000000 0x0000000000000000 0x0000000000000000
0x00000000004ec0c4 0x00000000004ec0c4 R E 200000
LOAD 0x00000000004ec0c8 0x00000000006ec0c8 0x00000000006ec0c8
0x0000000000002c40 0x0000000000003de0 RW 200000
DYNAMIC 0x00000000004ee3f0 0x00000000006ee3f0 0x00000000006ee3f0
0x0000000000000240 0x0000000000000240 RW 8
NOTE 0x0000000000000190 0x0000000000000190 0x0000000000000190
0x0000000000000024 0x0000000000000024 R 4
GNU_EH_FRAME 0x00000000004dff90 0x00000000004dff90 0x00000000004dff90
0x000000000000274c 0x000000000000274c R 4
GNU_STACK 0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 RW 10
Section to Segment mapping:
Segment Sections...
00 .note.gnu.build-id .gnu.hash .dynsym .dynstr .gnu.version .gnu.version_r .rela.dyn .rela.plt .init .plt .text .fini .rodata .nv_fatbin .eh_frame_hdr .eh_frame
01 .init_array .fini_array .jcr .data.rel.ro .dynamic .got .got.plt .data .nvFatBinSegment .bss
02 .dynamic
03 .note.gnu.build-id
04 .eh_frame_hdr
05
There is no TLS in program headers. So we should check all the so
needed by libwarpctc.so
.
readelf -d libwarpctc.so
Dynamic section at offset 0x4ee3f0 contains 32 entries:
Tag Type Name/Value
0x0000000000000001 (NEEDED) Shared library: [libpthread.so.0]
0x0000000000000001 (NEEDED) Shared library: [libdl.so.2]
0x0000000000000001 (NEEDED) Shared library: [librt.so.1]
0x0000000000000001 (NEEDED) Shared library: [libstdc++.so.6]
0x0000000000000001 (NEEDED) Shared library: [libm.so.6]
0x0000000000000001 (NEEDED) Shared library: [libgomp.so.1]
0x0000000000000001 (NEEDED) Shared library: [libgcc_s.so.1]
0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
0x000000000000000e (SONAME) Library soname: [libwarpctc.so]
ldd libwarpctc.so
linux-vdso.so.1 (0x00007ffd8bda5000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f852982f000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f852962a000)
librt.so.1 => /lib64/librt.so.1 (0x00007f8529422000)
libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f852911c000)
libm.so.6 => /lib64/libm.so.6 (0x00007f8528e19000)
libgomp.so.1 => /usr/lib64/libgomp.so.1 (0x00007f8528c0c000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f85289f6000)
libc.so.6 => /lib64/libc.so.6 (0x00007f8528634000)
/opt/compiler/gcc-4.8.2/lib64/ld-linux-x86-64.so.2 (0x00007f852a150000)
/usr/lib64/libgomp.so.1
is an openmpi library:
readelf -l /usr/lib64/libgomp.so.1
Elf file type is DYN (Shared object file)
Entry point 0x38c0003640
There are 7 program headers, starting at offset 64
Program Headers:
Type Offset VirtAddr PhysAddr
FileSiz MemSiz Flags Align
TLS 0x000000000000caa0 0x00000038c020caa0 0x00000038c020caa0
0x0000000000000000 0x0000000000000064 R 8
/usr/lib64/libgomp.so.1
need 0x0000000000000064 (100bytes) TLS.
0x0000000000000064 (100bytes) TLS is necessary for loading libwarpctc.so.
readelf -l libmkldnn.so.0
Program Headers:
Type Offset VirtAddr PhysAddr
FileSiz MemSiz Flags Align
TLS 0x00000000009583b0 0x0000000000b583b0 0x0000000000b583b0
0x0000000000000000 0x0000000000000038 R 10
readelf -l libmklml_intel.so
Program Headers:
Type Offset VirtAddr PhysAddr
FileSiz MemSiz Flags Align
TLS 0x0000000007a043e8 0x0000000007c043e8 0x0000000007c043e8
0x0000000000000000 0x0000000000000020 R 8
0x0000000000000058(88bytes) TLS is occupied by mkl library.
readelf -l ./paddle1.6/lib/python2.7/site-packages/paddle/libs/libpaddle_framework.so
Program Headers:
Type Offset VirtAddr PhysAddr
FileSiz MemSiz Flags Align
TLS 0x0000000000b88000 0x0000000000d88000 0x0000000000d88000
0x0000000000000020 0x0000000000000430 R 20
readelf -l ./paddle1.6/lib/python2.7/site-packages/paddle/fluid/core_noavx.so
TLS 0x000000001fa1b020 0x000000001fc1b020 0x000000001fc1b020
0x0000000000000014 0x00000000000013f0 R 10
readelf -l ./paddle1.6/lib/python2.7/site-packages/paddle/fluid/core_avx.so
TLS 0x000000001fbf3000 0x000000001fdf3000 0x000000001fdf3000
0x0000000000000024 0x0000000000001ad0 R 20
Analyse TLS in core_avx.so:
objdump -t ./paddle1.6/lib/python2.7/site-packages/paddle/fluid/core_avx.so | grep ".tbss" |awk -F" " '{cmd="c++filt "$5; system(cmd); print $4}'
guard variable for paddle::framework::MultiSlotDataFeed::ParseOneInstanceFromPipe(std::vector<paddle::framework::MultiSlotType, std::allocator<paddle::framework::MultiSlotType> >*)::reader
0000000000000008
paddle::framework::MultiSlotDataFeed::ParseOneInstanceFromPipe(std::vector<paddle::framework::MultiSlotType, std::allocator<paddle::framework::MultiSlotType> >*)::reader
0000000000000018
guard variable for paddle::framework::MultiSlotInMemoryDataFeed::ParseOneInstanceFromPipe(paddle::framework::Record*)::reader
0000000000000008
paddle::framework::MultiSlotInMemoryDataFeed::ParseOneInstanceFromPipe(paddle::framework::Record*)::reader
0000000000000018
guard variable for paddle::framework::FleetWrapper::LocalRandomEngine()::r
0000000000000008
paddle::framework::FleetWrapper::LocalRandomEngine()::r
0000000000000008
paddle::platform::g_event_list
0000000000000010
__tls_guard
0000000000000001
paddle::platform::g_mem_event_list
0000000000000010
paddle::platform::g_thread_id
0000000000000004
paddle::platform::g_mem_thread_id
0000000000000004
g_current_thread_worker
0000000000000008
g_current_thread_pollset
0000000000000008
g_current_thread_worker
0000000000000008
g_current_thread_pollset
0000000000000008
g_current_thread_worker
0000000000000008
g_current_thread_pollset
0000000000000008
g_initialized_sigmask
0000000000000001
g_current_thread_polling_island
0000000000000008
g_orig_sigmask
0000000000000080
__tls_guard
0000000000000001
paddle::platform::(anonymous namespace)::cur_input_shape_str
0000000000000008
paddle::platform::(anonymous namespace)::cur_mkldnn_session_id
0000000000000008
__tls_guard
0000000000000001
paddle::platform::(anonymous namespace)::block_id_stack
0000000000000050
paddle::platform::(anonymous namespace)::annotation_stack
0000000000000050
g_this_thread_state
0000000000000008
g_last_seen_min_timer
0000000000000008
g_cached_cq
0000000000000008
g_cached_event
0000000000000008
guard variable for paddle::framework::global_transfer_data_cache()::x
0000000000000008
paddle::framework::global_transfer_data_cache()::x
0000000000000008
guard variable for paddle::framework::global_transfer_scope_cache()::x
0000000000000008
paddle::framework::global_transfer_scope_cache()::x
0000000000000008
gpr_default_log(gpr_log_func_args*)::tid
0000000000000008
g_current_thread_worker
0000000000000008
g_current_thread_poller
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::MatMulTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)9>::Instance()::g_jit_codes
0000000000000008
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)16>::Instance()::g_jit_codes
0000000000000008
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)3>::Instance()::g_jit_codes
0000000000000030
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::CRFDecodingTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)30>::Instance()::g_jit_codes
0000000000000030
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::VIdentityTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::VExpTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)27>::Instance()::g_jit_codes
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::GRUH1Tuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)5>::Instance()::g_jit_codes
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::VIdentityTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)6>::Instance()::g_jit_codes
0000000000000008
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::VSquareTuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)15>::Instance()::g_jit_codes
0000000000000030
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::GRUHtPart2Tuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::CRFDecodingTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
paddle::operators::jit::KernelFuncs<paddle::operators::jit::HMaxTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
paddle::operators::jit::KernelFuncs<paddle::operators::jit::SgdTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)24>::Instance()::g_jit_codes
0000000000000030
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)29>::Instance()::g_jit_codes
0000000000000030
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)25>::Instance()::g_jit_codes
0000000000000030
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)26>::Instance()::g_jit_codes
0000000000000030
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::VBroadcastTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::VScalTuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::VSigmoidTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::LSTMCtHtTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
paddle::operators::jit::KernelFuncs<paddle::operators::jit::VScalTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)30>::Instance()::g_jit_codes
0000000000000008
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::VExpTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::SeqPoolTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)18>::Instance()::g_jit_codes
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::VBroadcastTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::GRUHtPart2Tuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::HSumTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
paddle::operators::jit::KernelFuncs<paddle::operators::jit::GRUHtPart1Tuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
paddle::operators::jit::KernelFuncs<paddle::operators::jit::GRUHtPart1Tuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::SoftmaxTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::MatMulTuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)17>::Instance()::g_jit_codes
0000000000000008
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::SgdTuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::GRUH1Tuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)31>::Instance()::g_jit_codes
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::StrideASumTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::VAddTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::HSumTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)9>::Instance()::g_jit_codes
0000000000000030
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::VAddReluTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)24>::Instance()::g_jit_codes
0000000000000008
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)7>::Instance()::g_jit_codes
0000000000000008
paddle::platform::DeviceTracerImpl::AddMemInfoRecord(unsigned long, unsigned long, unsigned long, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long)::local_mem_info_record
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::VAddTuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::LSTMCtHtTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::GRUH1Tuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)31>::Instance()::g_jit_codes
0000000000000030
paddle::operators::jit::KernelFuncs<paddle::operators::jit::GRUHtPart2Tuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)14>::Instance()::g_jit_codes
0000000000000030
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)8>::Instance()::g_jit_codes
0000000000000030
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)23>::Instance()::g_jit_codes
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::VAddReluTuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)22>::Instance()::g_jit_codes
0000000000000008
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::LSTMCtHtTuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)4>::Instance()::g_jit_codes
0000000000000030
paddle::operators::jit::KernelFuncs<paddle::operators::jit::VSubTuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::VScalTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)5>::Instance()::g_jit_codes
0000000000000030
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)1>::Instance()::g_jit_codes
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::VMulTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)26>::Instance()::g_jit_codes
0000000000000008
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)20>::Instance()::g_jit_codes
0000000000000030
paddle::operators::jit::KernelFuncs<paddle::operators::jit::SoftmaxTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)14>::Instance()::g_jit_codes
0000000000000008
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::CRFDecodingTuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::VSigmoidTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::GRUHtPart1Tuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::VAddBiasTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::VMulTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)13>::Instance()::g_jit_codes
0000000000000008
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)15>::Instance()::g_jit_codes
0000000000000008
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)29>::Instance()::g_jit_codes
0000000000000008
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::VReluTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)12>::Instance()::g_jit_codes
0000000000000008
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::VSubTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::SgdTuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
paddle::operators::jit::KernelFuncs<paddle::operators::jit::VSubTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)17>::Instance()::g_jit_codes
0000000000000030
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)13>::Instance()::g_jit_codes
0000000000000030
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)3>::Instance()::g_jit_codes
0000000000000008
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::SeqPoolTuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::GRUHtPart2Tuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
paddle::operators::jit::KernelFuncs<paddle::operators::jit::SeqPoolTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
paddle::platform::DeviceTracerImpl::AddAnnotation(unsigned int, paddle::platform::Event*)::local_correlations_pairs
0000000000000008
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)12>::Instance()::g_jit_codes
0000000000000030
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::MatMulTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::HMaxTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)23>::Instance()::g_jit_codes
0000000000000030
paddle::operators::jit::KernelFuncs<paddle::operators::jit::VTanhTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::GRUHtPart1Tuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)28>::Instance()::g_jit_codes
0000000000000008
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::VBroadcastTuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::LSTMC1H1Tuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
paddle::operators::jit::KernelFuncs<paddle::operators::jit::VAddTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
paddle::platform::DeviceTracerImpl::AddActiveKindRecords(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, unsigned long, long, long, unsigned int)::local_active_kind_records
0000000000000008
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::VAddBiasTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::MatMulTuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::SgdTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::SeqPoolTuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)16>::Instance()::g_jit_codes
0000000000000030
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::VAddReluTuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)27>::Instance()::g_jit_codes
0000000000000030
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)1>::Instance()::g_jit_codes
0000000000000030
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::LSTMC1H1Tuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)11>::Instance()::g_jit_codes
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::CRFDecodingTuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
paddle::platform::DeviceTracerImpl::AddCPURecords(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, unsigned long, long, long)::local_cpu_records_
0000000000000008
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::NCHW16CMulNCTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::VSquareTuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
paddle::operators::jit::KernelFuncs<paddle::operators::jit::VReluTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::VSubTuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)20>::Instance()::g_jit_codes
0000000000000008
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::VTanhTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)19>::Instance()::g_jit_codes
0000000000000030
paddle::operators::jit::KernelFuncs<paddle::operators::jit::StrideScalTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::StrideASumTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)19>::Instance()::g_jit_codes
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::LSTMC1H1Tuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)4>::Instance()::g_jit_codes
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::VAddReluTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::VAddTuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)28>::Instance()::g_jit_codes
0000000000000030
paddle::operators::jit::KernelFuncs<paddle::operators::jit::LSTMCtHtTuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)6>::Instance()::g_jit_codes
0000000000000030
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)18>::Instance()::g_jit_codes
0000000000000030
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::VScalTuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)7>::Instance()::g_jit_codes
0000000000000030
paddle::operators::jit::KernelFuncs<paddle::operators::jit::VBroadcastTuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)25>::Instance()::g_jit_codes
0000000000000008
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)22>::Instance()::g_jit_codes
0000000000000030
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::GRUH1Tuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
guard variable for paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)8>::Instance()::g_jit_codes
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::NCHW16CMulNCTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
grpc_core::ExecCtx::exec_ctx_
0000000000000008
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::LSTMC1H1Tuple<double>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::StrideScalTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
guard variable for paddle::operators::jit::KernelFuncs<paddle::operators::jit::VSquareTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000008
paddle::operators::jit::KernelFuncs<paddle::operators::jit::VSquareTuple<float>, paddle::platform::CPUPlace>::Cache()::g_func_cache
0000000000000030
paddle::operators::jit::JitCodePool<(paddle::operators::jit::KernelType)11>::Instance()::g_jit_codes
0000000000000030
@bingyanghuang Could you help see it?
Error log: