apache / brpc

brpc is an Industrial-grade RPC framework using C++ Language, which is often used in high performance system such as Search, Storage, Machine learning, Advertisement, Recommendation etc. "brpc" means "better RPC".
https://brpc.apache.org
Apache License 2.0
16.54k stars 3.97k forks source link

容器部署绑定cpu后coredump #2650

Open yybear opened 5 months ago

yybear commented 5 months ago

Describe the bug (描述bug) 服务使用了brpc库,使用容器绑定cpu部署(numa感知)。服务启动后会报Program terminated with signal 11, Segmentation fault.。

To Reproduce (复现方法) 容器绑定cpu部署(numa感知),必现。

Expected behavior (期望行为) 容器绑定cpu部署(numa感知)服务正常运行。

Versions (各种版本) OS: 龙蜥os 7.9 Compiler: brpc: 1.2 protobuf:

Additional context/screenshots (更多上下文/截图) Program terminated with signal 11, Segmentation fault.

0 ticker_ticks (nticks=1, ticker=0x7fe6f2ffc088) at include/jemalloc/internal/ticker.h:66

66 include/jemalloc/internal/ticker.h: No such file or directory. Missing separate debuginfos, use: debuginfo-install glibc-2.17-323.1.an7.1.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-55.0.1.an7.x86_64 libcom_err-1.43.5-8.4.an7.x86_64 libgcc-4.8.5-44.0.1.an7.x86_64 libgomp-4.8.5-44.0.1.an7.x86_64 libselinux-2.5-15.el7.x86_64 libstdc++-4.8.5-44.0.1.an7.x86_64 openssl-libs-1.0.2k-26.an7.x86_64 pcre-8.32-17.el7.x86_64 zlib-1.2.7-21.an7.x86_64 (gdb) bt

0 ticker_ticks (nticks=1, ticker=0x7fe6f2ffc088) at include/jemalloc/internal/ticker.h:66

1 arena_decay_ticks (nticks=1, arena=0x7fe6230008c0, tsdn=0x7fe575669200) at include/jemalloc/internal/arena_inlines_b.h:125

2 arena_decay_tick (arena=0x7fe6230008c0, tsdn=0x7fe575669200) at include/jemalloc/internal/arena_inlines_b.h:135

3 je_arena_tcache_fill_small (tsdn=, arena=, tcache=, tbin=, binind=, prof_accumbytes=) at src/arena.c:1442

4 0x0000000001998964 in je_tcache_alloc_small_hard (tsdn=tsdn@entry=0x7fe575669200, arena=, tcache=tcache@entry=0x7fe5756693f0, tbin=tbin@entry=0x7fe575669550, binind=binind@entry=14,

tcache_success=tcache_success@entry=0x7fe5756685a0) at src/tcache.c:94

5 0x00000000019493d7 in tcache_alloc_small (size=, slow_path=false, zero=false, binind=14, tcache=0x7fe5756693f0, arena=, tsd=0x7fe575669200)

at include/jemalloc/internal/tcache_inlines.h:60

6 arena_malloc (slow_path=false, tcache=0x7fe5756693f0, zero=false, ind=14, size=, arena=0x0, tsdn=0x7fe575669200) at include/jemalloc/internal/arena_inlines_b.h:165

7 iallocztm (slow_path=false, arena=0x0, is_internal=false, tcache=0x7fe5756693f0, zero=false, ind=14, size=, tsdn=0x7fe575669200)

at include/jemalloc/internal/jemalloc_internal_inlines_c.h:53

8 imalloc_no_sample (sopts=, ind=14, usize=384, size=, tsd=0x7fe575669200, dopts=) at src/jemalloc.c:1949

9 imalloc_body (tsd=0x7fe575669200, dopts=, sopts=) at src/jemalloc.c:2149

10 imalloc (dopts=, sopts=) at src/jemalloc.c:2258

11 je_malloc_default (size=) at src/jemalloc.c:2289

12 0x00007fe7d8ec121d in operator new(unsigned long, std::nothrow_t const&) () from /lib64/libstdc++.so.6

13 0x0000000001bb467f in bthread::TaskControl::create_group (this=this@entry=0x7fe7b0b03c00) at src/bthread/task_control.cpp:89

14 0x0000000001bb47e1 in bthread::TaskControl::worker_thread (arg=0x7fe7b0b03c00) at src/bthread/task_control.cpp:65

15 0x00007fe7b67fbea5 in start_thread () from /lib64/libpthread.so.0

16 0x00007fe7b65249fd in __libc_ifunc_impl_list () from /lib64/libc.so.6

lorinlee commented 5 months ago

请问jemalloc是什么版本?NUMA配的cpu分配策略和内存分配策略是什么?另外可以开启ASAN跑一下吗,不确定是不是其他地方的内存踩坏导致这里出错