twitter / fatcache

Memcache on SSD
Apache License 2.0
1.3k stars 179 forks source link

SIGSEGV crashes, known issue? #34

Open jjones-smug opened 6 years ago

jjones-smug commented 6 years ago

We're evaluating the potential benefits of deploying Fatcache and I've run into a problem I haven't been able to resolve. The code compiles cleanly on the platform we're using, Ubuntu 16/Trusty, and the server process start just fine. But if I run performance tests with "memtier_benchmark" the server process crashes with a segmentation fault for any non-trivial test configuration.

I've tried running minimal configurations that use all defaults, I've compiled the v0.1.1 tag and the lastest code from the GitHub repo. And I've compiled with full debugging enabled and optimization (O0 flag) disabled. The process crashes each time no matter which configuration I use.

I'm hoping it's a known issue or something dumb I'm doing to setup/configure Fatcache. But since I haven't found anything by searching public documents related to Fatcache I wanted to post here to look for clues.

Here's the stack trace from "gdb" in case it helps.

Program received signal SIGSEGV, Segmentation fault. __memcpy_sse2_unaligned () at ../sysdeps/x86_64/multiarch/memcpy-sse2-unaligned.S:36 36 ../sysdeps/x86_64/multiarch/memcpy-sse2-unaligned.S: No such file or directory. (gdb) where

0 __memcpy_sse2_unaligned () at ../sysdeps/x86_64/multiarch/memcpy-sse2-unaligned.S:36

1 0x000000000040ccfc in mbuf_copy (mbuf=0x5889f60, pos=0x7ffff6ff0cf0 "", size=8144) at fc_mbuf.c:233

2 0x000000000040ce1c in mbuf_copy_from (mhdr=0x737b68, pos=0x7ffff6ff0cf0 "", size=2347828697) at fc_mbuf.c:267

3 0x000000000040c46f in rsp_send_value (ctx=0x7fffffffea00, conn=0x634ab0, msg=0x735a30, it=0x7ffff1ef104c, cas=423) at fc_response.c:338

4 0x000000000040aaae in req_process_get (ctx=0x7fffffffea00, conn=0x634ab0, msg=0x735a30) at fc_request.c:212

5 0x000000000040b8ae in req_process (ctx=0x7fffffffea00, conn=0x634ab0, msg=0x735a30) at fc_request.c:548

6 0x000000000040b9f6 in req_recv_done (ctx=0x7fffffffea00, conn=0x634ab0, msg=0x735a30, nmsg=0x0) at fc_request.c:610

7 0x0000000000409898 in msg_parsed (ctx=0x7fffffffea00, conn=0x634ab0, msg=0x735a30) at fc_message.c:168

8 0x0000000000409c99 in msg_parse (ctx=0x7fffffffea00, conn=0x634ab0, msg=0x735a30) at fc_message.c:317

9 0x0000000000409e85 in msg_recv_chain (ctx=0x7fffffffea00, conn=0x634ab0, msg=0x735a30) at fc_message.c:383

10 0x0000000000409f30 in msg_recv (ctx=0x7fffffffea00, conn=0x634ab0) at fc_message.c:417

11 0x000000000040415f in core_recv (ctx=0x7fffffffea00, conn=0x634ab0) at fc_core.c:77

12 0x000000000040454f in core_core (ctx=0x7fffffffea00, conn=0x634ab0, events=1) at fc_core.c:157

13 0x00000000004046f2 in core_loop (ctx=0x7fffffffea00) at fc_core.c:215

14 0x000000000041396d in main (argc=13, argv=0x7fffffffeb08) at fc.c:746

hlitz commented 6 years ago

I am having the same problem, very easy to reproduce

shanemarsh28 commented 3 years ago

Hi,

I can confirm this is still an issue.

[Tue Apr 20 10:16:04 2021] fc_util.c:694 [0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f4dd927c3c0] [Tue Apr 20 10:16:04 2021] fc_util.c:694 [1] fatcache(+0x667c) [0x55f42073967c] [Tue Apr 20 10:16:04 2021] fc_util.c:694 [2] fatcache(slab_get_item+0x14d) [0x55f420739c6d] [Tue Apr 20 10:16:04 2021] fc_util.c:694 [3] fatcache(item_get+0x3d) [0x55f42073a47d] [Tue Apr 20 10:16:04 2021] fc_util.c:694 [4] fatcache(+0x93f5) [0x55f42073c3f5] [Tue Apr 20 10:16:04 2021] fc_util.c:694 [5] fatcache(req_recv_done+0x286) [0x55f42073c926] [Tue Apr 20 10:16:04 2021] fc_util.c:694 [6] fatcache(msg_recv+0x1de) [0x55f42073c02e] [Tue Apr 20 10:16:04 2021] fc_util.c:694 [7] fatcache(core_loop+0x56) [0x55f420738906] [Tue Apr 20 10:16:04 2021] fc_util.c:694 [8] fatcache(main+0x758) [0x55f420738028] [Tue Apr 20 10:16:04 2021] fc_util.c:694 [9] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f4dd909c0b3] [Tue Apr 20 10:16:04 2021] fc_util.c:694 [10] fatcache(_start+0x2e) [0x55f42073868e] Segmentation fault

I've tried compiling from the repository and from the tarball. No joy. Such a shame. Is there any chance of this being fixed?

git-hulk commented 3 years ago

fatcache seems no maintenance for many years, you can have a try at https://github.com/bitleak/kvrocks which supports Redis protocol.

shanemarsh28 commented 3 years ago

Amazing! Thanks for the link - I will definitely try it out.

pataquets commented 3 years ago

Is fatcache no longer used/maintained as stated above? cc: @manjuraj