Closed GoogleCodeExporter closed 9 years ago
memcached version 1.4.11 & redhat6.0
Original comment by lys0...@qq.com
on 13 Mar 2012 at 7:36
can you upgrade to 1.4.13?
if it still happens, you'll need to get a gdb backtrace for us.
Original comment by dorma...@rydia.net
on 13 Mar 2012 at 7:37
how can i get a gdb backtrace ?
Original comment by lys0...@qq.com
on 13 Mar 2012 at 9:00
To get a backtrace you can either take a core dump from memcached, or start it
under gdb.
"gdb memcached etc" in a screen session. then when it crashes, run "bt" and
"thread apply all bt" and give us the output of both.
Original comment by dorma...@rydia.net
on 10 Apr 2012 at 10:58
Never got a backtrace :/ closing.
if you see this and have crash info for 1.4.13 please re-open a bug.
Original comment by dorma...@rydia.net
on 14 Jul 2012 at 11:35
I use memcached 1.4.15 and fedora 14
I had the same error message
kernel: memcached[xxxxxx]: segfault at 0 ip xxxxxx sp xxxxxx error 4 in
memcached[400000+19000]
Original comment by cans...@gmail.com
on 16 Nov 2012 at 2:28
> I had the same error message
Yes! I have the same error!
Memcached 1.4.15 & CentOS 6.3
Original comment by dunaev.y...@gmail.com
on 30 Dec 2012 at 6:17
Please provide a backtrace!
We really really need the backtrace!
Open a new bug report with a back trace!
Original comment by dorma...@rydia.net
on 31 Dec 2012 at 12:59
> Please provide a backtrace!
Sorry for my English.
But I can not afford it.
When GDB is working properly.
But for GDB to disable the option "-d"
Otherwise, GDB does not work.
But the error occurs only when the option "-d" is enabled.
Again error:
tails@srv01 ~ % dmesg | grep 'error'
memcached[28680]: segfault at 0 ip 000000000040dfd8 sp 00007faf8ff16a70 error 4
in memcached[400000+19000]
Original comment by dunaev.y...@gmail.com
on 13 Jan 2013 at 1:19
You can use GDB to attach to a running process:
memcached -d -m etc
gdb memcached $(pidof memcached)
ignore sigpipe and continue.
With just the segfault we will never be able to figure out the crash.
Original comment by dorma...@rydia.net
on 13 Jan 2013 at 1:50
I'm doing this:
1) screen -fn -S gdb_memcached -dm gdb memcached $(pidof memcached)
2) screen -r gdb_memcached
Displays:
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-56.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/bin/memcached...(no debugging symbols found)...done.
Attaching to program: /usr/bin/memcached, process 15090
Reading symbols from /usr/lib64/libevent-1.4.so.2...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libevent-1.4.so.2
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /usr/lib64/libsasl2.so.2...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libsasl2.so.2
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols
found)...done.
[Thread debugging using libthread_db enabled]
[New Thread 0x7f04f7e13700 (LWP 15096)]
[New Thread 0x7f04f8614700 (LWP 15094)]
[New Thread 0x7f04f8e15700 (LWP 15093)]
[New Thread 0x7f04f9616700 (LWP 15092)]
[New Thread 0x7f04f9e17700 (LWP 15091)]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnsl.so.1
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/libfreebl3.so...(no debugging symbols found)...done.
Loaded symbols for /lib64/libfreebl3.so
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libnss_files.so.2
0x00007f04fabf8713 in epoll_wait () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install
memcached-1.4.15-2.el6.remi.x86_64
(gdb)
Whether or not to run the "run" in the debugger?
Or leave as is?
Original comment by dunaev.y...@gmail.com
on 13 Jan 2013 at 9:49
$ gdb memcached $(pidof memcached)
GNU gdb (GDB) 7.1-ubuntu
Copyright (C) 2010 Free Software Foundation, Inc.
[...etc...]
(gdb) handle SIGPIPE nostop noprint pass
Signal Stop Print Pass to program Description
SIGPIPE No No Yes Broken pipe
(gdb) c
Continuing.
Original comment by dorma...@rydia.net
on 13 Jan 2013 at 7:19
frens, this would soon be a big problem to me since i m setting up a server
with memcache 1.4.15. This would soon become a production hence i am very
concerned.
I hit this error once a day. my setup is like a RHEL6 64bit VM with 2vCPUs 20G
RAM, and i run memcache as "memcached -d -l 127.0.0.1 -p 11211 -m 128 -t 12 -u
root"
dmesg tells me :
memcached[10519]: segfault at 0 ip 000000000040d858 sp 00007fd9acd0b9b0 error 4
in memcached[400000+18000]
memcached[4260]: segfault at 0 ip 000000000040d858 sp 00007f5389bcea50 error 4
in memcached[400000+18000]
memcached[30828]: segfault at 0 ip 000000000040d858 sp 00007f095f8c2a50 error 4
in memcached[400000+18000]
memcached[23240]: segfault at 0 ip 000000000040d858 sp 00007f7a4ec6ca50 error 4
in memcached[400000+18000]
memcached[8702]: segfault at 0 ip 000000000040d858 sp 00007fe49a624a50 error 4
in memcached[400000+18000]
Is there a workaround ? If previous versions of memcache work stable, i am ok
to switch back, please assist.
PK
Original comment by mkpa...@gmail.com
on 15 Feb 2013 at 7:28
For the love of god, and all that is holy, please provide a backtrace.
Do you not SEE all of the other posts in this thread wherein I ask for a
backtrace? Wherein I describe how to get a back trace. Are you blind? Are all
of you blind or are you trolling me?
I HAVE NO FUCKING IDEA WHAT YOUR CRASH IS WITHOUT SUPPLYING A BACKTRACE. PLEASE
DO IT.
Original comment by dorma...@rydia.net
on 15 Feb 2013 at 7:36
thanks for being polite !! i watched it again and have run a gdb as well, and
did "c" to continuing..
should i just paste the stuff here when it fails the next time ? I would do, if
that is what you are expecting to provide me a solution.. thanks.
Original comment by mkpa...@gmail.com
on 15 Feb 2013 at 7:42
When it crashes again, and you have a backtrace, paste it here so we can read
it. Depending on how it looks we may be able to fix your issue.
Without a backtrace, we have no idea why it crashed. A "segfault" is a generic
error, and doesn't mean anything. That just says it crashed. We need to know
why. This thread is full of me explaining this.
Original comment by dorma...@rydia.net
on 15 Feb 2013 at 7:47
still no backtrace. ennui.
Original comment by dorma...@rydia.net
on 2 Mar 2013 at 7:29
Here is a back trace. I get the segmentation error every day (fedora 18
memcached-1.4.15-2.fc18.i686)
[root@www2 quick]# gdb memcached 9128
GNU gdb (GDB) Fedora (7.5.1-36.fc18)
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/bin/memcached...Reading symbols from
/usr/lib/debug/usr/bin/memcached.debug...done.
done.
Attaching to program: /usr/bin/memcached, process 9128
Reading symbols from /lib/libevent-2.0.so.5...Reading symbols from
/usr/lib/debug/usr/lib/libevent-2.0.so.5.1.6.debug...done.
done.
Loaded symbols for /lib/libevent-2.0.so.5
Reading symbols from /lib/librt.so.1...Reading symbols from
/usr/lib/debug/lib/librt-2.16.so.debug...done.
done.
Loaded symbols for /lib/librt.so.1
Reading symbols from /lib/libpthread.so.0...Reading symbols from
/usr/lib/debug/lib/libpthread-2.16.so.debug...done.
done.
[New LWP 9133]
[New LWP 9132]
[New LWP 9131]
[New LWP 9130]
[New LWP 9129]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/libthread_db.so.1".
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libc.so.6...Reading symbols from
/usr/lib/debug/lib/libc-2.16.so.debug...done.
done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux.so.2...Reading symbols from
/usr/lib/debug/lib/ld-2.16.so.debug...done.
done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /lib/libnss_files.so.2...Reading symbols from
/usr/lib/debug/lib/libnss_files-2.16.so.debug...done.
done.
Loaded symbols for /lib/libnss_files.so.2
0xb77dc416 in __kernel_vsyscall ()
(gdb) where
#0 0xb77dc416 in __kernel_vsyscall ()
#1 0xb76a2a96 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
#2 0xb77a9968 in epoll_dispatch (base=0x9f48de8, tv=0xbfb93040) at epoll.c:407
#3 0xb7793132 in event_base_loop (base=0x9f48de8, flags=0) at event.c:1590
#4 0x0804baae in main (argc=10, argv=0xbfb94204) at memcached.c:5228
(gdb) where
#0 0xb77dc416 in __kernel_vsyscall ()
#1 0xb76a2a96 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
#2 0xb77a9968 in epoll_dispatch (base=0x9f48de8, tv=0xbfb93040) at epoll.c:407
#3 0xb7793132 in event_base_loop (base=0x9f48de8, flags=0) at event.c:1590
#4 0x0804baae in main (argc=10, argv=0xbfb94204) at memcached.c:5228
(gdb) bt
#0 0xb77dc416 in __kernel_vsyscall ()
#1 0xb76a2a96 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
#2 0xb77a9968 in epoll_dispatch (base=0x9f48de8, tv=0xbfb93040) at epoll.c:407
#3 0xb7793132 in event_base_loop (base=0x9f48de8, flags=0) at event.c:1590
#4 0x0804baae in main (argc=10, argv=0xbfb94204) at memcached.c:5228
(gdb) thread
[Current thread is 1 (Thread 0xb75ab6c0 (LWP 9128))]
(gdb) c
Continuing.
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xb655ab40 (LWP 9131)]
do_slabs_alloc (id=1, size=size@entry=76) at slabs.c:241
241 p->slots = it->next;
(gdb) bt
#0 do_slabs_alloc (id=1, size=size@entry=76) at slabs.c:241
#1 slabs_alloc (size=size@entry=76, id=id@entry=1) at slabs.c:404
#2 0x0805644d in do_item_alloc (key=0xb49fff30 "my_wiki:stats:pcache_hit\n 0
3\r\n559\r\n9\r\n\n", nkey=24, flags=0, exptime=exptime@entry=0,
nbytes=nbytes@entry=5, cur_hv=cur_hv@entry=123356228) at items.c:188
#3 0x080535fa in do_add_delta (c=c@entry=0xb4a23988, key=key@entry=0xb4a25e2d
"my_wiki:stats:pcache_hit", nkey=nkey@entry=24, incr=true, delta=1,
buf=buf@entry=0xb655a0b4 "560", cas=cas@entry=0x0, hv=hv@entry=123356228) at memcached.c:3102
#4 0x080586e5 in add_delta (c=c@entry=0xb4a23988, key=key@entry=0xb4a25e2d
"my_wiki:stats:pcache_hit", nkey=nkey@entry=24, incr=incr@entry=1, delta=1,
buf=buf@entry=0xb655a0b4 "560", cas=cas@entry=0x0) at thread.c:585
#5 0x0804da3d in process_arithmetic_command (c=c@entry=0xb4a23988,
tokens=tokens@entry=0xb655a140, ntokens=ntokens@entry=4, incr=incr@entry=true)
at memcached.c:3015
#6 0x080506cc in process_command (c=c@entry=0xb4a23988, command=<optimized
out>) at memcached.c:3266
#7 0x08050f23 in try_read_command (c=0xb4a23988) at memcached.c:3504
#8 drive_machine (c=0xb4a23988) at memcached.c:3824
#9 event_handler (fd=30, which=2, arg=0xb4a23988) at memcached.c:4065
#10 0xb77933da in event_process_active_single_queue (activeq=<optimized out>,
base=<optimized out>) at event.c:1340
#11 event_process_active (base=<optimized out>) at event.c:1407
#12 event_base_loop (base=0x9f6eaf0, flags=flags@entry=0) at event.c:1604
#13 0x08057cdf in worker_libevent (arg=0x9f67a68) at thread.c:384
#14 0xb7769aff in start_thread (arg=0xb655ab40) at pthread_create.c:308
#15 0xb76a209e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:132
(gdb)
Original comment by edwardqu...@gmail.com
on 11 Mar 2013 at 11:51
Do the backtraces always the look the same for you?
can we get a look at "stats settings" / "stats" / "stats slabs" / "stats items"
as well?
thank you!
Original comment by dorma...@rydia.net
on 11 Mar 2013 at 6:37
This was the first one I did, so I'll check again and get back to you on
that tomorrow (probably).
Sure. I've copied this info to http://linuxproblems.org/memcached/ along
with the core dump I took earlier in case that's of any use.
Original comment by edwardqu...@gmail.com
on 11 Mar 2013 at 7:07
It happened again sooner than I expected. I'll see if I can get another one
for you.
(gdb) bt
#0 do_slabs_alloc (id=1, size=size@entry=73) at slabs.c:241
#1 slabs_alloc (size=size@entry=73, id=id@entry=1) at slabs.c:404
#2 0x08056269 in do_item_alloc (key=key@entry=0x86c5384
"my_wiki:valid-tags", nkey=nkey@entry=18, flags=flags@entry=1,
exptime=exptime@entry=38234,
nbytes=nbytes@entry=8, cur_hv=cur_hv@entry=0) at items.c:150
#3 0x080583d8 in item_alloc (key=key@entry=0x86c5384 "my_wiki:valid-tags",
nkey=nkey@entry=18, flags=1, exptime=38234, nbytes=nbytes@entry=8)
at thread.c:486
#4 0x0804d808 in process_update_command (c=c@entry=0x86bcba0,
tokens=tokens@entry=0xb6cc5140, ntokens=ntokens@entry=6, comm=2,
handle_cas=handle_cas@entry=false) at memcached.c:2917
#5 0x0804fc08 in process_command (c=c@entry=0x86bcba0, command=<optimized
out>) at memcached.c:3258
#6 0x08050f23 in try_read_command (c=0x86bcba0) at memcached.c:3504
#7 drive_machine (c=0x86bcba0) at memcached.c:3824
#8 event_handler (fd=30, which=2, arg=0x86bcba0) at memcached.c:4065
#9 0xb76fd3da in event_process_active_single_queue (activeq=<optimized
out>, base=<optimized out>) at event.c:1340
#10 event_process_active (base=<optimized out>) at event.c:1407
#11 event_base_loop (base=0x8694598, flags=flags@entry=0) at event.c:1604
#12 0x08057cdf in worker_libevent (arg=0x868a73c) at thread.c:384
#13 0xb76d3aff in start_thread (arg=0xb6cc5b40) at pthread_create.c:308
#14 0xb760c09e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:132
(gdb) generate-core-file
Saved corefile core.11732
Original comment by edwardqu...@gmail.com
on 11 Mar 2013 at 10:45
what was the original error for that last backtrace?
Original comment by dorma...@rydia.net
on 11 Mar 2013 at 10:46
(gdb) c
Continuing.
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xb6cc5b40 (LWP 11734)]
do_slabs_alloc (id=1, size=size@entry=73) at slabs.c:241
241 p->slots = it->next;
Original comment by edwardqu...@gmail.com
on 11 Mar 2013 at 11:03
Happened again. Looks like it happens in the same place everytime.
(gdb) c
Continuing.
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xb6cbfb40 (LWP 14753)]
do_slabs_alloc (id=1, size=size@entry=76) at slabs.c:241
241 p->slots = it->next;
(gdb) bt
#0 do_slabs_alloc (id=1, size=size@entry=76) at slabs.c:241
#1 slabs_alloc (size=size@entry=76, id=id@entry=1) at slabs.c:404
#2 0x08056269 in do_item_alloc (key=0xb48fffd0 "my_wiki:stats:pcache_hit_ 0
3\r\n281\r\n1\r\n\n", nkey=24, flags=0, exptime=exptime@entry=0,
nbytes=nbytes@entry=5, cur_hv=cur_hv@entry=123356228) at items.c:150
#3 0x080535fa in do_add_delta (c=c@entry=0xb4b23b00, key=key@entry=0xb4b25d95
"my_wiki:stats:pcache_hit", nkey=nkey@entry=24, incr=true, delta=1,
buf=buf@entry=0xb6cbf0b4 "282", cas=cas@entry=0x0, hv=hv@entry=123356228) at memcached.c:3102
#4 0x080586e5 in add_delta (c=c@entry=0xb4b23b00, key=key@entry=0xb4b25d95
"my_wiki:stats:pcache_hit", nkey=nkey@entry=24, incr=incr@entry=1, delta=1,
buf=buf@entry=0xb6cbf0b4 "282", cas=cas@entry=0x0) at thread.c:585
#5 0x0804da3d in process_arithmetic_command (c=c@entry=0xb4b23b00,
tokens=tokens@entry=0xb6cbf140, ntokens=ntokens@entry=4, incr=incr@entry=true)
at memcached.c:3015
#6 0x080506cc in process_command (c=c@entry=0xb4b23b00, command=<optimized
out>) at memcached.c:3266
#7 0x08050f23 in try_read_command (c=0xb4b23b00) at memcached.c:3504
#8 drive_machine (c=0xb4b23b00) at memcached.c:3824
#9 event_handler (fd=30, which=2, arg=0xb4b23b00) at memcached.c:4065
#10 0xb76f73da in event_process_active_single_queue (activeq=<optimized out>,
base=<optimized out>) at event.c:1340
#11 event_process_active (base=<optimized out>) at event.c:1407
#12 event_base_loop (base=0x95cf598, flags=flags@entry=0) at event.c:1604
#13 0x08057cdf in worker_libevent (arg=0x95c573c) at thread.c:384
#14 0xb76cdaff in start_thread (arg=0xb6cbfb40) at pthread_create.c:308
#15 0xb760609e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:132
(gdb) generate-core-file
Saved corefile core.14751
(gdb) quit
Original comment by edwardqu...@gmail.com
on 12 Mar 2013 at 10:00
dorma...@rydia.net, have some ideas to solve the problem? Оо
Original comment by dunaev.y...@gmail.com
on 8 Sep 2013 at 8:51
I'm encountering this same problem on CentOS 6.3 and the github source as of
yesterday 2013/09/27 (1.4.15_6_g87e2f36).
My commandline is:
memcached -d -p 11211 -m 1024 -c 4096 -P /var/run/memcached/memcached.pid -l
127.0.0.1 -k -vv
I'm running it under gdb, but have you made any progress with the above bt?
Original comment by nicjan...@gmail.com
on 27 Sep 2013 at 1:24
Hi,
If anyone listening is still having crashes, please try 1.4.16, with a -debug
binary, under gdb. Don't forget "handle SIGPIPE nostop" in gdb before starting.
If you do get a crash, build from this tree:
https://github.com/memcached/memcached/tree/next
... and try again (run ./autogen.sh before ./configure, still use a -debug
binary)
Not confident I've found the crash but I'm trying to narrow it down.
Original comment by dorma...@rydia.net
on 9 Dec 2013 at 7:38
memcached version 1.4.17
redhat6.4 kernel 2.6.32-358.el6.x86_64
memcached will automatically die, logging the following information
more /var/log/messages|grep memca
Feb 13 14:03:22 simp-nginx kernel: memcached[1346]: segfault at 0 ip
000000000040ddf8 sp 00007fa3fd637a50 error 4 in memcached[400000+19000]
Original comment by mazhongx...@gmail.com
on 13 Feb 2014 at 6:33
My team encounter this issue too, but we cann't reproduce it. But i found a
similar issue in Couchbase project, hope it help us to address this issue.
"Couchbase Server MB-4783
overstressed memcached keeps conns in close_wait even if the client has already
closed its conns"
http://www.couchbase.com/issues/browse/MB-4783?page=com.atlassian.jira.plugin.sy
stem.issuetabpanels:all-tabpanel
----------
I don't understand the way that they fixed it. Could anyone tell me more about
it?
Revision: 7ef0ffa78ba28a120d61d01f0e5967bf5875317b
Author: Bin Cui <bin.cui@gmail.com>
Date: 2012/2/15 9:54:05
Message:
MB-4738 Have tcmalloc support for windows
Use tcmalloc to address memory fragmentation issue.
Change-Id: Ib3a2ff0fc54f2000ec6a1d4eab3568d3d3fe46e7
Reviewed-on: http://review.couchbase.org/13243
Reviewed-by: Steve Yen <steve.yen@gmail.com>
Tested-by: Steve Yen <steve.yen@gmail.com>
----
Modified: win32/Makefile.mingw
--------------------------
You can get the code here:
https://github.com/couchbase/ep-engine/tree/1.8-MB-4738
Original comment by Z.W.Chan...@gmail.com
on 22 Apr 2014 at 1:23
Please for the love of god stop necroing this thread. Open a new bug, and
provide a backtrace or any random sort of information.
A link to a couchbase fix from two years ago is not helpful at all. That is a
completely unrelated codebase.
I have no idea what you were seeing. Are you seeing a hang? a crash? a segfault?
Original comment by dorma...@rydia.net
on 22 Apr 2014 at 1:42
Hi,
For anyone about to necro this thread again:
https://github.com/memcached/memcached/pull/67
I'm pretty sure this is the fix for the original crash, which has now been
"improved" in three different ways.
If you are seeing memcached segfault, please open a new ticket with information
on the backtrace/etc.
These latest fixes should be in 1.4.19 or higher.
Original comment by dorma...@rydia.net
on 28 Apr 2014 at 6:44
Original issue reported on code.google.com by
lys0...@qq.com
on 13 Mar 2012 at 7:32