Closed GoogleCodeExporter closed 9 years ago
The same for me :(
Original comment by alexey.s...@hivext.net
on 26 Jun 2012 at 4:13
Guys, can you please tell us where we need search for solution, it's very
important for us. Thanks.
Original comment by i...@hivext.net
on 26 Jun 2012 at 4:43
Guys, here is debug with GDB
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff202a700 (LWP 5697)]
0x00007ffff77835ce in sasl_server_step (serveroutlen=0x7ffff2029c2c,
serverout=0x7ffff2029c20, clientinlen=17, clientin=0x7ffff7e2a055 "",
conn=0x47bf90) at ../../lib/server.c:1614
1614 ../../lib/server.c: No such file or directory.
Original comment by i...@hivext.net
on 26 Jun 2012 at 5:15
Can you include a gdb backtrace (bt)?
Original comment by dorma...@rydia.net
on 14 Jul 2012 at 11:25
[deleted comment]
gdb output attached. i didn't see where the file went so i'll just paste the
file in the comment box:
[root@<HOSTNAME> ~]# memcached -u nobody -d <IP_ADDRESS> -m 1024 -p 11211 -S
[root@<HOSTNAME> ~]# ps -eaf | grep nobody | grep -v grep
nobody 13883 1 0 10:04 ? 00:00:00 memcached -u nobody -d
<IP_ADDRESS> -m 1024 -p 11211 -S
[root@<HOSTNAME> ~]# gdb --pid 13883
GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-23.el5_5.2)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Attaching to process 13883
Reading symbols from /usr/local/bin/memcached...done.
Reading symbols from /usr/lib64/libevent-2.0.so.5...done.
Loaded symbols for /usr/lib64/libevent-2.0.so.5
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /usr/lib64/libsasl2.so.2...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libsasl2.so.2
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols
found)...done.
[Thread debugging using libthread_db enabled]
[New Thread 0x44d64940 (LWP 13888)]
[New Thread 0x44363940 (LWP 13887)]
[New Thread 0x43962940 (LWP 13886)]
[New Thread 0x42f61940 (LWP 13885)]
[New Thread 0x42560940 (LWP 13884)]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Reading symbols from /usr/lib64/sasl2/libplain.so.2...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/sasl2/libplain.so.2
Reading symbols from /usr/lib64/sasl2/libsasl2.so...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/sasl2/libsasl2.so
Reading symbols from /usr/lib64/sasl2/liblogin.so.2...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/sasl2/liblogin.so.2
Reading symbols from /usr/lib64/sasl2/libsasldb.so.2...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/sasl2/libsasldb.so.2
Reading symbols from /usr/lib64/sasl2/libanonymous.so.2...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/sasl2/libanonymous.so.2
0x0000003abc4d4108 in epoll_wait () from /lib64/libc.so.6
(gdb) bt
#0 0x0000003abc4d4108 in epoll_wait () from /lib64/libc.so.6
#1 0x00002b09a6e3d5d8 in epoll_dispatch (base=0x69ae7a0, tv=<value optimized
out>) at epoll.c:407
#2 0x00002b09a6e2c2bc in event_base_loop (base=0x69ae7a0, flags=0) at
event.c:1575
#3 0x00000000004089cc in main (argc=<value optimized out>, argv=<value
optimized out>) at memcached.c:5224
(gdb) thread apply all bt
Thread 6 (Thread 0x42560940 (LWP 13884)):
#0 0x0000003abc4d4108 in epoll_wait () from /lib64/libc.so.6
#1 0x00002b09a6e3d5d8 in epoll_dispatch (base=0x69af220, tv=<value optimized
out>) at epoll.c:407
#2 0x00002b09a6e2c2bc in event_base_loop (base=0x69af220, flags=0) at
event.c:1575
#3 0x000000000040f654 in worker_libevent (arg=0x69b87f0) at thread.c:297
#4 0x0000003abd00673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003abc4d3d1d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x42f61940 (LWP 13885)):
#0 0x0000003abc4d4108 in epoll_wait () from /lib64/libc.so.6
#1 0x00002b09a6e3d5d8 in epoll_dispatch (base=0x69afa80, tv=<value optimized
out>) at epoll.c:407
#2 0x00002b09a6e2c2bc in event_base_loop (base=0x69afa80, flags=0) at
event.c:1575
#3 0x000000000040f654 in worker_libevent (arg=0x69bbb78) at thread.c:297
#4 0x0000003abd00673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003abc4d3d1d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x43962940 (LWP 13886)):
#0 0x0000003abc4d4108 in epoll_wait () from /lib64/libc.so.6
#1 0x00002b09a6e3d5d8 in epoll_dispatch (base=0x69b02c0, tv=<value optimized
out>) at epoll.c:407
#2 0x00002b09a6e2c2bc in event_base_loop (base=0x69b02c0, flags=0) at
event.c:1575
#3 0x000000000040f654 in worker_libevent (arg=0x69bef00) at thread.c:297
#4 0x0000003abd00673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003abc4d3d1d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x44363940 (LWP 13887)):
#0 0x0000003abc4d4108 in epoll_wait () from /lib64/libc.so.6
#1 0x00002b09a6e3d5d8 in epoll_dispatch (base=0x69b0b40, tv=<value optimized
out>) at epoll.c:407
#2 0x00002b09a6e2c2bc in event_base_loop (base=0x69b0b40, flags=0) at
event.c:1575
#3 0x000000000040f654 in worker_libevent (arg=0x69c2288) at thread.c:297
#4 0x0000003abd00673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003abc4d3d1d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x44d64940 (LWP 13888)):
#0 0x0000003abd00aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x000000000040edb0 in assoc_maintenance_thread (arg=<value optimized out>)
at assoc.c:235
#2 0x0000003abd00673d in start_thread () from /lib64/libpthread.so.0
#3 0x0000003abc4d3d1d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x2b09a705fd40 (LWP 13883)):
#0 0x0000003abc4d4108 in epoll_wait () from /lib64/libc.so.6
#1 0x00002b09a6e3d5d8 in epoll_dispatch (base=0x69ae7a0, tv=<value optimized
out>) at epoll.c:407
#2 0x00002b09a6e2c2bc in event_base_loop (base=0x69ae7a0, flags=0) at
event.c:1575
#3 0x00000000004089cc in main (argc=<value optimized out>, argv=<value
optimized out>) at memcached.c:5224
(gdb)
Original comment by douglas....@apollogrp.edu
on 16 Jul 2012 at 5:13
We need the backtrace during a crash, not on a properly functioning server.
Original comment by dsalli...@gmail.com
on 16 Jul 2012 at 5:15
I'm new to this GDB thing so sorry about the wrong stack trace. This link
helped me to figure out how to get a core dump on a process crash:
http://www.randombugs.com/linux/core-dumps-linux.html. Let me know if this is
still not what you are looking for.
[root@<HOSTNAME> dumps]# gdb memcached core.memcached.20237
GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-23.el5_5.2)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/local/bin/memcached...done.
[New Thread 20242]
[New Thread 20241]
[New Thread 20240]
[New Thread 20238]
[New Thread 20237]
Reading symbols from /usr/lib64/libevent-2.0.so.5...done.
Loaded symbols for /usr/lib64/libevent-2.0.so.5
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /usr/lib64/libsasl2.so.2...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libsasl2.so.2
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /usr/lib64/sasl2/libplain.so.2...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/sasl2/libplain.so.2
Reading symbols from /usr/lib64/sasl2/libsasl2.so...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/sasl2/libsasl2.so
Reading symbols from /usr/lib64/sasl2/liblogin.so.2...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/sasl2/liblogin.so.2
Reading symbols from /usr/lib64/sasl2/libsasldb.so.2...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/sasl2/libsasldb.so.2
Reading symbols from /usr/lib64/sasl2/libanonymous.so.2...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/sasl2/libanonymous.so.2
Core was generated by `/usr/local/bin/memcached -d -p 11211 -u nobody -c 640 -m
1024 -S'.
Program terminated with signal 11, Segmentation fault.
#0 0x0000003abc80eea8 in sasl_server_step () from /usr/lib64/libsasl2.so.2
(gdb) bt
#0 0x0000003abc80eea8 in sasl_server_step () from /usr/lib64/libsasl2.so.2
#1 0x0000000000405777 in process_bin_complete_sasl_auth (c=0x1189dd00) at
memcached.c:1716
#2 0x000000000040b897 in drive_machine (fd=<value optimized out>, which=9248,
arg=0x1189dd00) at memcached.c:3869
#3 event_handler (fd=<value optimized out>, which=9248, arg=0x1189dd00) at
memcached.c:4073
#4 0x00002af8682e0679 in event_process_active_single_queue (base=0x117f07f0,
flags=0) at event.c:1325
#5 event_process_active (base=0x117f07f0, flags=0) at event.c:1392
#6 event_base_loop (base=0x117f07f0, flags=0) at event.c:1589
#7 0x000000000040f654 in worker_libevent (arg=0x117fc8e8) at thread.c:297
#8 0x0000003abd00673d in start_thread () from /lib64/libpthread.so.0
#9 0x0000003abc4d3d1d in clone () from /lib64/libc.so.6
(gdb) thread apply all bt
Thread 6 (Thread 20237):
#0 0x0000003abc4d4108 in epoll_wait () from /lib64/libc.so.6
#1 0x00002af8682f15d8 in epoll_dispatch (base=0x117ef510, tv=<value optimized
out>) at epoll.c:407
#2 0x00002af8682e02bc in event_base_loop (base=0x117ef510, flags=0) at
event.c:1575
#3 0x00000000004089cc in main (argc=<value optimized out>, argv=<value
optimized out>) at memcached.c:5224
Thread 5 (Thread 20238):
#0 0x0000003abc4d4108 in epoll_wait () from /lib64/libc.so.6
#1 0x00002af8682f15d8 in epoll_dispatch (base=0x117eff90, tv=<value optimized
out>) at epoll.c:407
#2 0x00002af8682e02bc in event_base_loop (base=0x117eff90, flags=0) at
event.c:1575
#3 0x000000000040f654 in worker_libevent (arg=0x117f9560) at thread.c:297
#4 0x0000003abd00673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003abc4d3d1d in clone () from /lib64/libc.so.6
Thread 4 (Thread 20240):
#0 0x0000003abc4d4108 in epoll_wait () from /lib64/libc.so.6
#1 0x00002af8682f15d8 in epoll_dispatch (base=0x117f1030, tv=<value optimized
out>) at epoll.c:407
#2 0x00002af8682e02bc in event_base_loop (base=0x117f1030, flags=0) at
event.c:1575
#3 0x000000000040f654 in worker_libevent (arg=0x117ffc70) at thread.c:297
#4 0x0000003abd00673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003abc4d3d1d in clone () from /lib64/libc.so.6
Thread 3 (Thread 20241):
#0 0x0000003abc4d4108 in epoll_wait () from /lib64/libc.so.6
#1 0x00002af8682f15d8 in epoll_dispatch (base=0x117f18b0, tv=<value optimized
out>) at epoll.c:407
#2 0x00002af8682e02bc in event_base_loop (base=0x117f18b0, flags=0) at
event.c:1575
#3 0x000000000040f654 in worker_libevent (arg=0x11802ff8) at thread.c:297
#4 0x0000003abd00673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003abc4d3d1d in clone () from /lib64/libc.so.6
Thread 2 (Thread 20242):
#0 0x0000003abd00aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x000000000040edb0 in assoc_maintenance_thread (arg=<value optimized out>)
at assoc.c:235
#2 0x0000003abd00673d in start_thread () from /lib64/libpthread.so.0
#3 0x0000003abc4d3d1d in clone () from /lib64/libc.so.6
Thread 1 (Thread 20239):
#0 0x0000003abc80eea8 in sasl_server_step () from /usr/lib64/libsasl2.so.2
#1 0x0000000000405777 in process_bin_complete_sasl_auth (c=0x1189dd00) at
memcached.c:1716
#2 0x000000000040b897 in drive_machine (fd=<value optimized out>, which=9248,
arg=0x1189dd00) at memcached.c:3869
#3 event_handler (fd=<value optimized out>, which=9248, arg=0x1189dd00) at
memcached.c:4073
#4 0x00002af8682e0679 in event_process_active_single_queue (base=0x117f07f0,
flags=0) at event.c:1325
#5 event_process_active (base=0x117f07f0, flags=0) at event.c:1392
#6 event_base_loop (base=0x117f07f0, flags=0) at event.c:1589
#7 0x000000000040f654 in worker_libevent (arg=0x117fc8e8) at thread.c:297
#8 0x0000003abd00673d in start_thread () from /lib64/libpthread.so.0
#9 0x0000003abc4d3d1d in clone () from /lib64/libc.so.6
Original comment by douglas....@apollogrp.edu
on 16 Jul 2012 at 8:09
Dustin? Any clue?
Original comment by dorma...@rydia.net
on 30 Jul 2012 at 12:08
Almost seems like a library problem. Are you running on the same system you
built on? Any weird (non-package) sasl parts there?
I'll do some fuzzing around it to see if I can induce a crash, but it's a bit
hard to see from here as that's a pretty well tested path.
While I'm doing that, if you can make it happen with -vv on the server, the
output of that might gives us more details.
Original comment by dsalli...@gmail.com
on 30 Jul 2012 at 1:23
[root@<HOST_NAME> ~]# memcached -d -u nobody -m 1024 -p 11211 -S -vv
Initialized SASL.
slab class 1: chunk size 96 perslab 10922
slab class 2: chunk size 120 perslab 8738
slab class 3: chunk size 152 perslab 6898
slab class 4: chunk size 192 perslab 5461
slab class 5: chunk size 240 perslab 4369
slab class 6: chunk size 304 perslab 3449
slab class 7: chunk size 384 perslab 2730
slab class 8: chunk size 480 perslab 2184
slab class 9: chunk size 600 perslab 1747
slab class 10: chunk size 752 perslab 1394
slab class 11: chunk size 944 perslab 1110
slab class 12: chunk size 1184 perslab 885
slab class 13: chunk size 1480 perslab 708
slab class 14: chunk size 1856 perslab 564
slab class 15: chunk size 2320 perslab 451
slab class 16: chunk size 2904 perslab 361
slab class 17: chunk size 3632 perslab 288
slab class 18: chunk size 4544 perslab 230
slab class 19: chunk size 5680 perslab 184
slab class 20: chunk size 7104 perslab 147
slab class 21: chunk size 8880 perslab 118
slab class 22: chunk size 11104 perslab 94
slab class 23: chunk size 13880 perslab 75
slab class 24: chunk size 17352 perslab 60
slab class 25: chunk size 21696 perslab 48
slab class 26: chunk size 27120 perslab 38
slab class 27: chunk size 33904 perslab 30
slab class 28: chunk size 42384 perslab 24
slab class 29: chunk size 52984 perslab 19
slab class 30: chunk size 66232 perslab 15
slab class 31: chunk size 82792 perslab 12
slab class 32: chunk size 103496 perslab 10
slab class 33: chunk size 129376 perslab 8
slab class 34: chunk size 161720 perslab 6
slab class 35: chunk size 202152 perslab 5
slab class 36: chunk size 252696 perslab 4
slab class 37: chunk size 315872 perslab 3
slab class 38: chunk size 394840 perslab 2
slab class 39: chunk size 493552 perslab 2
slab class 40: chunk size 616944 perslab 1
slab class 41: chunk size 771184 perslab 1
slab class 42: chunk size 1048576 perslab 1
[root@<HOST_NAME> ~]# <26 server listening (binary)
<27 server listening (binary)
<28 send buffer was 129024, now 268435456
<28 server listening (udp)
<28 server listening (udp)
<29 send buffer was 129024, now 268435456
<29 server listening (udp)
<29 server listening (udp)
<28 server listening (udp)
<29 server listening (udp)
<28 server listening (udp)
<29 server listening (udp)
[root@<HOST_NAME> ~]# ps -eaf |grep memcached | grep -v grep
nobody 12188 1 0 09:48 ? 00:00:00 memcached -d -u nobody -m 1024
-p 11211 -S -vv
[root@<HOST_NAME> ~]# <30 new binary client connection.
<30 Read binary protocol data:
<30 0x80 0x22 0x00 0x05
<30 0x00 0x00 0x00 0x00
<30 0x00 0x00 0x00 0x19
<30 0x00 0x00 0x00 0x03
<30 0x00 0x00 0x00 0x00
<30 0x00 0x00 0x00 0x00
authenticated() in cmd 0x22 is true
mech: ``PLAIN'' with 20 bytes of data
[root@<HOST_NAME> ~]# ps -eaf |grep memcached | grep -v grep
[root@<HOST_NAME> ~]#
Original comment by douglas....@apollogrp.edu
on 31 Jul 2012 at 4:58
I am running on the same system I built on. There were no SASL libraries
existing when I installed on this box. Here are the steps I performed to
install memcached:
FTP these files to the box as mylogin
put libevent-2.0.16-stable.tar.gz
put memcached-1.4.13.tar.gz
SSH to the box and run the following commands as root
cd ~mylogin
tar -zxvf libevent-2.0.16-stable.tar.gz
cd libevent-2.0.16-stable
./configure --prefix=/usr/local
make
make install
cd ..
ln -s /usr/local/lib/libevent-2.0.so.5 /usr/lib64/libevent-2.0.so.5
yum install cyrus-sasl-devel-2.1.22
tar -zxvf memcached-1.4.13
cd memcached-1.4.13
./configure --enable-sasl
make
make install
saslpasswd2 -a memcached -c memcached
> memcached //Enter password
> memcached //Validate password
chmod 660 /etc/sasldb2
Start memcached with SASL:
memcached -d -u nobody -m 1024 -p 11211 -S
Original comment by douglas....@apollogrp.edu
on 31 Jul 2012 at 5:01
Thanks for all the info. This is definitely golden path stuff, so something
odd is going on.
Original comment by dsalli...@gmail.com
on 31 Jul 2012 at 5:20
Hi,
I'm observing similar segfault on memcached v1.4.15 (memcached 1.4.15-1) from
ArchLinux (stable), unfortunately I don't have any backtraces yet (it's on
production server) but I will try to get as much information as I can.
Original comment by lgromano...@gmail.com
on 19 Sep 2012 at 6:29
closing old stuff.
Original comment by dorma...@rydia.net
on 5 Jul 2015 at 3:48
Original issue reported on code.google.com by
douglas....@apollogrp.edu
on 25 Jun 2012 at 10:32