memcached 1.4.24 segfaults

GoogleCodeExporter commented 8 years ago

What steps will reproduce the problem?
1.
SLES 11.3 with slightly patched 3.16 kernel
Linux memcached9 3.16.3-4.1.100-default #1 SMP Thu Sep 18 06:32:16 UTC 2014 
(d2bbe7f) x86_64 x86_64 x86_64 GNU/Linux

64 Gig total memory

memcached 1.4.24, built from source with
CFLAGS="-g3 -O3" + --disable-coverage

2. memcached started as
/usr/sbin/memcached -d -F -m 60000 -k -u nobody -p 11211 -c 65535 -t 12 -o 
lru_maintainer lru_crawler hash_algorithm=murmur3 slab_reassign

3. crashes with
[1565425.309019] traps: memcached[17639] general protection ip:4123a7 
sp:7feade1ece70 error:0 in memcached-1.4.24-33[400000+20000]

$ addr2line -e /usr/sbin/memcached 4123a7
memcached-1.4.24/items.c:1183

we use lru_crawler quite a bit and have custom scripts that use "slabs 
reassign" to move memory between slabs based on slabs stats (i.e. in house 
custom slab reassign).

As an anecdote, 1.4.21 occasionally falls into infinite loop under these 
conditions as well, and we were trying to fix that by upgrading.

Original issue reported on code.google.com by ant...@corp.badoo.com on 2 Jun 2015 at 11:07

GoogleCodeExporter commented 8 years ago

crap, I wish I saw this sooner.

What is your items.c:1183?

It looks like:
        if (it->prev) {
            it->prev->next = it;
        }
^ middle line.

I'd love to know how that's possible. any chance of getting a backtrace?

Original comment by dorma...@rydia.net on 5 Jul 2015 at 12:49

GoogleCodeExporter commented 8 years ago

Yeah, that code.
But no backtrace, sorry. Can't reproduce in a lab, and scary to "test on 
production" :-|

Original comment by ant...@corp.badoo.com on 6 Jul 2015 at 12:21

sianla / memcached

memcached 1.4.24 segfaults #414