Snapchat / KeyDB

A Multithreaded Fork of Redis
https://keydb.dev
BSD 3-Clause "New" or "Revised" License
11.46k stars 579 forks source link

[NEW] Add memory leak checking feature #656

Open CrazyTennisFan opened 1 year ago

CrazyTennisFan commented 1 year ago

If I add the codes in src/t_string.cpp as below:

void incrCommand(client *c) { sds buf = sdsempty(); incrDecrCommand(c,1); }

Then I compile redis using the following commands: make SANITIZER=address When I run the test case with redis 7 and redis latest version, I will get the following memory leak issue report:

*** [err]: Sanitizer error:

==2088887==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 4 byte(s) in 1 object(s) allocated from:

0 0x7f7dcaf2a867 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:145

#1 0x56001c9d86e7 in ztrymalloc_usable_internal /home/ubuntu/opensource_redis/redis/src/zmalloc.c:117
#2 0x56001c9d86e7 in zmalloc_usable /home/ubuntu/opensource_redis/redis/src/zmalloc.c:160
#3 0x56001cd5bd4c in _sdsnewlen.constprop.0 /home/ubuntu/opensource_redis/redis/src/sds.c:117
#4 0x56001ca9ff6e in sdsnewlen /home/ubuntu/opensource_redis/redis/src/sds.c:169
#5 0x56001ca9ff6e in sdsempty /home/ubuntu/opensource_redis/redis/src/sds.c:179
#6 0x56001ca9ff6e in incrCommand /home/ubuntu/opensource_redis/redis/src/t_string.c:633
#7 0x56001c9b6aa2 in call /home/ubuntu/opensource_redis/redis/src/server.c:3486
#8 0x56001c9ba2b7 in processCommand /home/ubuntu/opensource_redis/redis/src/server.c:4115
#9 0x56001ca1b7cf in processCommandAndResetClient /home/ubuntu/opensource_redis/redis/src/networking.c:2449
#10 0x56001ca1b7cf in processInputBuffer /home/ubuntu/opensource_redis/redis/src/networking.c:2557
#11 0x56001ca1c933 in readQueryFromClient /home/ubuntu/opensource_redis/redis/src/networking.c:2696
#12 0x56001ccfc27c in callHandler /home/ubuntu/opensource_redis/redis/src/connhelpers.h:79
#13 0x56001ccfc27c in connSocketEventHandler /home/ubuntu/opensource_redis/redis/src/socket.c:297
#14 0x56001c999c71 in aeProcessEvents /home/ubuntu/opensource_redis/redis/src/ae.c:436
#15 0x56001c999c71 in aeMain /home/ubuntu/opensource_redis/redis/src/ae.c:496
#16 0x56001c97a565 in main /home/ubuntu/opensource_redis/redis/src/server.c:7292
#17 0x7f7dcab90d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f)

SUMMARY: AddressSanitizer: 4 byte(s) leaked in 1 allocation(s).

*** [err]: Sanitizer error:

==2089116==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 4 byte(s) in 1 object(s) allocated from:

0 0x7f6bf3d7a867 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:145

#1 0x55afc3fd86e7 in ztrymalloc_usable_internal /home/ubuntu/opensource_redis/redis/src/zmalloc.c:117
#2 0x55afc3fd86e7 in zmalloc_usable /home/ubuntu/opensource_redis/redis/src/zmalloc.c:160
#3 0x55afc435bd4c in _sdsnewlen.constprop.0 /home/ubuntu/opensource_redis/redis/src/sds.c:117
#4 0x55afc409ff6e in sdsnewlen /home/ubuntu/opensource_redis/redis/src/sds.c:169
#5 0x55afc409ff6e in sdsempty /home/ubuntu/opensource_redis/redis/src/sds.c:179
#6 0x55afc409ff6e in incrCommand /home/ubuntu/opensource_redis/redis/src/t_string.c:633
#7 0x55afc3fb6aa2 in call /home/ubuntu/opensource_redis/redis/src/server.c:3486
#8 0x55afc3fba2b7 in processCommand /home/ubuntu/opensource_redis/redis/src/server.c:4115
#9 0x55afc401b7cf in processCommandAndResetClient /home/ubuntu/opensource_redis/redis/src/networking.c:2449
#10 0x55afc401b7cf in processInputBuffer /home/ubuntu/opensource_redis/redis/src/networking.c:2557
#11 0x55afc401c933 in readQueryFromClient /home/ubuntu/opensource_redis/redis/src/networking.c:2696
#12 0x55afc42fc27c in callHandler /home/ubuntu/opensource_redis/redis/src/connhelpers.h:79
#13 0x55afc42fc27c in connSocketEventHandler /home/ubuntu/opensource_redis/redis/src/socket.c:297
#14 0x55afc3f99c71 in aeProcessEvents /home/ubuntu/opensource_redis/redis/src/ae.c:436
#15 0x55afc3f99c71 in aeMain /home/ubuntu/opensource_redis/redis/src/ae.c:496
#16 0x55afc3f7a565 in main /home/ubuntu/opensource_redis/redis/src/server.c:7292
#17 0x7f6bf39e0d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f)

SUMMARY: AddressSanitizer: 4 byte(s) leaked in 1 allocation(s).

*** [err]: Sanitizer error:

==2088636==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 20 byte(s) in 5 object(s) allocated from:

0 0x7f5ae13f0867 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:145

#1 0x56205afac6e7 in ztrymalloc_usable_internal /home/ubuntu/opensource_redis/redis/src/zmalloc.c:117
#2 0x56205afac6e7 in zmalloc_usable /home/ubuntu/opensource_redis/redis/src/zmalloc.c:160
#3 0x56205b32fd4c in _sdsnewlen.constprop.0 /home/ubuntu/opensource_redis/redis/src/sds.c:117
#4 0x56205b073f6e in sdsnewlen /home/ubuntu/opensource_redis/redis/src/sds.c:169
#5 0x56205b073f6e in sdsempty /home/ubuntu/opensource_redis/redis/src/sds.c:179
#6 0x56205b073f6e in incrCommand /home/ubuntu/opensource_redis/redis/src/t_string.c:633
#7 0x56205af8aaa2 in call /home/ubuntu/opensource_redis/redis/src/server.c:3486
#8 0x56205af8e2b7 in processCommand /home/ubuntu/opensource_redis/redis/src/server.c:4115
#9 0x56205afef7cf in processCommandAndResetClient /home/ubuntu/opensource_redis/redis/src/networking.c:2449
#10 0x56205afef7cf in processInputBuffer /home/ubuntu/opensource_redis/redis/src/networking.c:2557
#11 0x56205aff0933 in readQueryFromClient /home/ubuntu/opensource_redis/redis/src/networking.c:2696
#12 0x56205b2d027c in callHandler /home/ubuntu/opensource_redis/redis/src/connhelpers.h:79
#13 0x56205b2d027c in connSocketEventHandler /home/ubuntu/opensource_redis/redis/src/socket.c:297
#14 0x56205af6dc71 in aeProcessEvents /home/ubuntu/opensource_redis/redis/src/ae.c:436
#15 0x56205af6dc71 in aeMain /home/ubuntu/opensource_redis/redis/src/ae.c:496
#16 0x56205af4e565 in main /home/ubuntu/opensource_redis/redis/src/server.c:7292
#17 0x7f5ae1056d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f)

SUMMARY: AddressSanitizer: 20 byte(s) leaked in 5 allocation(s).

However, when I compile the keydb with SANITIZE=address option as following commands:

make BUILD_TLS=yes NO_MOTD=yes NO_LICENSE_CHECK=yes ENABLE_FLASH=yes SANITIZE=address

and I run the test case, there is no any memory leak issue report

CrazyTennisFan commented 1 year ago

The PR for redis memory leak check is https://github.com/redis/redis/pull/9601

JohnSully commented 1 year ago

Confirmed repro

JohnSully commented 1 year ago
cd src && make all
make[1]: Entering directory '/home/jsully/keydb/src'
printf '    %b %b\n' "\033[34m"CC"\033[0m" "\033[33m"t_string.o"\033[0m";g++ -std=c++17 -pedantic -fno-rtti -D__STDC_FORMAT_MACROS   -Wall -W -Wno-missing-field-initializers -Wno-address-of-packed-member -Wno-atomic-alignment -O0 -g -ggdb -fsanitize=address -DSANITIZE -fno-omit-frame-pointer   -DMOTD -I../deps/hiredis -I../deps/linenoise -I../deps/lua/src -I../deps/hdr_histogram -I../deps/rocksdb/include/ -I../deps/concurrentqueue -I../deps/cpp-statsd-client/include/ -DUSE_OPENSSL  -MMD -o t_string.o -c t_string.cpp
printf '    %b %b\n' "\033[34;1m"LINK"\033[0m" "\033[37;1m"keydb-server"\033[0m";g++ -fsanitize=address  -g -ggdb -rdynamic  -o keydb-server adlist.o quicklist.o ae.o anet.o dict.o server.o sds.o zmalloc.o lzf_c.o lzf_d.o pqsort.o zipmap.o sha1.o ziplist.o release.o networking.o util.o object.o db.o replication.o rdb.o t_string.o t_list.o t_set.o t_zset.o t_hash.o t_nhash.o config.o aof.o pubsub.o multi.o debug.o sort.o intset.o syncio.o cluster.o crc16.o endianconv.o slowlog.o scripting.o bio.o rio.o rand.o memtest.o crcspeed.o crc64.o bitops.o sentinel.o notify.o setproctitle.o blocked.o hyperloglog.o latency.o sparkline.o redis-check-rdb.o redis-check-aof.o geo.o lazyfree.o module.o evict.o expire.o geohash.o geohash_helper.o childinfo.o defrag.o siphash.o rax.o t_stream.o listpack.o localtime.o acl.o storage.o rdb-s3.o fastlock.o new.o tracking.o cron.o connection.o tls.o sha256.o motd_server.o timeout.o setcpuaffinity.o AsyncWorkQueue.o snapshot.o storage/teststorageprovider.o keydbutils.o StorageCache.o monotonic.o cli_common.o mt19937-64.o SnapshotPayloadParseState.o ../deps/hiredis/libhiredis.a ../deps/lua/src/liblua.a -lm -lz -lcrypto -latomic -ldl -pthread -lrt -luuid -lcurl ../deps/hiredis/libhiredis_ssl.a -lssl -lcrypto
printf '    %b %b\n' "\033[34;1m"INSTALL"\033[0m" "\033[37;1m"keydb-sentinel"\033[0m";install keydb-server keydb-sentinel
printf '    %b %b\n' "\033[34;1m"INSTALL"\033[0m" "\033[37;1m"keydb-check-rdb"\033[0m";install keydb-server keydb-check-rdb
printf '    %b %b\n' "\033[34;1m"INSTALL"\033[0m" "\033[37;1m"keydb-check-aof"\033[0m";install keydb-server keydb-check-aof
echo ""
echo "Hint: It's a good idea to run 'make test' ;)"
echo ""
make[1]: Leaving directory '/home/jsully/keydb/src'

Ran make-n and confirmed we're sending fsanitize=address so something else must be off