search

tarantool / tarantool-qa

QA related issues of Tarantool
3 stars 0 forks source link

`box/tx_man.test.lua`: SIGSEGV in ASAN during shadow memory unmap at exit #324

Open ylobankov opened 1 year ago

ylobankov commented 1 year ago

Tarantool

Tarantool 3.0.0-alpha1-110-g3774e85d2
Target: Linux-x86_64-RelWithDebInfo
Build options: cmake . -DCMAKE_INSTALL_PREFIX=/usr/local -DENABLE_BACKTRACE=TRUE
Compiler: Clang-16.0.6
C_FLAGS: -fexceptions -funwind-tables -fasynchronous-unwind-tables -fno-common -msse2 -fsanitize=fuzzer-no-link -fsanitize=address -fsanitize-blacklist=/tarantool/asan/asan.supp  -fmacro-prefix-map=/tarantool=. -std=c11 -Wall -Wextra -fsanitize=alignment,bool,bounds,builtin,enum,float-cast-overflow,float-divide-by-zero,function,integer-divide-by-zero,return,shift,unreachable,vla-bound -fno-sanitize-recover=alignment,bool,bounds,builtin,enum,float-cast-overflow,float-divide-by-zero,function,integer-divide-by-zero,return,shift,unreachable,vla-bound -Wno-gnu-alignof-expression -Wno-cast-function-type -Werror
CXX_FLAGS: -fexceptions -funwind-tables -fasynchronous-unwind-tables -fno-common -msse2 -fsanitize=fuzzer-no-link -fsanitize=address -fsanitize-blacklist=/tarantool/asan/asan.supp  -fmacro-prefix-map=/tarantool=. -std=c++11 -Wall -Wextra -fsanitize=alignment,bool,bounds,builtin,enum,float-cast-overflow,float-divide-by-zero,function,integer-divide-by-zero,return,shift,unreachable,vla-bound -fno-sanitize-recover=alignment,bool,bounds,builtin,enum,float-cast-overflow,float-divide-by-zero,function,integer-divide-by-zero,return,shift,unreachable,vla-bound -Wno-invalid-offsetof -Wno-gnu-alignof-expression -Wno-cast-function-type -Werror

Steps to reproduce

docker run -it --rm tarantool/testing:ubuntu-jammy-clang16
git clone https://github.com/tarantool/tarantool.git
cd tarantool && git submodule update --recursive --init --jobs $(nproc)
CC=clang-16 CXX=clang++-16 TEST_RUN_EXTRA_PARAMS=box/tx_man.test.lua make -f .test.mk test-release-asan

Expected result: Test passed.

Actual result:

======================================================================================
WORKR TEST                                            PARAMS          RESULT
---------------------------------------------------------------------------------
[001] box/tx_man.test.lua
[001]
[001] [Instance "tx_man" killed by signal: 11 (SIGSEGV)]
[001]
[001] [test-run server "tx_man"] Last 15 lines of the log file /tmp/t/001_box/tx_man.log:
[001]     #2 0x563bc4366465  (/tarantool/src/tarantool+0x13ab465) (BuildId: 8863958fcd235118833ef3cf4bb6990e4337eafb)
[001]     #3 0x563bc435d0b0  (/tarantool/src/tarantool+0x13a20b0) (BuildId: 8863958fcd235118833ef3cf4bb6990e4337eafb)
[001]     #4 0x563bc435ad45  (/tarantool/src/tarantool+0x139fd45) (BuildId: 8863958fcd235118833ef3cf4bb6990e4337eafb)
[001]     #5 0x563bc435a01b  (/tarantool/src/tarantool+0x139f01b) (BuildId: 8863958fcd235118833ef3cf4bb6990e4337eafb)
[001]     #6 0x563bc42e0575  (/tarantool/src/tarantool+0x1325575) (BuildId: 8863958fcd235118833ef3cf4bb6990e4337eafb)
[001]     #7 0x563bc433862c  (/tarantool/src/tarantool+0x137d62c) (BuildId: 8863958fcd235118833ef3cf4bb6990e4337eafb)
[001]     #8 0x563bc42f1239  (/tarantool/src/tarantool+0x1336239) (BuildId: 8863958fcd235118833ef3cf4bb6990e4337eafb)
[001]     #9 0x563bc42e2e33  (/tarantool/src/tarantool+0x1327e33) (BuildId: 8863958fcd235118833ef3cf4bb6990e4337eafb)
[001]     #10 0x563bc4312e42  (/tarantool/src/tarantool+0x1357e42) (BuildId: 8863958fcd235118833ef3cf4bb6990e4337eafb)
[001]     #11 0x563bc43130a7  (/tarantool/src/tarantool+0x13580a7) (BuildId: 8863958fcd235118833ef3cf4bb6990e4337eafb)
[001]     #12 0x563bc3f4d3f4  (/tarantool/src/tarantool+0xf923f4) (BuildId: 8863958fcd235118833ef3cf4bb6990e4337eafb)
[001]     #13 0x563bc3f9f9e2  (/tarantool/src/tarantool+0xfe49e2) (BuildId: 8863958fcd235118833ef3cf4bb6990e4337eafb)
[001]     #14 0x563bc40e18a2  (/tarantool/src/tarantool+0x11268a2) (BuildId: 8863958fcd235118833ef3cf4bb6990e4337eafb)
[001]
[001] SUMMARY: AddressSanitizer: 193262 byte(s) leaked in 546 allocation(s).
[001] [ fail ]
Gumix commented 1 year ago

SIGSEGV is in:

[0] from 0x0000555556622b9d in sighandler_dispatcher
[1] from 0x00007ffff5b30140 in __restore_rt
[2] from 0x0000555555e12397 in __sanitizer::internal_munmap(void*, unsigned long)
[3] from 0x0000555555e13b4c in __sanitizer::UnmapOrDie(void*, unsigned long)
[4] from 0x0000555555e1b23c in __sanitizer::Die()
[5] from 0x0000555555e2ada8 in __lsan::CheckForLeaks()
[6] from 0x0000555555e2a4db in __lsan::DoLeakCheck()
[7] from 0x00007ffff563fac6 in __cxa_finalize
[8] from 0x0000555555d60a03 in __do_global_dtors_aux
[9] from 0x00007fffffffd930

Something goes wrong in the sanitizer during symbolification at exit (errno == 12 means "Out of memory"):

[...]
2023-08-24 18:20:59.544 [984122] main/119/lua txn.c:1504 W> Transaction has been aborted by a fiber yield
2023-08-24 18:21:08.122 [984122] main/118/console/unix/: memtx_tx.c:699 W> Transaction committing DDL (id=33048) has aborted another TX (id=33047)
2023-08-24 18:21:08.135 [984122] main/118/console/unix/: memtx_tx.c:699 W> Transaction committing DDL (id=33056) has aborted another TX (id=33055)
2023-08-24 18:21:08.155 [984122] main C> got signal 15 - Terminated
2023-08-24 18:21:08.155 [984122] main/10143/iproto.shutdown I> tx_binary: stopped
==984122==WARNING: failed to fork (errno 12)
==984122==WARNING: failed to fork (errno 12)
==984122==WARNING: failed to fork (errno 12)
==984122==WARNING: failed to fork (errno 12)
==984122==WARNING: failed to fork (errno 12)
==984122==WARNING: Failed to use and restart external symbolizer!

=================================================================
==984122==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 63168 byte(s) in 329 object(s) allocated from:
    #0 0x5630937964be  (build/src/tarantool+0x8a54be) (BuildId: 7ecc0d812e071c6b3f9d2a5fdf48d54009edfb1a)
    #1 0x56309444dd03  (build/src/tarantool+0x155cd03) (BuildId: 7ecc0d812e071c6b3f9d2a5fdf48d54009edfb1a)
    #2 0x563093e7c47b  (build/src/tarantool+0xf8b47b) (BuildId: 7ecc0d812e071c6b3f9d2a5fdf48d54009edfb1a)
    #3 0x563093e42c73  (build/src/tarantool+0xf51c73) (BuildId: 7ecc0d812e071c6b3f9d2a5fdf48d54009edfb1a)
[...]
SUMMARY: AddressSanitizer: 196630 byte(s) leaked in 569 allocation(s).

BTW, If I change ASAN_OPTIONS=unmap_shadow_on_exit=1 to 0, there are still "failed to fork" errors and unresolved symbols, but the process exits with return code 1 rather than got killed with SIGSEGV.

Gumix commented 1 year ago

It's unclear why ASAN fails to fork a symbolizer with ENOMEM, how it is related to SIGSEGV in AsanDie(), and why only box/tx_man.test.lua fails with this error. Let's disable ASAN for this test.