search

ksh93 / ksh

ksh 93u+m: KornShell lives! | Latest release: https://github.com/ksh93/ksh/releases
Eclipse Public License 2.0
192 stars 32 forks source link

Tracking issue for the various test failures under ASan #518

Open JohnoKing opened 2 years ago

JohnoKing commented 2 years ago

Currently, shtests counts 12 errors under ASan when the regression tests are run with the ASAN_OPTIONS variable set to detect_leaks=0. Below is a regression test log from running the tests under ASan (last updated 2022-09-30):

ASan test results ``` $ ASAN_OPTIONS='detect_leaks=0' bin/shtests -u #### Regression-testing /home/johno/GitRepos/KornShell/ksh/arch/linux.i386-64/bin/ksh #### test alias(C.UTF-8) begins at 2022-09-30+18:28:44 test alias(C.UTF-8) passed at 2022-09-30+18:28:44 [ 47 tests 0 errors ] test append(C.UTF-8) begins at 2022-09-30+18:28:44 test append(C.UTF-8) passed at 2022-09-30+18:28:44 [ 17 tests 0 errors ] test arith(C.UTF-8) begins at 2022-09-30+18:28:44 test arith(C.UTF-8) passed at 2022-09-30+18:28:44 [ 252 tests 0 errors ] test arrays(C.UTF-8) begins at 2022-09-30+18:28:44 test arrays(C.UTF-8) passed at 2022-09-30+18:28:46 [ 170 tests 0 errors ] test arrays2(C.UTF-8) begins at 2022-09-30+18:28:46 test arrays2(C.UTF-8) passed at 2022-09-30+18:28:46 [ 57 tests 0 errors ] test attributes(C.UTF-8) begins at 2022-09-30+18:28:46 test attributes(C.UTF-8) passed at 2022-09-30+18:28:47 [ 167 tests 0 errors ] test basic(C.UTF-8) begins at 2022-09-30+18:28:47 test basic(C.UTF-8) passed at 2022-09-30+18:29:00 [ 147 tests 0 errors ] test bracket(C.UTF-8) begins at 2022-09-30+18:29:00 test bracket(C.UTF-8) passed at 2022-09-30+18:29:01 [ 158 tests 0 errors ] test builtins(C.UTF-8) begins at 2022-09-30+18:29:01 builtins.sh[661]: FAIL: read not terminating when receiving USR1 signal test builtins(C.UTF-8) failed at 2022-09-30+18:29:11 with exit code 1 [ 279 tests 1 error ] test case(C.UTF-8) begins at 2022-09-30+18:29:11 test case(C.UTF-8) passed at 2022-09-30+18:29:11 [ 22 tests 0 errors ] test comvar(C.UTF-8) begins at 2022-09-30+18:29:11 test comvar(C.UTF-8) passed at 2022-09-30+18:29:11 [ 102 tests 0 errors ] test comvario(C.UTF-8) begins at 2022-09-30+18:29:11 test comvario(C.UTF-8) passed at 2022-09-30+18:29:26 [ 73 tests 0 errors ] test coprocess(C.UTF-8) begins at 2022-09-30+18:29:26 coprocess.sh[195]: FAIL: traps when reading from cat coprocess not working coprocess.sh[227]: FAIL: cat coprocess 2 hung /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/tests/coprocess.sh: line 233: coprocess is running; cannot create a new coprocess test coprocess(C.UTF-8) failed at 2022-09-30+18:29:33 with exit code 1 [ 35 tests 1 error ] test cubetype(C.UTF-8) begins at 2022-09-30+18:29:33 test cubetype(C.UTF-8) passed at 2022-09-30+18:29:33 [ 68 tests 0 errors ] test enum(C.UTF-8) begins at 2022-09-30+18:29:33 test enum(C.UTF-8) passed at 2022-09-30+18:29:33 [ 47 tests 0 errors ] test exit(C.UTF-8) begins at 2022-09-30+18:29:33 exit.sh[49]: FAIL: exit in .profile is ignored test exit(C.UTF-8) failed at 2022-09-30+18:29:33 with exit code 1 [ 35 tests 1 error ] test expand(C.UTF-8) begins at 2022-09-30+18:29:33 test expand(C.UTF-8) passed at 2022-09-30+18:29:33 [ 7 tests 0 errors ] test functions(C.UTF-8) begins at 2022-09-30+18:29:33 functions.sh[1024]: FAIL: cannot handle comsub depth > 256 in function test functions(C.UTF-8) failed at 2022-09-30+18:29:36 with exit code 1 [ 131 tests 1 error ] test glob(C.UTF-8) begins at 2022-09-30+18:29:36 test glob(C.UTF-8) passed at 2022-09-30+18:29:36 [ 174 tests 0 errors ] test grep(C.UTF-8) begins at 2022-09-30+18:29:36 test grep(C.UTF-8) passed at 2022-09-30+18:29:36 [ 1 test 0 errors ] test heredoc(C.UTF-8) begins at 2022-09-30+18:29:36 test heredoc(C.UTF-8) passed at 2022-09-30+18:29:37 [ 43 tests 0 errors ] test io(C.UTF-8) begins at 2022-09-30+18:29:38 kill: 607410: no such process io.sh[349]: FAIL: read -n3 from fifo failed -- expected 'a', got 'abc' io.sh[352]: FAIL: read -n1 from fifo failed -- expected 'b', got 'd' test io(C.UTF-8) failed at 2022-09-30+18:29:41 with exit code 2 [ 162 tests 2 errors ] test jobs(C.UTF-8) begins at 2022-09-30+18:29:41 test jobs(C.UTF-8) passed at 2022-09-30+18:29:42 [ 25 tests 0 errors ] test leaks(C.UTF-8) begins at 2022-09-30+18:29:42 leaks.sh[169]: warning: skipping test for known leak "defining associative array in subshell"; export DEBUG=y to test and help us fix it at: https://github.com/ksh93/ksh/issues/94 leaks.sh[354]: warning: skipping test for known leak "set PATH attribute in main shell"; export DEBUG=y to test and help us fix it at: https://github.com/ksh93/ksh/issues/405 leaks.sh[362]: warning: skipping test for known leak "unset PATH in main shell"; export DEBUG=y to test and help us fix it at: https://github.com/ksh93/ksh/issues/405 leaks.sh[375]: warning: skipping test for known leak "set PATH value in subshell"; export DEBUG=y to test and help us fix it at: https://github.com/ksh93/ksh/issues/405 leaks.sh[380]: warning: skipping test for known leak "run command with preceding PATH assignment in subshell"; export DEBUG=y to test and help us fix it at: https://github.com/ksh93/ksh/issues/405 leaks.sh[385]: warning: skipping test for known leak "set PATH attribute in subshell"; export DEBUG=y to test and help us fix it at: https://github.com/ksh93/ksh/issues/405 leaks.sh[390]: warning: skipping test for known leak "unset PATH in subshell"; export DEBUG=y to test and help us fix it at: https://github.com/ksh93/ksh/issues/405 leaks.sh[414]: warning: skipping test for known leak "variable with discipline function in subshell"; export DEBUG=y to test and help us fix it at: https://github.com/ksh93/ksh/issues/404 test leaks(C.UTF-8) passed at 2022-09-30+18:29:54 [ 36 tests 0 errors ] test libcmd(C.UTF-8) begins at 2022-09-30+18:29:54 test libcmd(C.UTF-8) passed at 2022-09-30+18:29:54 [ 136 tests 0 errors ] test math(C.UTF-8) begins at 2022-09-30+18:29:54 test math(C.UTF-8) passed at 2022-09-30+18:30:06 [ 17 tests 0 errors ] test nameref(C.UTF-8) begins at 2022-09-30+18:30:06 nameref.sh[252]: FAIL: nameref x=a[$c] not working for c=[ nameref.sh[252]: FAIL: nameref x=a[$c] not working for c=\ nameref.sh[258]: FAIL: nameref x=$b with b=a[$c] not working for c=[ nameref.sh[258]: FAIL: nameref x=$b with b=a[$c] not working for c=\ test nameref(C.UTF-8) failed at 2022-09-30+18:30:07 with exit code 4 [ 95 tests 4 errors ] test namespace(C.UTF-8) begins at 2022-09-30+18:30:07 test namespace(C.UTF-8) passed at 2022-09-30+18:30:07 [ 23 tests 0 errors ] test options(C.UTF-8) begins at 2022-09-30+18:30:07 test options(C.UTF-8) passed at 2022-09-30+18:30:11 [ 177 tests 0 errors ] test path(C.UTF-8) begins at 2022-09-30+18:30:11 test path(C.UTF-8) passed at 2022-09-30+18:30:30 [ 147 tests 0 errors ] test pointtype(C.UTF-8) begins at 2022-09-30+18:30:30 test pointtype(C.UTF-8) passed at 2022-09-30+18:30:31 [ 36 tests 0 errors ] test posix(C.UTF-8) begins at 2022-09-30+18:30:31 test posix(C.UTF-8) passed at 2022-09-30+18:30:31 [ 61 tests 0 errors ] test pty(C.UTF-8) begins at 2022-09-30+18:30:31 pty.sh[995]: FAIL: suspend a blocked write to a FIFO: line 1008: expected "^\^C.*: testfifo: cannot create \[.*\]\r\n$", got "^C" pty.sh[995]: FAIL: suspend a blocked write to a FIFO: line 1009: read timeout test pty(C.UTF-8) failed at 2022-09-30+18:32:39 with exit code 2 [ 51 tests 2 errors ] test quoting(C.UTF-8) begins at 2022-09-30+18:32:39 test quoting(C.UTF-8) passed at 2022-09-30+18:32:40 [ 96 tests 0 errors ] test quoting2(C.UTF-8) begins at 2022-09-30+18:32:40 test quoting2(C.UTF-8) passed at 2022-09-30+18:32:40 [ 83 tests 0 errors ] test readcsv(C.UTF-8) begins at 2022-09-30+18:32:40 test readcsv(C.UTF-8) passed at 2022-09-30+18:32:40 [ 4 tests 0 errors ] test readonly(C.UTF-8) begins at 2022-09-30+18:32:40 test readonly(C.UTF-8) passed at 2022-09-30+18:32:41 [ 12 tests 0 errors ] test recttype(C.UTF-8) begins at 2022-09-30+18:32:41 test recttype(C.UTF-8) passed at 2022-09-30+18:32:41 [ 8 tests 0 errors ] test restricted(C.UTF-8) begins at 2022-09-30+18:32:41 test restricted(C.UTF-8) passed at 2022-09-30+18:32:41 [ 21 tests 0 errors ] test return(C.UTF-8) begins at 2022-09-30+18:32:41 test return(C.UTF-8) passed at 2022-09-30+18:32:41 [ 47 tests 0 errors ] test select(C.UTF-8) begins at 2022-09-30+18:32:41 test select(C.UTF-8) passed at 2022-09-30+18:32:41 [ 4 tests 0 errors ] test sh_match(C.UTF-8) begins at 2022-09-30+18:32:41 test sh_match(C.UTF-8) passed at 2022-09-30+18:32:42 [ 129 tests 0 errors ] test sigchld(C.UTF-8) begins at 2022-09-30+18:32:42 test sigchld(C.UTF-8) passed at 2022-09-30+18:33:01 [ 14 tests 0 errors ] test signal(C.UTF-8) begins at 2022-09-30+18:33:01 test signal(C.UTF-8) passed at 2022-09-30+18:33:07 [ 55 tests 0 errors ] test statics(C.UTF-8) begins at 2022-09-30+18:33:07 test statics(C.UTF-8) passed at 2022-09-30+18:33:08 [ 7 tests 0 errors ] test subshell(C.UTF-8) begins at 2022-09-30+18:33:08 test subshell(C.UTF-8) passed at 2022-09-30+18:33:41 [ 150 tests 0 errors ] test substring(C.UTF-8) begins at 2022-09-30+18:33:41 test substring(C.UTF-8) passed at 2022-09-30+18:33:43 [ 217 tests 0 errors ] test tilde(C.UTF-8) begins at 2022-09-30+18:33:43 test tilde(C.UTF-8) passed at 2022-09-30+18:33:43 [ 24 tests 0 errors ] test timetype(C.UTF-8) begins at 2022-09-30+18:33:43 test timetype(C.UTF-8) passed at 2022-09-30+18:33:43 [ 18 tests 0 errors ] test treemove(C.UTF-8) begins at 2022-09-30+18:33:43 test treemove(C.UTF-8) passed at 2022-09-30+18:33:43 [ 10 tests 0 errors ] test types(C.UTF-8) begins at 2022-09-30+18:33:43 test types(C.UTF-8) passed at 2022-09-30+18:33:44 [ 100 tests 0 errors ] test variables(C.UTF-8) begins at 2022-09-30+18:33:44 ================================================================= ==628064==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x615000000280 at pc 0x55a2f4c4c5ee bp 0x7ffc2c08c3a0 sp 0x7ffc2c08bb60 READ of size 568 at 0x615000000280 thread T0 #0 0x55a2f4c4c5ed in __asan_memcpy (/home/johno/GitRepos/KornShell/ksh/arch/linux.i386-64/bin/ksh+0x1525ed) #1 0x55a2f4ca5a9d in nv_clone_disc /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/nvdisc.c:676:2 #2 0x55a2f4ca625b in clone_all_disc /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/nvdisc.c:873:10 #3 0x55a2f4ca6a69 in nv_clone /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/nvdisc.c:917:3 #4 0x55a2f4df23d6 in sh_assignok /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/subshell.c:317:2 #5 0x55a2f4d946e3 in nv_putval /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/name.c:1582:3 #6 0x55a2f4d8a092 in nv_open /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/name.c:1511:4 #7 0x55a2f4d86429 in nv_setlist /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/name.c:573:8 #8 0x55a2f4e06051 in sh_exec /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/xec.c:1052:7 #9 0x55a2f4e029fe in sh_eval /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/xec.c:659:3 #10 0x55a2f4e7a2d5 in b_eval /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/bltins/misc.c:210:3 #11 0x55a2f4e09286 in sh_exec /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/xec.c:1263:21 #12 0x55a2f4df601f in sh_subshell /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/subshell.c:653:4 #13 0x55a2f4e0ff5b in sh_exec /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/xec.c:1828:5 #14 0x55a2f4e15b83 in sh_exec /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/xec.c:2312:5 #15 0x55a2f4e15135 in sh_exec /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/xec.c:2240:7 #16 0x55a2f4e12f4d in sh_exec /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/xec.c:2082:5 #17 0x55a2f4c9eb04 in exfile /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/main.c:605:4 #18 0x55a2f4c9ae10 in sh_main /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/main.c:370:2 #19 0x55a2f4c98585 in main /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/pmain.c:42:9 #20 0x7f7545fa228f (/usr/lib/libc.so.6+0x2328f) (BuildId: 26c81e7e05ebaf40bac3523b7d76be0cd71fad82) #21 0x7f7545fa2349 in __libc_start_main (/usr/lib/libc.so.6+0x23349) (BuildId: 26c81e7e05ebaf40bac3523b7d76be0cd71fad82) #22 0x55a2f4b908d4 in _start /build/glibc/src/glibc/csu/../sysdeps/x86_64/start.S:115 0x615000000280 is located 0 bytes to the right of 512-byte region [0x615000000080,0x615000000280) allocated by thread T0 here: #0 0x55a2f4c4ea11 in __interceptor_calloc (/home/johno/GitRepos/KornShell/ksh/arch/linux.i386-64/bin/ksh+0x154a11) #1 0x55a2f4d01b8c in sh_calloc /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/init.c:259:13 #2 0x55a2f4d0f9c7 in stat_init /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/init.c:1801:21 #3 0x55a2f4d0abcc in nv_init /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/init.c:1947:3 #4 0x55a2f4d061d2 in sh_init /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/init.c:1348:20 #5 0x55a2f4c986a0 in sh_main /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/main.c:132:2 #6 0x55a2f4c98585 in main /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/pmain.c:42:9 #7 0x7f7545fa228f (/usr/lib/libc.so.6+0x2328f) (BuildId: 26c81e7e05ebaf40bac3523b7d76be0cd71fad82) SUMMARY: AddressSanitizer: heap-buffer-overflow (/home/johno/GitRepos/KornShell/ksh/arch/linux.i386-64/bin/ksh+0x1525ed) in __asan_memcpy Shadow bytes around the buggy address: 0x0c2a7fff8000: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c2a7fff8010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c2a7fff8020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c2a7fff8030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c2a7fff8040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =>0x0c2a7fff8050:[fa]fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c2a7fff8060: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c2a7fff8070: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c2a7fff8080: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c2a7fff8090: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c2a7fff80a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb ==628064==ABORTING test variables(C.UTF-8) passed at 2022-09-30+18:33:51 [ 198 tests 0 errors ] test vartree1(C.UTF-8) begins at 2022-09-30+18:33:51 test vartree1(C.UTF-8) passed at 2022-09-30+18:33:51 [ 9 tests 0 errors ] test vartree2(C.UTF-8) begins at 2022-09-30+18:33:51 test vartree2(C.UTF-8) passed at 2022-09-30+18:33:51 [ 21 tests 0 errors ] Total errors: 12 CPU time user: system: main: 0m00.159s 0m00.087s tests: 1m17.414s 1m18.307s ```

I should note that the pty tests freeze under ASan, so in order to get those tests to finish the frozen pty process was killed with SIGKILL. Additionally, if ASan's memory leak detection is left enabled, then there are many more test failures due to a multitude of small memory leaks.

McDutchie commented 2 years ago

The buffer overflow in expand.sh is fixed as follows:

expand.sh failure patch v1 ```diff diff --git a/src/cmd/ksh93/sh/lex.c b/src/cmd/ksh93/sh/lex.c index db0cefcff..1be19c596 100644 --- a/src/cmd/ksh93/sh/lex.c +++ b/src/cmd/ksh93/sh/lex.c @@ -1187,7 +1187,7 @@ int sh_lex(Lex_t* lp) if(lp->lex.reservok && state[n]==S_BREAK && isfirst) break; #if SHOPT_BRACEPAT - if(sh_isoption(SH_BRACEEXPAND) && c==LBRACE && !assignment && state[n]!=S_BREAK + if(sh_isoption(SH_BRACEEXPAND) && c==LBRACE && !assignment && n>0 && state[n]!=S_BREAK && !lp->lex.incase && !lp->lex.intest && !lp->lex.skipword) { ```

n may be <0 because the fcgetc(n) macro expansion on line 1178 may fail (e.g. on EOF) and assign -1 to n.

Come to think of it, since the rest of the code really does nothing in that case until the break is reached, a better patch may be:

expand.sh failure patch v2 ```diff diff --git a/src/cmd/ksh93/sh/lex.c b/src/cmd/ksh93/sh/lex.c index db0cefcff..5af15b49d 100644 --- a/src/cmd/ksh93/sh/lex.c +++ b/src/cmd/ksh93/sh/lex.c @@ -1181,7 +1181,7 @@ int sh_lex(Lex_t* lp) break; if(n>0) fcseek(-LEN); - else if(lp->lex.reservok) + else break; /* check for reserved word { or } */ if(lp->lex.reservok && state[n]==S_BREAK && isfirst) ```

No regression test failures are caused by either patch on my end.

edit: The code can be further simplified like this:

expand.sh failure patch v3 ```diff diff --git a/src/cmd/ksh93/sh/lex.c b/src/cmd/ksh93/sh/lex.c index db0cefcff..4d0ad60be 100644 --- a/src/cmd/ksh93/sh/lex.c +++ b/src/cmd/ksh93/sh/lex.c @@ -1175,14 +1175,12 @@ int sh_lex(Lex_t* lp) goto do_reg; } isfirst = (lp->lexd.first&&fcseek(0)==lp->lexd.first+1); - fcgetc(n); + if(fcgetc(n)<=0) + break; /* check for {} */ if(c==LBRACE && n==RBRACE) break; - if(n>0) - fcseek(-LEN); - else if(lp->lex.reservok) - break; + fcseek(-LEN); /* check for reserved word { or } */ if(lp->lex.reservok && state[n]==S_BREAK && isfirst) break; ```
JohnoKing commented 2 years ago

I've investigated the builtins.sh ASan test failure and have found out that it was introduced in commit 9ba2c2e0 / ff385e5a. It's a fairly minor test failure that may be caused by something ASan-specific (perhaps it's related to ASan's signal handling).