Closed p5pRT closed 4 years ago
This is a bug report for perl from dom@earth.li\, generated with the help of perlbug 1.40 running under perl 5.24.1.
----------------------------------------------------------------- The test introduced at 25d7b7aa379d33ce2e8fe3e2bef4206b35739bc5 fails in environments where LANG is set to a locale which is not installed and LC_ALL is set to a valid locale. Such an environment is arguably broken\, but arises in a common use case in Debian build tools. This modification takes a more robust approach to modifying the environment.
On Fri\, 07 Jun 2019 10:06:58 GMT\, dom wrote:
This is a bug report for perl from dom@earth.li\, generated with the help of perlbug 1.40 running under perl 5.24.1.
----------------------------------------------------------------- The test introduced at 25d7b7aa379d33ce2e8fe3e2bef4206b35739bc5 fails in environments where LANG is set to a locale which is not installed and LC_ALL is set to a valid locale. Such an environment is arguably broken\, but arises in a common use case in Debian build tools. This modification takes a more robust approach to modifying the environment.
Pushed to blead in commit 69b89a0f0bb2cbb4c1607e78c3b414bf45244bea\, with one committer's edit -- I had to remove a non-printing character in the patch:
#####
#\
Dom\, since I doubt any of our smoke-testing rigs are set up to reproduce this problem\, could you send us some sort of evidence that the problem has been fixed?
Thank you very much. -- James E Keenan (jkeenan@cpan.org)
The RT System itself - Status changed from 'new' to 'open'
On Fri\, Jun 07\, 2019 at 05:11:10AM -0700\, James E Keenan via RT wrote:
Dom\, since I doubt any of our smoke-testing rigs are set up to reproduce this problem\, could you send us some sort of evidence that the problem has been fixed?
Before the patch was applied\, this test failed in my Debian sbuild environment. Afterwards\, it succeeded.
(The relevant detail here is that outside sbuild\, my LANG is en_GB.UTF-8. This is progagated to within sbuild (which sets up a chroot) but that environment does not have the en_GB.UTF-8 locale data installed (since it's supposed to be a minimal environment).
sbuild itself corrects for this problem by setting LC_ALL\, so the fact that the test overrode LC_ALL caused the non-working en_GB.UTF-8 locale to be used.)
Thanks\, Domninic.
Dom\,
Unfortunately I have to call your attention to 2 smoke-test failures in ext/POSIX/t/mb.t which were recorded *after* I applied your patch.
http://perl5.test-smoke.org/report/89146 logs at: http://perl5.test-smoke.org/logfile/89146
http://perl5.test-smoke.org/report/89211 logs at: http://perl5.test-smoke.org/logfile/89211
(These can be tracked via this search: http://perl5.test-smoke.org/submatrix?test=../ext/POSIX/t/mb.t&pversion=5.31.1)
In each case the failures in mb.t occurred when blead was configured as follows:
[stdio] -Dcc=clang -Accflags="-Werror=declaration-after-statement -g -fno-omit-frame-pointer -fsanitize=address -fno-common -fsanitize-blacklist=`pwd`/asan_ignore" -Aldflags="-fsanitize=address"
With and without -DDEBUGGING.
A couple of points:
1. Tester is using what I would guess is an advanced version of the Linux kernel: 5.0.9-200 versus my own 4.15.0-51 (Ubuntu 18.04 LTS). OTOH\, we are getting smoke-test reports from rigs with even higher-numbered Linux kernels.
2. I myself don't understand all those compiler switches the tester is using. In particular\, 'make' fails for me on FreeBSD-11.2 when I use those compiler switches.
3. Nonetheless\, when I build a perl with all those switches (except -DDEBUGGING)\, I get those test same failures. See attachments.
4. When I build blead with those same compiler switches at the commit immediately prior to the one where I applied your patch\, I get a PASS.
##### $ git show | head -1 commit fb55ce6b7596b9e94f941cf83eac5ff84f760ea2 $ cd t;./perl harness -v ../ext/POSIX/t/mb.t; cd -
ok 1 - mblen() basically works ok 2 - MB_CUR_MAX is at least 4 in a UTF-8 locale ok 3 - mblen() recognizes invalid multibyte characters ok 4 - mblen() works on UTF-8 characters ok All tests successful. Files=1\, Tests=4\, 0 wallclock secs ( 0.02 usr 0.00 sys + 0.26 cusr 0.12 csys = 0.40 CPU) Result: PASS #####
So your patch has triggered test failures\, albeit under these very obscure conditions. I'm going to revert your patch from blead and then re-apply it in a branch so that we can continue to gather smoke-test reports.
Thank you very much.
-- James E Keenan (jkeenan@cpan.org)
# Failed test 3 - mblen() recognizes invalid multibyte characters at ../../t/test.pl line 1062 # got "=================================================================\n==1656==ERROR: AddressSanitizer: heap-use-after-free on address 0x6020000006f0 at pc 0x0000004b166d bp 0x7fff1853e910 sp 0x7fff1853e0c0\nREAD of size 2 at 0x6020000006f0 thread T0\n #0 0x4b166c (/home/jkeenan/gitwork/perl/perl+0x4b166c)\n #1 0x6f0758 (/home/jkeenan/gitwork/perl/perl+0x6f0758)\n #2 0x6efb1f (/home/jkeenan/gitwork/perl/perl+0x6efb1f)\n #3 0x81bd6f (/home/jkeenan/gitwork/perl/perl+0x81bd6f)\n #4 0x73e3c9 (/home/jkeenan/gitwork/perl/perl+0x73e3c9)\n #5 0x5a27b1 (/home/jkeenan/gitwork/perl/perl+0x5a27b1)\n #6 0x598b20 (/home/jkeenan/gitwork/perl/perl+0x598b20)\n #7 0x56ca35 (/home/jkeenan/gitwork/perl/perl+0x56ca35)\n #8 0x554aff (/home/jkeenan/gitwork/perl/perl+0x554aff)\n #9 0x558399 (/home/jkeenan/gitwork/perl/perl+0x558399)\n #10 0x6230bd (/home/jkeenan/gitwork/perl/perl+0x6230bd)\n #11 0x823bdb (/home/jkeenan/gitwork/perl/perl+0x823bdb)\n #12 0x81f02c (/home/jkeenan/gitwork/perl/perl+0x81f02c)\n #13 0x73e3c9 (/home/jkeenan/gitwork/perl/perl+0x73e3c9)\n #14 0x5a27b1 (/home/jkeenan/gitwork/perl/perl+0x5a27b1)\n #15 0x598b20 (/home/jkeenan/gitwork/perl/perl+0x598b20)\n #16 0x56ca35 (/home/jkeenan/gitwork/perl/perl+0x56ca35)\n #17 0x554aff (/home/jkeenan/gitwork/perl/perl+0x554aff)\n #18 0x558399 (/home/jkeenan/gitwork/perl/perl+0x558399)\n #19 0x6230bd (/home/jkeenan/gitwork/perl/perl+0x6230bd)\n #20 0x59d88c (/home/jkeenan/gitwork/perl/perl+0x59d88c)\n #21 0x52f0fc (/home/jkeenan/gitwork/perl/perl+0x52f0fc)\n #22 0x7f57db6adb96 (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)\n #23 0x436b69 (/home/jkeenan/gitwork/perl/perl+0x436b69)\n\n0x6020000006f0 is located 0 bytes inside of 8-byte region [0x6020000006f0\,0x6020000006f8)\nfreed by thread T0 here:\n #0 0x4f6850 (/home/jkeenan/gitwork/perl/perl+0x4f6850)\n #1 0x7f57db6b94cf (/lib/x86_64-linux-gnu/libc.so.6+0x2d4cf)\n\npreviously allocated by thread T0 here:\n #0 0x4f6a20 (/home/jkeenan/gitwork/perl/perl+0x4f6a20)\n #1 0x7f57db7299b9 (/lib/x86_64-linux-gnu/libc.so.6+0x9d9b9)\n\nSUMMARY: AddressSanitizer: heap-use-after-free (/home/jkeenan/gitwork/perl/perl+0x4b166c) \nShadow bytes around the buggy address:\n 0x0c047fff8080: fa fa fd fa fa fa fd fa fa fa 00 fa fa fa 00 fa\n 0x0c047fff8090: fa fa fd fa fa fa fd fa fa fa 00 fa fa fa 00 fa\n 0x0c047fff80a0: fa fa fd fa fa fa fd fa fa fa 00 fa fa fa 00 fa\n 0x0c047fff80b0: fa fa fd fa fa fa fd fa fa fa 00 fa fa fa 00 fa\n 0x0c047fff80c0: fa fa fd fa fa fa fd fa fa fa 00 fa fa fa 00 fa\n=>0x0c047fff80d0: fa fa fd fa fa fa fd fa fa fa 00 fa fa fa[fd]fa\n 0x0c047fff80e0: fa fa fd fa fa fa fd fa fa fa 00 fa fa fa 00 fa\n 0x0c047fff80f0: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa\n 0x0c047fff8100: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa\n 0x0c047fff8110: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa\n 0x0c047fff8120: fa fa fd fa fa fa 06 fa fa fa 00 fa fa fa fd fa\nShadow byte legend (one shadow byte represents 8 application bytes):\n Addressable: 00\n Partially addressable: 01 02 03 04 05 06 07 \n Heap left redzone: fa\n Freed heap region: fd\n Stack left redzone: f1\n Stack mid redzone: f2\n Stack right redzone: f3\n Stack after return: f5\n Stack use after scope: f8\n Global redzone: f9\n Global init order: f6\n Poisoned by user: f7\n Container overflow: fc\n Array cookie: ac\n Intra object redzone: bb\n ASan internal: fe\n Left alloca redzone: ca\n Right alloca redzone: cb\n==1656==ABORTING" # expected "-1" # PROG: # use POSIX; print &POSIX::mblen("Ã("\, 2) # STATUS: 256 # Failed test 4 - mblen() works on UTF-8 characters at ../../t/test.pl line 1062 # got "=================================================================\n==1658==ERROR: AddressSanitizer: heap-use-after-free on address 0x6020000006f0 at pc 0x0000004b166d bp 0x7ffe19094070 sp 0x7ffe19093820\nREAD of size 2 at 0x6020000006f0 thread T0\n #0 0x4b166c (/home/jkeenan/gitwork/perl/perl+0x4b166c)\n #1 0x6f0758 (/home/jkeenan/gitwork/perl/perl+0x6f0758)\n #2 0x6efb1f (/home/jkeenan/gitwork/perl/perl+0x6efb1f)\n #3 0x81bd6f (/home/jkeenan/gitwork/perl/perl+0x81bd6f)\n #4 0x73e3c9 (/home/jkeenan/gitwork/perl/perl+0x73e3c9)\n #5 0x5a27b1 (/home/jkeenan/gitwork/perl/perl+0x5a27b1)\n #6 0x598b20 (/home/jkeenan/gitwork/perl/perl+0x598b20)\n #7 0x56ca35 (/home/jkeenan/gitwork/perl/perl+0x56ca35)\n #8 0x554aff (/home/jkeenan/gitwork/perl/perl+0x554aff)\n #9 0x558399 (/home/jkeenan/gitwork/perl/perl+0x558399)\n #10 0x6230bd (/home/jkeenan/gitwork/perl/perl+0x6230bd)\n #11 0x823bdb (/home/jkeenan/gitwork/perl/perl+0x823bdb)\n #12 0x81f02c (/home/jkeenan/gitwork/perl/perl+0x81f02c)\n #13 0x73e3c9 (/home/jkeenan/gitwork/perl/perl+0x73e3c9)\n #14 0x5a27b1 (/home/jkeenan/gitwork/perl/perl+0x5a27b1)\n #15 0x598b20 (/home/jkeenan/gitwork/perl/perl+0x598b20)\n #16 0x56ca35 (/home/jkeenan/gitwork/perl/perl+0x56ca35)\n #17 0x554aff (/home/jkeenan/gitwork/perl/perl+0x554aff)\n #18 0x558399 (/home/jkeenan/gitwork/perl/perl+0x558399)\n #19 0x6230bd (/home/jkeenan/gitwork/perl/perl+0x6230bd)\n #20 0x59d88c (/home/jkeenan/gitwork/perl/perl+0x59d88c)\n #21 0x52f0fc (/home/jkeenan/gitwork/perl/perl+0x52f0fc)\n #22 0x7fdf1d3bfb96 (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)\n #23 0x436b69 (/home/jkeenan/gitwork/perl/perl+0x436b69)\n\n0x6020000006f0 is located 0 bytes inside of 8-byte region [0x6020000006f0\,0x6020000006f8)\nfreed by thread T0 here:\n #0 0x4f6850 (/home/jkeenan/gitwork/perl/perl+0x4f6850)\n #1 0x7fdf1d3cb4cf (/lib/x86_64-linux-gnu/libc.so.6+0x2d4cf)\n\npreviously allocated by thread T0 here:\n #0 0x4f6a20 (/home/jkeenan/gitwork/perl/perl+0x4f6a20)\n #1 0x7fdf1d43b9b9 (/lib/x86_64-linux-gnu/libc.so.6+0x9d9b9)\n\nSUMMARY: AddressSanitizer: heap-use-after-free (/home/jkeenan/gitwork/perl/perl+0x4b166c) \nShadow bytes around the buggy address:\n 0x0c047fff8080: fa fa fd fa fa fa fd fa fa fa 00 fa fa fa 00 fa\n 0x0c047fff8090: fa fa fd fa fa fa fd fa fa fa 00 fa fa fa 00 fa\n 0x0c047fff80a0: fa fa fd fa fa fa fd fa fa fa 00 fa fa fa 00 fa\n 0x0c047fff80b0: fa fa fd fa fa fa fd fa fa fa 00 fa fa fa 00 fa\n 0x0c047fff80c0: fa fa fd fa fa fa fd fa fa fa 00 fa fa fa 00 fa\n=>0x0c047fff80d0: fa fa fd fa fa fa fd fa fa fa 00 fa fa fa[fd]fa\n 0x0c047fff80e0: fa fa fd fa fa fa fd fa fa fa 00 fa fa fa 00 fa\n 0x0c047fff80f0: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa\n 0x0c047fff8100: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa\n 0x0c047fff8110: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa\n 0x0c047fff8120: fa fa fd fa fa fa 06 fa fa fa 00 fa fa fa fd fa\nShadow byte legend (one shadow byte represents 8 application bytes):\n Addressable: 00\n Partially addressable: 01 02 03 04 05 06 07 \n Heap left redzone: fa\n Freed heap region: fd\n Stack left redzone: f1\n Stack mid redzone: f2\n Stack right redzone: f3\n Stack after return: f5\n Stack use after scope: f8\n Global redzone: f9\n Global init order: f6\n Poisoned by user: f7\n Container overflow: fc\n Array cookie: ac\n Intra object redzone: bb\n ASan internal: fe\n Left alloca redzone: ca\n Right alloca redzone: cb\n==1658==ABORTING" # expected "2" # PROG: # use POSIX; print &POSIX::mblen("\N{GREEK SMALL LETTER SIGMA}"\, 2) # STATUS: 256 ../ext/POSIX/t/mb.t .. 1..4 ok 1 - mblen() basically works ok 2 - MB_CUR_MAX is at least 4 in a UTF-8 locale not ok 3 - mblen() recognizes invalid multibyte characters not ok 4 - mblen() works on UTF-8 characters Failed 2/4 subtests
Test Summary Report
../ext/POSIX/t/mb.t (Wstat: 0 Tests: 4 Failed: 2) Failed tests: 3-4 Files=1\, Tests=4\, 0 wallclock secs ( 0.01 usr 0.01 sys + 0.17 cusr 0.07 csys = 0.26 CPU) Result: FAIL
Summary of my perl5 (revision 5 version 31 subversion 1) configuration: Commit id: 7c21f0042bbf5d88b72d07661d64903e627ccf29 Platform: osname=linux osvers=4.15.0-51-generic archname=x86_64-linux uname='linux zareason 4.15.0-51-generic #55-ubuntu smp wed may 15 14:27:21 utc 2019 x86_64 x86_64 x86_64 gnulinux ' config_args='-des -Dusedevel -Dcc=clang -Accflags=-Werror=declaration-after-statement -g -fno-omit-frame-pointer -fsanitize=address -fno-common -fsanitize-blacklist=/home/jkeenan/gitwork/perl/asan_ignore -Aldflags=-fsanitize=address' hint=recommended useposix=true d_sigaction=define useithreads=undef usemultiplicity=undef use64bitint=define use64bitall=define uselongdouble=undef usemymalloc=n default_inc_excludes_dot=define bincompat5005=undef Compiler: cc='clang' ccflags ='-Werror=declaration-after-statement -g -fno-omit-frame-pointer -fsanitize=address -fno-common -fsanitize-blacklist=/home/jkeenan/gitwork/perl/asan_ignore -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2' optimize='-O2' cppflags='-Werror=declaration-after-statement -g -fno-omit-frame-pointer -fsanitize=address -fno-common -fsanitize-blacklist=/home/jkeenan/gitwork/perl/asan_ignore -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include' ccversion='' gccversion='4.2.1 Compatible Clang 6.0.0 (tags/RELEASE_600/final)' gccosandvers='' intsize=4 longsize=8 ptrsize=8 doublesize=8 byteorder=12345678 doublekind=3 d_longlong=define longlongsize=8 d_longdbl=define longdblsize=16 longdblkind=3 ivtype='long' ivsize=8 nvtype='double' nvsize=8 Off_t='off_t' lseeksize=8 alignbytes=8 prototype=define Linker and Libraries: ld='clang' ldflags =' -fsanitize=address -fstack-protector-strong -L/usr/local/lib' libpth=/usr/local/lib /usr/lib/llvm-6.0/lib/clang/6.0.0/lib /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib /lib64 /usr/lib64 libs=-lpthread -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc libc=libc-2.27.so so=so useshrplib=false libperl=libperl.a gnulibc_version='2.27' Dynamic Linking: dlsrc=dl_dlopen.xs dlext=so d_dlsymun=undef ccdlflags='-Wl\,-E' cccdlflags='-fPIC' lddlflags='-shared -O2 -L/usr/local/lib -fstack-protector-strong'
Characteristics of this binary (from libperl): Compile-time options: HAS_TIMES PERLIO_LAYERS PERL_COPY_ON_WRITE PERL_DONT_CREATE_GVSV PERL_MALLOC_WRAP PERL_OP_PARENT PERL_PRESERVE_IVUV PERL_USE_DEVEL USE_64_BIT_ALL USE_64_BIT_INT USE_LARGE_FILES USE_LOCALE USE_LOCALE_COLLATE USE_LOCALE_CTYPE USE_LOCALE_NUMERIC USE_LOCALE_TIME USE_PERLIO USE_PERL_ATOF Built under linux Compiled at Jun 8 2019 17:09:02 %ENV: PERL2DIR="/home/jkeenan/gitwork/perl2" PERLBREW_HOME="/home/jkeenan/.perlbrew" PERLBREW_MANPATH="/home/jkeenan/perl5/perlbrew/perls/perl-5.30.0/man" PERLBREW_PATH="/home/jkeenan/perl5/perlbrew/bin:/home/jkeenan/perl5/perlbrew/perls/perl-5.30.0/bin" PERLBREW_PERL="perl-5.30.0" PERLBREW_ROOT="/home/jkeenan/perl5/perlbrew" PERLBREW_SHELLRC_VERSION="0.84" PERLBREW_VERSION="0.84" PERL_WORKDIR="/home/jkeenan/gitwork/perl" @INC: lib /usr/local/lib/perl5/site_perl/5.31.1/x86_64-linux /usr/local/lib/perl5/site_perl/5.31.1 /usr/local/lib/perl5/5.31.1/x86_64-linux /usr/local/lib/perl5/5.31.1
On Sat\, 08 Jun 2019 21:43:16 GMT\, jkeenan wrote:
Dom\,
Unfortunately I have to call your attention to 2 smoke-test failures in ext/POSIX/t/mb.t which were recorded *after* I applied your patch.
http://perl5.test-smoke.org/report/89146 logs at: http://perl5.test-smoke.org/logfile/89146
http://perl5.test-smoke.org/report/89211 logs at: http://perl5.test-smoke.org/logfile/89211
(These can be tracked via this search: http://perl5.test- smoke.org/submatrix?test=../ext/POSIX/t/mb.t&pversion=5.31.1)
In each case the failures in mb.t occurred when blead was configured as follows:
[stdio] -Dcc=clang -Accflags="-Werror=declaration-after-statement -g -fno-omit-frame-pointer -fsanitize=address -fno-common -fsanitize- blacklist=`pwd`/asan_ignore" -Aldflags="-fsanitize=address"
With and without -DDEBUGGING.
A couple of points:
1. Tester is using what I would guess is an advanced version of the Linux kernel: 5.0.9-200 versus my own 4.15.0-51 (Ubuntu 18.04 LTS). OTOH\, we are getting smoke-test reports from rigs with even higher- numbered Linux kernels.
2. I myself don't understand all those compiler switches the tester is using. In particular\, 'make' fails for me on FreeBSD-11.2 when I use those compiler switches.
3. Nonetheless\, when I build a perl with all those switches (except -DDEBUGGING)\, I get those test same failures. See attachments.
4. When I build blead with those same compiler switches at the commit immediately prior to the one where I applied your patch\, I get a PASS.
##### $ git show | head -1 commit fb55ce6b7596b9e94f941cf83eac5ff84f760ea2 $ cd t;./perl harness -v ../ext/POSIX/t/mb.t; cd -
ok 1 - mblen() basically works ok 2 - MB_CUR_MAX is at least 4 in a UTF-8 locale ok 3 - mblen() recognizes invalid multibyte characters ok 4 - mblen() works on UTF-8 characters ok All tests successful. Files=1\, Tests=4\, 0 wallclock secs ( 0.02 usr 0.00 sys + 0.26 cusr 0.12 csys = 0.40 CPU) Result: PASS #####
So your patch has triggered test failures\, albeit under these very obscure conditions. I'm going to revert your patch from blead and then re-apply it in a branch so that we can continue to gather smoke- test reports.
Thank you very much.
The smoke-test branch is:
smoke-me/jkeenan/dom/134182-mb
-- James E Keenan (jkeenan@cpan.org)
On Sat\, Jun 08\, 2019 at 02:43:16PM -0700\, James E Keenan via RT wrote:
Dom\,
Unfortunately I have to call your attention to 2 smoke-test failures in ext/POSIX/t/mb.t which were recorded *after* I applied your patch.
http://perl5.test-smoke.org/report/89146 logs at: http://perl5.test-smoke.org/logfile/89146
http://perl5.test-smoke.org/report/89211 logs at: http://perl5.test-smoke.org/logfile/89211
(These can be tracked via this search: http://perl5.test-smoke.org/submatrix?test=../ext/POSIX/t/mb.t&pversion=5.31.1)
In each case the failures in mb.t occurred when blead was configured as follows:
[stdio] -Dcc=clang -Accflags="-Werror=declaration-after-statement -g -fno-omit-frame-pointer -fsanitize=address -fno-common -fsanitize-blacklist=`pwd`/asan_ignore" -Aldflags="-fsanitize=address"
With and without -DDEBUGGING.
Very curious\, this looks like the original bug that the test was written for. Niko\, do you understand what's happening here?
Cheers\, Dominic\,
On Fri\, Jun 14\, 2019 at 06:21:16PM +0100\, Dominic Hargreaves wrote:
On Sat\, Jun 08\, 2019 at 02:43:16PM -0700\, James E Keenan via RT wrote:
Unfortunately I have to call your attention to 2 smoke-test failures in ext/POSIX/t/mb.t which were recorded *after* I applied your patch.
In each case the failures in mb.t occurred when blead was configured as follows:
[stdio] -Dcc=clang -Accflags="-Werror=declaration-after-statement -g -fno-omit-frame-pointer -fsanitize=address -fno-common -fsanitize-blacklist=`pwd`/asan_ignore" -Aldflags="-fsanitize=address"
With and without -DDEBUGGING.
Very curious\, this looks like the original bug that the test was written for. Niko\, do you understand what's happening here?
It's a different thing that just happened to get triggered here; this is with non-threaded builds for starters.
I can reproduce it on 5.30.0. It seems to be related to version strings and LC_NUMERIC. I reduced it to this:
==21403==ERROR: AddressSanitizer: heap-use-after-free on address 0x602000000190 at pc 0x0000004813aa bp 0x7fff4f62ea90 sp 0x7fff4f62e230 READ of size 2 at 0x602000000190 thread T0 #0 0x4813a9 in __interceptor_setlocale (/tmp/perl-5.30.0/perl+0x4813a9) #1 0x6d7feb in Perl_upg_version /tmp/perl-5.30.0/./vutil.c:717:17 #2 0x6d73bf in Perl_new_version /tmp/perl-5.30.0/./vutil.c:551:12 #3 0x8019a4 in S_require_version /tmp/perl-5.30.0/pp_ctl.c:3719:10 #4 0x8019a4 in Perl_pp_require /tmp/perl-5.30.0/pp_ctl.c:4345 #5 0x725bf9 in Perl_runops_standard /tmp/perl-5.30.0/run.c:41:26 #6 0x588f71 in S_run_body /tmp/perl-5.30.0/perl.c #7 0x588381 in perl_run /tmp/perl-5.30.0/perl.c:2639:2 #8 0x516e1c in main /tmp/perl-5.30.0/perlmain.c:127:9 #9 0x7f073082a09a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2409a) #10 0x43fc49 in _start (/tmp/perl-5.30.0/perl+0x43fc49)
0x602000000190 is located 0 bytes inside of 8-byte region [0x602000000190\,0x602000000198) freed by thread T0 here: #0 0x4e7712 in __interceptor_free (/tmp/perl-5.30.0/perl+0x4e7712) #1 0x7f0730833963 in setlocale (/lib/x86_64-linux-gnu/libc.so.6+0x2d963)
previously allocated by thread T0 here: #0 0x4e7a93 in malloc (/tmp/perl-5.30.0/perl+0x4e7a93) #1 0x7f073088ddb9 in __strdup (/lib/x86_64-linux-gnu/libc.so.6+0x87db9)
SUMMARY: AddressSanitizer: heap-use-after-free (/tmp/perl-5.30.0/perl+0x4813a9) in __interceptor_setlocale Shadow bytes around the buggy address: 0x0c047fff7fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c047fff7ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c047fff8000: fa fa 01 fa fa fa 00 02 fa fa 00 02 fa fa 00 02 0x0c047fff8010: fa fa 00 02 fa fa 00 02 fa fa 00 02 fa fa 06 fa 0x0c047fff8020: fa fa 00 02 fa fa fd fa fa fa fd fa fa fa 00 fa =>0x0c047fff8030: fa fa[fd]fa fa fa fd fa fa fa fd fa fa fa fd fa 0x0c047fff8040: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa 0x0c047fff8050: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa 0x0c047fff8060: fa fa fd fa fa fa 02 fa fa fa 00 fa fa fa 02 fa 0x0c047fff8070: fa fa fd fa fa fa fd fa fa fa 00 00 fa fa 00 06 0x0c047fff8080: fa fa 00 03 fa fa 00 03 fa fa 00 fa fa fa 00 04 Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb Shadow gap: cc ==21403==ABORTING
-- Niko Tyni ntyni@debian.org
On Mon\, Jun 17\, 2019 at 09:47:04AM +0300\, Niko Tyni wrote:
I can reproduce it on 5.30.0. It seems to be related to version strings and LC_NUMERIC. I reduced it to this:
$ LC_NUMERIC=C.UTF-8 ./perl -l -Ilib -e 'require 5.006;'
And further to this. It's not clear to me if this is a problem with asan or the code.
==17625==ERROR: AddressSanitizer: heap-use-after-free on address 0x602000000070 at pc 0x00000045ea3a bp 0x7ffce1e85f70 sp 0x7ffce1e85710 READ of size 2 at 0x602000000070 thread T0 #0 0x45ea39 in __interceptor_setlocale (/home/ntyni/a.out+0x45ea39) #1 0x4f4327 in main /home/ntyni/t.c:7:5 #2 0x7fd77885209a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2409a) #3 0x41d2d9 in _start (/home/ntyni/a.out+0x41d2d9)
0x602000000070 is located 0 bytes inside of 8-byte region [0x602000000070\,0x602000000078) freed by thread T0 here: #0 0x4c4da2 in __interceptor_free (/home/ntyni/a.out+0x4c4da2) #1 0x7fd77885b963 in setlocale (/lib/x86_64-linux-gnu/libc.so.6+0x2d963)
previously allocated by thread T0 here: #0 0x4c5123 in malloc (/home/ntyni/a.out+0x4c5123) #1 0x7fd7788b5db9 in __strdup (/lib/x86_64-linux-gnu/libc.so.6+0x87db9)
SUMMARY: AddressSanitizer: heap-use-after-free (/home/ntyni/a.out+0x45ea39) in __interceptor_setlocale Shadow bytes around the buggy address: 0x0c047fff7fb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c047fff7fc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c047fff7fd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c047fff7fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c047fff7ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =>0x0c047fff8000: fa fa fd fa fa fa fd fa fa fa 00 fa fa fa[fd]fa 0x0c047fff8010: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c047fff8020: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c047fff8030: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c047fff8040: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c047fff8050: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb Shadow gap: cc ==17625==ABORTING
-- Niko
On Mon\, Jun 17\, 2019 at 02:49:43PM +0300\, Niko Tyni wrote:
On Mon\, Jun 17\, 2019 at 09:47:04AM +0300\, Niko Tyni wrote:
I can reproduce it on 5.30.0. It seems to be related to version strings and LC_NUMERIC. I reduced it to this:
$ LC_NUMERIC=C.UTF-8 ./perl -l -Ilib -e 'require 5.006;'
And further to this. It's not clear to me if this is a problem with asan or the code.
$ cat t.c; clang -g -fsanitize=address t.c; ./a.out #include \<locale.h> int main(void) { char *l; setlocale(LC_NUMERIC\, "C.UTF-8"); l = setlocale(LC_NUMERIC\, NULL); setlocale(LC_NUMERIC\, "C"); setlocale(LC_NUMERIC\, l); }
Presumably the intervening setlocale() call clobbers the buffer that l points to.
The attached patch to vutil.c seems to fix this issue for me\, but eyeballs appreciated of course. -- Niko
On Mon\, Jun 17\, 2019 at 07:20:32PM +0300\, Niko Tyni wrote:
On Mon\, Jun 17\, 2019 at 02:49:43PM +0300\, Niko Tyni wrote:
On Mon\, Jun 17\, 2019 at 09:47:04AM +0300\, Niko Tyni wrote:
#include \<locale.h> int main(void) { char *l; setlocale(LC_NUMERIC\, "C.UTF-8"); l = setlocale(LC_NUMERIC\, NULL); setlocale(LC_NUMERIC\, "C"); setlocale(LC_NUMERIC\, l); }
Presumably the intervening setlocale() call clobbers the buffer that l points to.
The attached patch to vutil.c seems to fix this issue for me\, but eyeballs appreciated of course.
I see vutil.c comes from the version.pm distribution so I've submitted the patch there as
https://github.com/Perl/version.pm/pull/7
-- Niko
On Mon\, 17 Jun 2019 09:21:04 -0700\, ntyni@debian.org wrote:
On Mon\, Jun 17\, 2019 at 02:49:43PM +0300\, Niko Tyni wrote:
On Mon\, Jun 17\, 2019 at 09:47:04AM +0300\, Niko Tyni wrote:
I can reproduce it on 5.30.0. It seems to be related to version strings and LC_NUMERIC. I reduced it to this:
$ LC_NUMERIC=C.UTF-8 ./perl -l -Ilib -e 'require 5.006;'
And further to this. It's not clear to me if this is a problem with asan or the code.
$ cat t.c; clang -g -fsanitize=address t.c; ./a.out #include \<locale.h> int main(void) { char *l; setlocale(LC_NUMERIC\, "C.UTF-8"); l = setlocale(LC_NUMERIC\, NULL); setlocale(LC_NUMERIC\, "C"); setlocale(LC_NUMERIC\, l); }
Presumably the intervening setlocale() call clobbers the buffer that l points to.
The attached patch to vutil.c seems to fix this issue for me\, but eyeballs appreciated of course.
https://rt-archive.perl.org/perl5/Ticket/Display.html?id=134212 has a more complete fix (I didn't see this until I diagnosed it.)
Tony
I believe this ticket can be closed, since the patch referenced above has been applied.
Any disagreement?
Yes, looks good and closeable to me. Thanks!
Uh, taking this back: the original issue @jmdh filed here about mb.t failing on semi-broken locales seems to be still present. @jkeenan reverted the proposed patch after smokers caught the separate issue with memory corruption around vutil.c that's now fixed, but the patch is not reinstated yet afaics.
I rebased the patch, and am smoking it at https://git.io/Jvd6h @jkeenan could you see if it passes things the previous versions failed on; or were those irrelevant to this patch?
I built perl in the smoke-me/khw-mb branch at v5.31.10-26-ge37211489e with these config_args:
./perl -Ilib -V:config_args
config_args='-des -Dusedevel -Dcc=clang -Accflags=-Werror=declaration-after-statement -g -fno-omit-frame-pointer -fsanitize=address -fno-common -fsanitize-blacklist=/home/jkeenan/gitwork/perl/asan_ignore -Aldflags=-fsanitize=address';
I then ran:
$ cd t;./perl harness -v ../ext/POSIX/t/mb.t; cd -
ok 1 - mblen() works on ASCII input
ok 2 - ... and the 2nd parameter is optional
ok 3 - MB_CUR_MAX is at least 4 in a UTF-8 locale
ok 4 - mblen() recognizes invalid multibyte characters
ok 5 - mblen() works on UTF-8 characters
ok 6 - mblen() returns -1 when input length is too short
ok 7 - mbtowc() returns correct length on ASCII input
ok 8 - mbtowc() returns correct ordinal on ASCII input
ok 9 - mbtowc() recognizes invalid multibyte characters
ok 10 - mbtowc() works on UTF-8 characters
ok 11 - mbtowc() returns -1 when input length is too short
ok 12 - wctomb() works on ASCII input
ok 13 - wctomb() works on UTF-8 characters
ok
All tests successful.
Files=1, Tests=13, 1 wallclock secs ( 0.03 usr 0.01 sys + 0.77 cusr 0.20 csys = 1.01 CPU)
Result: PASS
That may resolve the test failures. But I can't be very confident of my results because (a) when I build perl with address sanitizer my computer slows to a halt during make test_harness
, makes it impossible to toggle between programs and forces me to reboot; (b) my Linux kernel is well behind the one that is regularly used with these configure args.
Thank you very much. Jim Keenan
@xsawyerx I would like permission to merge this patch for 5.32 It is very low risk, as it affects just one .t file that didn't even exist in 5.30, and makes life easier for our downstream Debian partners. And it fell through the cracks for months. I have tested that things fail before the patch is applied in the situation it applies to, and pass after it is applied.
Approved!
Migrated from rt.perl.org#134182 (status was 'open')
Searchable as RT134182$