Fix test failure in POSIX/t/mb.t with semi-broken locales

[p5pRT]

Migrated from rt.perl.org#134182 (status was 'open')

Searchable as RT134182$

[p5pRT]

From @jmdh

Created by @jmdh

This is a bug report for perl from dom@​earth.li\, generated with the help of perlbug 1.40 running under perl 5.24.1.

----------------------------------------------------------------- The test introduced at 25d7b7aa379d33ce2e8fe3e2bef4206b35739bc5 fails in environments where LANG is set to a locale which is not installed and LC_ALL is set to a valid locale. Such an environment is arguably broken\, but arises in a common use case in Debian build tools. This modification takes a more robust approach to modifying the environment.

Perl Info

``` Flags: category=library severity=low module=POSIX Site configuration information for perl 5.24.1: Configured by Debian Project at Thu Nov 29 11:11:57 UTC 2018. Summary of my perl5 (revision 5 version 24 subversion 1) configuration: Platform: osname=linux, osvers=3.16.0, archname=x86_64-linux-gnu-thread-multi uname='linux localhost 3.16.0 #1 smp debian 3.16.0 x86_64 gnulinux ' config_args='-Dusethreads -Duselargefiles -Dcc=x86_64-linux-gnu-gcc -Dcpp=x86_64-linux-gnu-cpp -Dld=x86_64-linux-gnu-gcc -Dccflags=-DDEBIAN -Wdate-time -D_FORTIFY_SOURCE=2 -g -O2 -fdebug-prefix-map=/build/perl-CWhbRh/perl-5.24.1=. -fstack-protector-strong -Wformat -Werror=format-security -Dldflags= -Wl,-z,relro -Dlddlflags=-shared -Wl,-z,relro -Dcccdlflags=-fPIC -Darchname=x86_64-linux-gnu -Dprefix=/usr -Dprivlib=/usr/share/perl/5.24 -Darchlib=/usr/lib/x86_64-linux-gnu/perl/5.24 -Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5 -Dvendorarch=/usr/lib/x86_64-linux-gnu/perl5/5.24 -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl/5.24.1 -Dsitearch=/usr/local/lib/x86_64-linux-gnu/perl/5.24.1 -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3 -Dsiteman1dir=/usr/local/man/man1 -Dsiteman3dir=/usr/local/man/man3 -Dusesitecustomize -Duse64bitint -Dman1ext=1 -Dman3ext=3perl -Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Ud_ualarm -Uusesfio -Uusenm -Ui_libutil -Uversiononly -DDEBUGGING=-g -Doptimize=-O2 -dEs -Duseshrplib -Dlibperl=libperl.so.5.24.1' hint=recommended, useposix=true, d_sigaction=define useithreads=define, usemultiplicity=define use64bitint=define, use64bitall=define, uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='x86_64-linux-gnu-gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fwrapv -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64', optimize='-O2 -g', cppflags='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fwrapv -fno-strict-aliasing -pipe -I/usr/local/include' ccversion='', gccversion='6.3.0 20170516', gccosandvers='' intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678, doublekind=3 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16, longdblkind=3 ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8 alignbytes=8, prototype=define Linker and Libraries: ld='x86_64-linux-gnu-gcc', ldflags =' -fstack-protector-strong -L/usr/local/lib' libpth=/usr/local/lib /usr/lib/gcc/x86_64-linux-gnu/6/include-fixed /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt perllibs=-ldl -lm -lpthread -lc -lcrypt libc=libc-2.24.so, so=so, useshrplib=true, libperl=libperl.so.5.24 gnulibc_version='2.24' Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E' cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib -fstack-protector-strong' Locally applied patches: DEBPKG:debian/cpan_definstalldirs - Provide a sensible INSTALLDIRS default for modules installed from CPAN. DEBPKG:debian/db_file_ver - https://bugs.debian.org/340047 Remove overly restrictive DB_File version check. DEBPKG:debian/doc_info - Replace generic man(1) instructions with Debian-specific information. DEBPKG:debian/enc2xs_inc - https://bugs.debian.org/290336 Tweak enc2xs to follow symlinks and ignore missing @INC directories. DEBPKG:debian/errno_ver - https://bugs.debian.org/343351 Remove Errno version check due to upgrade problems with long-running processes. DEBPKG:debian/libperl_embed_doc - https://bugs.debian.org/186778 Note that libperl-dev package is required for embedded linking DEBPKG:fixes/respect_umask - Respect umask during installation DEBPKG:debian/writable_site_dirs - Set umask approproately for site install directories DEBPKG:debian/extutils_set_libperl_path - EU:MM: set location of libperl.a under /usr/lib DEBPKG:debian/no_packlist_perllocal - Don't install .packlist or perllocal.pod for perl or vendor DEBPKG:debian/fakeroot - Postpone LD_LIBRARY_PATH evaluation to the binary targets. DEBPKG:debian/instmodsh_doc - Debian policy doesn't install .packlist files for core or vendor. DEBPKG:debian/ld_run_path - Remove standard libs from LD_RUN_PATH as per Debian policy. DEBPKG:debian/libnet_config_path - Set location of libnet.cfg to /etc/perl/Net as /usr may not be writable. DEBPKG:debian/mod_paths - Tweak @INC ordering for Debian DEBPKG:debian/prune_libs - https://bugs.debian.org/128355 Prune the list of libraries wanted to what we actually need. DEBPKG:fixes/net_smtp_docs - [rt.cpan.org #36038] https://bugs.debian.org/100195 Document the Net::SMTP 'Port' option DEBPKG:debian/perlivp - https://bugs.debian.org/510895 Make perlivp skip include directories in /usr/local DEBPKG:debian/deprecate-with-apt - https://bugs.debian.org/747628 Point users to Debian packages of deprecated core modules DEBPKG:debian/squelch-locale-warnings - https://bugs.debian.org/508764 Squelch locale warnings in Debian package maintainer scripts DEBPKG:debian/skip-upstream-git-tests - Skip tests specific to the upstream Git repository DEBPKG:debian/patchlevel - https://bugs.debian.org/567489 List packaged patches for 5.24.1-3+deb9u5 in patchlevel.h DEBPKG:debian/skip-kfreebsd-crash - https://bugs.debian.org/628493 [perl #96272] Skip a crashing test case in t/op/threads.t on GNU/kFreeBSD DEBPKG:fixes/document_makemaker_ccflags - https://bugs.debian.org/628522 [rt.cpan.org #68613] Document that CCFLAGS should include $Config{ccflags} DEBPKG:debian/find_html2text - https://bugs.debian.org/640479 Configure CPAN::Distribution with correct name of html2text DEBPKG:debian/perl5db-x-terminal-emulator.patch - https://bugs.debian.org/668490 Invoke x-terminal-emulator rather than xterm in perl5db.pl DEBPKG:debian/cpan-missing-site-dirs - https://bugs.debian.org/688842 Fix CPAN::FirstTime defaults with nonexisting site dirs if a parent is writable DEBPKG:fixes/memoize_storable_nstore - [rt.cpan.org #77790] https://bugs.debian.org/587650 Memoize::Storable: respect 'nstore' option not respected DEBPKG:debian/regen-skip - Skip a regeneration check in unrelated git repositories DEBPKG:debian/makemaker-pasthru - https://bugs.debian.org/758471 Pass LD settings through to subdirectories DEBPKG:debian/makemaker-manext - https://bugs.debian.org/247370 Make EU::MakeMaker honour MANnEXT settings in generated manpage headers DEBPKG:debian/devel-ppport-reproducibility - https://bugs.debian.org/801523 Sort the list of XS code files when generating RealPPPort.xs DEBPKG:debian/encode-unicode-bom-doc - https://bugs.debian.org/798727 Document Debian backport of Encode::Unicode fix DEBPKG:debian/kfreebsd-softupdates - https://bugs.debian.org/796798 Work around Debian Bug#796798 DEBPKG:fixes/autodie-scope - https://bugs.debian.org/798096 Fix a scoping issue with "no autodie" and the "system" sub DEBPKG:fixes/crosscompile-no-targethost - [23695c0] [perl #127234] Fix the Configure escape with usecrosscompile but no targethost DEBPKG:fixes/memoize-pod - [rt.cpan.org #89441] Fix POD errors in Memoize DEBPKG:fixes/ok-pod - Added encoding for pod. DEBPKG:debian/hurd-softupdates - https://bugs.debian.org/822735 Fix t/op/stat.t failures on hurd DEBPKG:fixes/nntp_docs - https://bugs.debian.org/51962 Net::NNTP: Correct innd/nnrpd confusion in relation to Reader option DEBPKG:fixes/math_complex_doc_great_circle - https://bugs.debian.org/697567 [rt.cpan.org #114104] Math::Trig: clarify definition of great_circle_midpoint DEBPKG:fixes/math_complex_doc_see_also - https://bugs.debian.org/697568 [rt.cpan.org #114105] Math::Trig: add missing SEE ALSO DEBPKG:fixes/math_complex_doc_angle_units - https://bugs.debian.org/731505 [rt.cpan.org #114106] Math::Trig: document angle units DEBPKG:fixes/cpan_web_link - https://bugs.debian.org/367291 CPAN: Add link to main CPAN web site DEBPKG:fixes/time_piece_doc - https://bugs.debian.org/817925 Time::Piece: Improve documentation for add_months and add_years DEBPKG:fixes/perlbug-refactor - https://bugs.debian.org/822463 [perl #128020] perlbug: Refactor duplicated file reading code DEBPKG:fixes/perlbug-linewrap - https://bugs.debian.org/822463 [perl #128020] perlbug: wrap overly long lines DEBPKG:fixes/hurd_sigaction - https://bugs.debian.org/825016 [d54f4ed] ext/POSIX/t/sigaction.t: Skip uid and pid tests on GNU/Hurd DEBPKG:fixes/hurd_hints - [4694301] https://bugs.debian.org/825020 [perl #128279] Modify hints for Hurd per Debian ticket 825020. DEBPKG:fixes/extutils-parsexs-reproducibility - [perl #128517] https://bugs.debian.org/829296 Make the output of ExtUtils::ParseXS reproducible DEBPKG:debian/CVE-2016-1238/sitecustomize-in-etc - Look for sitecustomize.pl in /etc/perl rather than sitelib on Debian systems DEBPKG:debian/CVE-2016-1238/test-suite-without-dot - [perl #127810] Patch unit tests to explicitly insert "." into @INC when needed. DEBPKG:debian/CVE-2016-1238/eumm-without-dot - [perl #127810] Add PERL_USE_UNSAFE_INC support to EU::MM for fortify_inc support. DEBPKG:debian/CVE-2016-1238/cpan-without-dot - [perl #127810] Set PERL_USE_UNSAFE_INC for cpan usage DEBPKG:debian/document_inc_removal - Document in perlvar that we remove '.' from @INC by default DEBPKG:fixes/extutils_makemaker_reproducible - https://bugs.debian.org/835815 https://bugs.debian.org/834190 Make perllocal.pod files reproducible DEBPKG:debian/CVE-2016-1238/remove-inc-test - Remove test for '.' in @INC as it might not be DEBPKG:fixes/file_path_hurd_errno - File-Path: Fix test failure in Hurd due to hard-coded ENOENT DEBPKG:debian/hppa_op_optimize_workaround - https://bugs.debian.org/838613 Temporarily lower the optimization of op.c on hppa due to gcc-6 problems DEBPKG:fixes/test-builder-warning - https://bugs.debian.org/840968 Silence a 'used only once' warning in Test::Builder DEBPKG:fixes/longdblinf-randomness - [dd68853] [perl #130133] https://bugs.debian.org/844752 Configure: fix garbage filtering with 80-bit long doubles DEBPKG:debian/installman-utf8 - https://bugs.debian.org/840211 Generate man pages with UTF-8 characters DEBPKG:fixes/list_assign_leak - [1050723] [perl #130766] https://bugs.debian.org/855064 avoid a leak in list assign from/to magic values DEBPKG:fixes/perlfunc_inc_doc - [a03e9f8] https://bugs.debian.org/839536 [perl #130832] Documentation fixes for '.' possibly no longer being in @INC DEBPKG:fixes/file_path_chmod_race - https://bugs.debian.org/863870 [rt.cpan.org #121951] Prevent directory chmod race attack. DEBPKG:fixes/extutils_file_path_compat - Correct the order of tests of chmod(). (#294) DEBPKG:debian/customized - Update customized.dat for files patched in Debian DEBPKG:fixes/getopt-long-1 - https://bugs.debian.org/855532 [rt.cpan.org #114999] Fix bug RT#114999 DEBPKG:fixes/getopt-long-2 - [rt.cpan.org #120300] Withdraw part of commit 5d9947fb445327c7299d8beb009d609bc70066c0, which tries to implement more GNU getopt_long campatibility. GNU DEBPKG:fixes/getopt-long-3 - provide a default value for optional arguments DEBPKG:fixes/getopt-long-4 - https://bugs.debian.org/864544 [rt.cpan.org #122068] Fix issue #122068. DEBPKG:fixes/fbm-instr-crash - [bb152a4] [perl #131575] https://bugs.debian.org/864782 don't call Perl_fbm_instr() with negative length DEBPKG:debian/CVE-2016-1238/base-pm-amends-pt2 - [1afa289] Limit dotless-INC effect on base.pm with guard: DEBPKG:fixes/CVE-2017-12837 - https://bugs.debian.org/875596 [perl #131582] [f7e5417] regcomp [perl #131582] DEBPKG:fixes/CVE-2017-12883 - https://bugs.debian.org/875597 [perl #131598] [40b3cda] PATCH: [perl #131598] DEBPKG:fixes/CVE-2018-6797 - [perl #132227] (perl #132227) restart a node if we change to uni rules within the node and encounter a sharp S DEBPKG:fixes/CVE-2018-6798/pt1 - [perl #132063] Heap buffer overflow DEBPKG:fixes/CVE-2018-6798/pt2 - [perl #132063] v5.24.3: fix TRIE_READ_CHAR and DECL_TRIE_TYPE to account for non-utf8 target DEBPKG:fixes/CVE-2018-6798/pt3 - [perl #132063] (perl #132063) we should no longer warn for this code DEBPKG:fixes/CVE-2018-6913 - [perl #131844] (perl #131844) fix various space calculation issues in pp_pack.c DEBPKG:fixes/CVE-2018-12015-Archive-Tar-directory-traversal - https://bugs.debian.org/900834 [rt.cpan.org #125523] Remove existing files before overwriting them DEBPKG:fixes/CVE-2018-18311 - Perl_my_setenv(); handle integer wrap DEBPKG:fixes/CVE-2018-18312 - for 5.26 maint DEBPKG:fixes/CVE-2018-18313 - regcomp.c: Convert some strchr to memchr DEBPKG:fixes/CVE-2018-18314 - fix #131649 - extended charclass can trigger assert @INC for perl 5.24.1: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.24.1 /usr/local/share/perl/5.24.1 /usr/lib/x86_64-linux-gnu/perl5/5.24 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.24 /usr/share/perl/5.24 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base Environment for perl 5.24.1: HOME=/home/dom LANG=en_GB.UTF-8 LANGUAGE=en_GB:en LD_LIBRARY_PATH (unset) LOGDIR (unset) PATH=/home/dom/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/sbin:/usr/sbin PERL_BADLANG (unset) SHELL=/bin/bash ```

[p5pRT]

From @jmdh

0001-Fix-edge-case-test-failure-in-ext-POSIX-t-mb.t.patch

```diff From ba80ce1f59e6aa82532d84627b8c5d094eeda1a4 Mon Sep 17 00:00:00 2001 From: Dominic Hargreaves Date: Fri, 7 Jun 2019 10:04:26 +0100 Subject: [PATCH] Fix edge case test failure in ext/POSIX/t/mb.t This new test fails in an environment where LANG is set to one thing and LC_ALL is set to another, and where LANG is set to a locale which is not installed in the environment in question. Such a test environment is arguably broken, but appears in common chroot setups such as Debian's sbuild tool where LANG is inherited from the parent environment, and LC_ALL is used to override it. --- ext/POSIX/t/mb.t | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/ext/POSIX/t/mb.t b/ext/POSIX/t/mb.t index 053693e611..3312b0d737 100644 --- a/ext/POSIX/t/mb.t +++ b/ext/POSIX/t/mb.t @@ -34,9 +34,13 @@ SKIP: { my $utf8_locale = find_utf8_ctype_locale(); skip("no utf8 locale available", 3) unless $utf8_locale; + # Here we need to influence LC_CTYPE, but it's not enough to just + # set this because LC_ALL could override it. It's also not enough + # to delete LC_ALL because it could be used to override other + # variables such as LANG in the underlying test environment. + # Continue to set LC_CTYPE just in case... local $ENV{LC_CTYPE} = $utf8_locale; - local $ENV{LC_ALL}; - delete $ENV{LC_ALL}; + local $ENV{LC_ALL} = $utf8_locale; fresh_perl_like( 'use POSIX; print &POSIX::MB_CUR_MAX', -- 2.11.0 ```

[p5pRT]

From @jkeenan

On Fri\, 07 Jun 2019 10​:06​:58 GMT\, dom wrote​:

This is a bug report for perl from dom@​earth.li\, generated with the help of perlbug 1.40 running under perl 5.24.1.

----------------------------------------------------------------- The test introduced at 25d7b7aa379d33ce2e8fe3e2bef4206b35739bc5 fails in environments where LANG is set to a locale which is not installed and LC_ALL is set to a valid locale. Such an environment is arguably broken\, but arises in a common use case in Debian build tools. This modification takes a more robust approach to modifying the environment.

Pushed to blead in commit 69b89a0f0bb2cbb4c1607e78c3b414bf45244bea\, with one committer's edit -- I had to remove a non-printing character in the patch​:

#####   #\Continue to set LC_CTYPE just in case... #####

Dom\, since I doubt any of our smoke-testing rigs are set up to reproduce this problem\, could you send us some sort of evidence that the problem has been fixed?

Thank you very much. -- James E Keenan (jkeenan@​cpan.org)

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From @jmdh

On Fri\, Jun 07\, 2019 at 05​:11​:10AM -0700\, James E Keenan via RT wrote​:

Dom\, since I doubt any of our smoke-testing rigs are set up to reproduce this problem\, could you send us some sort of evidence that the problem has been fixed?

Before the patch was applied\, this test failed in my Debian sbuild environment. Afterwards\, it succeeded.

(The relevant detail here is that outside sbuild\, my LANG is en_GB.UTF-8. This is progagated to within sbuild (which sets up a chroot) but that environment does not have the en_GB.UTF-8 locale data installed (since it's supposed to be a minimal environment).

sbuild itself corrects for this problem by setting LC_ALL\, so the fact that the test overrode LC_ALL caused the non-working en_GB.UTF-8 locale to be used.)

Thanks\, Domninic.

[p5pRT]

From @jkeenan

Dom\,

Unfortunately I have to call your attention to 2 smoke-test failures in ext/POSIX/t/mb.t which were recorded *after* I applied your patch.

http​://perl5.test-smoke.org/report/89146 logs at​: http​://perl5.test-smoke.org/logfile/89146

http​://perl5.test-smoke.org/report/89211 logs at​: http​://perl5.test-smoke.org/logfile/89211

(These can be tracked via this search​: http​://perl5.test-smoke.org/submatrix?test=../ext/POSIX/t/mb.t&pversion=5.31.1)

In each case the failures in mb.t occurred when blead was configured as follows​:

[stdio] -Dcc=clang -Accflags="-Werror=declaration-after-statement -g -fno-omit-frame-pointer -fsanitize=address -fno-common -fsanitize-blacklist=`pwd`/asan_ignore" -Aldflags="-fsanitize=address"

With and without -DDEBUGGING.

A couple of points​:

1. Tester is using what I would guess is an advanced version of the Linux kernel​: 5.0.9-200 versus my own 4.15.0-51 (Ubuntu 18.04 LTS). OTOH\, we are getting smoke-test reports from rigs with even higher-numbered Linux kernels.

2. I myself don't understand all those compiler switches the tester is using. In particular\, 'make' fails for me on FreeBSD-11.2 when I use those compiler switches.

3. Nonetheless\, when I build a perl with all those switches (except -DDEBUGGING)\, I get those test same failures. See attachments.

4. When I build blead with those same compiler switches at the commit immediately prior to the one where I applied your patch\, I get a PASS.

##### $ git show | head -1 commit fb55ce6b7596b9e94f941cf83eac5ff84f760ea2 $ cd t;./perl harness -v ../ext/POSIX/t/mb.t; cd -

ok 1 - mblen() basically works ok 2 - MB_CUR_MAX is at least 4 in a UTF-8 locale ok 3 - mblen() recognizes invalid multibyte characters ok 4 - mblen() works on UTF-8 characters ok All tests successful. Files=1\, Tests=4\, 0 wallclock secs ( 0.02 usr 0.00 sys + 0.26 cusr 0.12 csys = 0.40 CPU) Result​: PASS #####

So your patch has triggered test failures\, albeit under these very obscure conditions. I'm going to revert your patch from blead and then re-apply it in a branch so that we can continue to gather smoke-test reports.

Thank you very much.

-- James E Keenan (jkeenan@​cpan.org)

[p5pRT]

From @jkeenan

# Failed test 3 - mblen() recognizes invalid multibyte characters at ../../t/test.pl line 1062 # got "=================================================================\n==1656==ERROR​: AddressSanitizer​: heap-use-after-free on address 0x6020000006f0 at pc 0x0000004b166d bp 0x7fff1853e910 sp 0x7fff1853e0c0\nREAD of size 2 at 0x6020000006f0 thread T0\n #0 0x4b166c (/home/jkeenan/gitwork/perl/perl+0x4b166c)\n #1 0x6f0758 (/home/jkeenan/gitwork/perl/perl+0x6f0758)\n #2 0x6efb1f (/home/jkeenan/gitwork/perl/perl+0x6efb1f)\n #3 0x81bd6f (/home/jkeenan/gitwork/perl/perl+0x81bd6f)\n #4 0x73e3c9 (/home/jkeenan/gitwork/perl/perl+0x73e3c9)\n #5 0x5a27b1 (/home/jkeenan/gitwork/perl/perl+0x5a27b1)\n #6 0x598b20 (/home/jkeenan/gitwork/perl/perl+0x598b20)\n #7 0x56ca35 (/home/jkeenan/gitwork/perl/perl+0x56ca35)\n #8 0x554aff (/home/jkeenan/gitwork/perl/perl+0x554aff)\n #9 0x558399 (/home/jkeenan/gitwork/perl/perl+0x558399)\n #10 0x6230bd (/home/jkeenan/gitwork/perl/perl+0x6230bd)\n #11 0x823bdb (/home/jkeenan/gitwork/perl/perl+0x823bdb)\n #12 0x81f02c (/home/jkeenan/gitwork/perl/perl+0x81f02c)\n #13 0x73e3c9 (/home/jkeenan/gitwork/perl/perl+0x73e3c9)\n #14 0x5a27b1 (/home/jkeenan/gitwork/perl/perl+0x5a27b1)\n #15 0x598b20 (/home/jkeenan/gitwork/perl/perl+0x598b20)\n #16 0x56ca35 (/home/jkeenan/gitwork/perl/perl+0x56ca35)\n #17 0x554aff (/home/jkeenan/gitwork/perl/perl+0x554aff)\n #18 0x558399 (/home/jkeenan/gitwork/perl/perl+0x558399)\n #19 0x6230bd (/home/jkeenan/gitwork/perl/perl+0x6230bd)\n #20 0x59d88c (/home/jkeenan/gitwork/perl/perl+0x59d88c)\n #21 0x52f0fc (/home/jkeenan/gitwork/perl/perl+0x52f0fc)\n #22 0x7f57db6adb96 (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)\n #23 0x436b69 (/home/jkeenan/gitwork/perl/perl+0x436b69)\n\n0x6020000006f0 is located 0 bytes inside of 8-byte region [0x6020000006f0\,0x6020000006f8)\nfreed by thread T0 here​:\n #0 0x4f6850 (/home/jkeenan/gitwork/perl/perl+0x4f6850)\n #1 0x7f57db6b94cf (/lib/x86_64-linux-gnu/libc.so.6+0x2d4cf)\n\npreviously allocated by thread T0 here​:\n #0 0x4f6a20 (/home/jkeenan/gitwork/perl/perl+0x4f6a20)\n #1 0x7f57db7299b9 (/lib/x86_64-linux-gnu/libc.so.6+0x9d9b9)\n\nSUMMARY​: AddressSanitizer​: heap-use-after-free (/home/jkeenan/gitwork/perl/perl+0x4b166c) \nShadow bytes around the buggy address​:\n 0x0c047fff8080​: fa fa fd fa fa fa fd fa fa fa 00 fa fa fa 00 fa\n 0x0c047fff8090​: fa fa fd fa fa fa fd fa fa fa 00 fa fa fa 00 fa\n 0x0c047fff80a0​: fa fa fd fa fa fa fd fa fa fa 00 fa fa fa 00 fa\n 0x0c047fff80b0​: fa fa fd fa fa fa fd fa fa fa 00 fa fa fa 00 fa\n 0x0c047fff80c0​: fa fa fd fa fa fa fd fa fa fa 00 fa fa fa 00 fa\n=>0x0c047fff80d0​: fa fa fd fa fa fa fd fa fa fa 00 fa fa fa[fd]fa\n 0x0c047fff80e0​: fa fa fd fa fa fa fd fa fa fa 00 fa fa fa 00 fa\n 0x0c047fff80f0​: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa\n 0x0c047fff8100​: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa\n 0x0c047fff8110​: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa\n 0x0c047fff8120​: fa fa fd fa fa fa 06 fa fa fa 00 fa fa fa fd fa\nShadow byte legend (one shadow byte represents 8 application bytes)​:\n Addressable​: 00\n Partially addressable​: 01 02 03 04 05 06 07 \n Heap left redzone​: fa\n Freed heap region​: fd\n Stack left redzone​: f1\n Stack mid redzone​: f2\n Stack right redzone​: f3\n Stack after return​: f5\n Stack use after scope​: f8\n Global redzone​: f9\n Global init order​: f6\n Poisoned by user​: f7\n Container overflow​: fc\n Array cookie​: ac\n Intra object redzone​: bb\n ASan internal​: fe\n Left alloca redzone​: ca\n Right alloca redzone​: cb\n==1656==ABORTING" # expected "-1" # PROG​: # use POSIX; print &POSIX​::mblen("Ã("\, 2) # STATUS​: 256 # Failed test 4 - mblen() works on UTF-8 characters at ../../t/test.pl line 1062 # got "=================================================================\n==1658==ERROR​: AddressSanitizer​: heap-use-after-free on address 0x6020000006f0 at pc 0x0000004b166d bp 0x7ffe19094070 sp 0x7ffe19093820\nREAD of size 2 at 0x6020000006f0 thread T0\n #0 0x4b166c (/home/jkeenan/gitwork/perl/perl+0x4b166c)\n #1 0x6f0758 (/home/jkeenan/gitwork/perl/perl+0x6f0758)\n #2 0x6efb1f (/home/jkeenan/gitwork/perl/perl+0x6efb1f)\n #3 0x81bd6f (/home/jkeenan/gitwork/perl/perl+0x81bd6f)\n #4 0x73e3c9 (/home/jkeenan/gitwork/perl/perl+0x73e3c9)\n #5 0x5a27b1 (/home/jkeenan/gitwork/perl/perl+0x5a27b1)\n #6 0x598b20 (/home/jkeenan/gitwork/perl/perl+0x598b20)\n #7 0x56ca35 (/home/jkeenan/gitwork/perl/perl+0x56ca35)\n #8 0x554aff (/home/jkeenan/gitwork/perl/perl+0x554aff)\n #9 0x558399 (/home/jkeenan/gitwork/perl/perl+0x558399)\n #10 0x6230bd (/home/jkeenan/gitwork/perl/perl+0x6230bd)\n #11 0x823bdb (/home/jkeenan/gitwork/perl/perl+0x823bdb)\n #12 0x81f02c (/home/jkeenan/gitwork/perl/perl+0x81f02c)\n #13 0x73e3c9 (/home/jkeenan/gitwork/perl/perl+0x73e3c9)\n #14 0x5a27b1 (/home/jkeenan/gitwork/perl/perl+0x5a27b1)\n #15 0x598b20 (/home/jkeenan/gitwork/perl/perl+0x598b20)\n #16 0x56ca35 (/home/jkeenan/gitwork/perl/perl+0x56ca35)\n #17 0x554aff (/home/jkeenan/gitwork/perl/perl+0x554aff)\n #18 0x558399 (/home/jkeenan/gitwork/perl/perl+0x558399)\n #19 0x6230bd (/home/jkeenan/gitwork/perl/perl+0x6230bd)\n #20 0x59d88c (/home/jkeenan/gitwork/perl/perl+0x59d88c)\n #21 0x52f0fc (/home/jkeenan/gitwork/perl/perl+0x52f0fc)\n #22 0x7fdf1d3bfb96 (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)\n #23 0x436b69 (/home/jkeenan/gitwork/perl/perl+0x436b69)\n\n0x6020000006f0 is located 0 bytes inside of 8-byte region [0x6020000006f0\,0x6020000006f8)\nfreed by thread T0 here​:\n #0 0x4f6850 (/home/jkeenan/gitwork/perl/perl+0x4f6850)\n #1 0x7fdf1d3cb4cf (/lib/x86_64-linux-gnu/libc.so.6+0x2d4cf)\n\npreviously allocated by thread T0 here​:\n #0 0x4f6a20 (/home/jkeenan/gitwork/perl/perl+0x4f6a20)\n #1 0x7fdf1d43b9b9 (/lib/x86_64-linux-gnu/libc.so.6+0x9d9b9)\n\nSUMMARY​: AddressSanitizer​: heap-use-after-free (/home/jkeenan/gitwork/perl/perl+0x4b166c) \nShadow bytes around the buggy address​:\n 0x0c047fff8080​: fa fa fd fa fa fa fd fa fa fa 00 fa fa fa 00 fa\n 0x0c047fff8090​: fa fa fd fa fa fa fd fa fa fa 00 fa fa fa 00 fa\n 0x0c047fff80a0​: fa fa fd fa fa fa fd fa fa fa 00 fa fa fa 00 fa\n 0x0c047fff80b0​: fa fa fd fa fa fa fd fa fa fa 00 fa fa fa 00 fa\n 0x0c047fff80c0​: fa fa fd fa fa fa fd fa fa fa 00 fa fa fa 00 fa\n=>0x0c047fff80d0​: fa fa fd fa fa fa fd fa fa fa 00 fa fa fa[fd]fa\n 0x0c047fff80e0​: fa fa fd fa fa fa fd fa fa fa 00 fa fa fa 00 fa\n 0x0c047fff80f0​: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa\n 0x0c047fff8100​: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa\n 0x0c047fff8110​: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa\n 0x0c047fff8120​: fa fa fd fa fa fa 06 fa fa fa 00 fa fa fa fd fa\nShadow byte legend (one shadow byte represents 8 application bytes)​:\n Addressable​: 00\n Partially addressable​: 01 02 03 04 05 06 07 \n Heap left redzone​: fa\n Freed heap region​: fd\n Stack left redzone​: f1\n Stack mid redzone​: f2\n Stack right redzone​: f3\n Stack after return​: f5\n Stack use after scope​: f8\n Global redzone​: f9\n Global init order​: f6\n Poisoned by user​: f7\n Container overflow​: fc\n Array cookie​: ac\n Intra object redzone​: bb\n ASan internal​: fe\n Left alloca redzone​: ca\n Right alloca redzone​: cb\n==1658==ABORTING" # expected "2" # PROG​: # use POSIX; print &POSIX​::mblen("\N{GREEK SMALL LETTER SIGMA}"\, 2) # STATUS​: 256 ../ext/POSIX/t/mb.t .. 1..4 ok 1 - mblen() basically works ok 2 - MB_CUR_MAX is at least 4 in a UTF-8 locale not ok 3 - mblen() recognizes invalid multibyte characters not ok 4 - mblen() works on UTF-8 characters Failed 2/4 subtests

Test Summary Report

---

../ext/POSIX/t/mb.t (Wstat​: 0 Tests​: 4 Failed​: 2)   Failed tests​: 3-4 Files=1\, Tests=4\, 0 wallclock secs ( 0.01 usr 0.01 sys + 0.17 cusr 0.07 csys = 0.26 CPU) Result​: FAIL

[p5pRT]

From @jkeenan

Summary of my perl5 (revision 5 version 31 subversion 1) configuration​:   Commit id​: 7c21f0042bbf5d88b72d07661d64903e627ccf29   Platform​:   osname=linux   osvers=4.15.0-51-generic   archname=x86_64-linux   uname='linux zareason 4.15.0-51-generic #55-ubuntu smp wed may 15 14​:27​:21 utc 2019 x86_64 x86_64 x86_64 gnulinux '   config_args='-des -Dusedevel -Dcc=clang -Accflags=-Werror=declaration-after-statement -g -fno-omit-frame-pointer -fsanitize=address -fno-common -fsanitize-blacklist=/home/jkeenan/gitwork/perl/asan_ignore -Aldflags=-fsanitize=address'   hint=recommended   useposix=true   d_sigaction=define   useithreads=undef   usemultiplicity=undef   use64bitint=define   use64bitall=define   uselongdouble=undef   usemymalloc=n   default_inc_excludes_dot=define   bincompat5005=undef   Compiler​:   cc='clang'   ccflags ='-Werror=declaration-after-statement -g -fno-omit-frame-pointer -fsanitize=address -fno-common -fsanitize-blacklist=/home/jkeenan/gitwork/perl/asan_ignore -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2'   optimize='-O2'   cppflags='-Werror=declaration-after-statement -g -fno-omit-frame-pointer -fsanitize=address -fno-common -fsanitize-blacklist=/home/jkeenan/gitwork/perl/asan_ignore -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include'   ccversion=''   gccversion='4.2.1 Compatible Clang 6.0.0 (tags/RELEASE_600/final)'   gccosandvers=''   intsize=4   longsize=8   ptrsize=8   doublesize=8   byteorder=12345678   doublekind=3   d_longlong=define   longlongsize=8   d_longdbl=define   longdblsize=16   longdblkind=3   ivtype='long'   ivsize=8   nvtype='double'   nvsize=8   Off_t='off_t'   lseeksize=8   alignbytes=8   prototype=define   Linker and Libraries​:   ld='clang'   ldflags =' -fsanitize=address -fstack-protector-strong -L/usr/local/lib'   libpth=/usr/local/lib /usr/lib/llvm-6.0/lib/clang/6.0.0/lib /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib /lib64 /usr/lib64   libs=-lpthread -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc   perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc   libc=libc-2.27.so   so=so   useshrplib=false   libperl=libperl.a   gnulibc_version='2.27'   Dynamic Linking​:   dlsrc=dl_dlopen.xs   dlext=so   d_dlsymun=undef   ccdlflags='-Wl\,-E'   cccdlflags='-fPIC'   lddlflags='-shared -O2 -L/usr/local/lib -fstack-protector-strong'

Characteristics of this binary (from libperl)​:   Compile-time options​:   HAS_TIMES   PERLIO_LAYERS   PERL_COPY_ON_WRITE   PERL_DONT_CREATE_GVSV   PERL_MALLOC_WRAP   PERL_OP_PARENT   PERL_PRESERVE_IVUV   PERL_USE_DEVEL   USE_64_BIT_ALL   USE_64_BIT_INT   USE_LARGE_FILES   USE_LOCALE   USE_LOCALE_COLLATE   USE_LOCALE_CTYPE   USE_LOCALE_NUMERIC   USE_LOCALE_TIME   USE_PERLIO   USE_PERL_ATOF   Built under linux   Compiled at Jun 8 2019 17​:09​:02   %ENV​:   PERL2DIR="/home/jkeenan/gitwork/perl2"   PERLBREW_HOME="/home/jkeenan/.perlbrew"   PERLBREW_MANPATH="/home/jkeenan/perl5/perlbrew/perls/perl-5.30.0/man"   PERLBREW_PATH="/home/jkeenan/perl5/perlbrew/bin​:/home/jkeenan/perl5/perlbrew/perls/perl-5.30.0/bin"   PERLBREW_PERL="perl-5.30.0"   PERLBREW_ROOT="/home/jkeenan/perl5/perlbrew"   PERLBREW_SHELLRC_VERSION="0.84"   PERLBREW_VERSION="0.84"   PERL_WORKDIR="/home/jkeenan/gitwork/perl"   @​INC​:   lib   /usr/local/lib/perl5/site_perl/5.31.1/x86_64-linux   /usr/local/lib/perl5/site_perl/5.31.1   /usr/local/lib/perl5/5.31.1/x86_64-linux   /usr/local/lib/perl5/5.31.1

[p5pRT]

From @jkeenan

On Sat\, 08 Jun 2019 21​:43​:16 GMT\, jkeenan wrote​:

Dom\,

Unfortunately I have to call your attention to 2 smoke-test failures in ext/POSIX/t/mb.t which were recorded *after* I applied your patch.

http​://perl5.test-smoke.org/report/89146 logs at​: http​://perl5.test-smoke.org/logfile/89146

http​://perl5.test-smoke.org/report/89211 logs at​: http​://perl5.test-smoke.org/logfile/89211

(These can be tracked via this search​: http​://perl5.test- smoke.org/submatrix?test=../ext/POSIX/t/mb.t&pversion=5.31.1)

In each case the failures in mb.t occurred when blead was configured as follows​:

[stdio] -Dcc=clang -Accflags="-Werror=declaration-after-statement -g -fno-omit-frame-pointer -fsanitize=address -fno-common -fsanitize- blacklist=`pwd`/asan_ignore" -Aldflags="-fsanitize=address"

With and without -DDEBUGGING.

A couple of points​:

1. Tester is using what I would guess is an advanced version of the Linux kernel​: 5.0.9-200 versus my own 4.15.0-51 (Ubuntu 18.04 LTS). OTOH\, we are getting smoke-test reports from rigs with even higher- numbered Linux kernels.

2. I myself don't understand all those compiler switches the tester is using. In particular\, 'make' fails for me on FreeBSD-11.2 when I use those compiler switches.

3. Nonetheless\, when I build a perl with all those switches (except -DDEBUGGING)\, I get those test same failures. See attachments.

4. When I build blead with those same compiler switches at the commit immediately prior to the one where I applied your patch\, I get a PASS.

##### $ git show | head -1 commit fb55ce6b7596b9e94f941cf83eac5ff84f760ea2 $ cd t;./perl harness -v ../ext/POSIX/t/mb.t; cd -

ok 1 - mblen() basically works ok 2 - MB_CUR_MAX is at least 4 in a UTF-8 locale ok 3 - mblen() recognizes invalid multibyte characters ok 4 - mblen() works on UTF-8 characters ok All tests successful. Files=1\, Tests=4\, 0 wallclock secs ( 0.02 usr 0.00 sys + 0.26 cusr 0.12 csys = 0.40 CPU) Result​: PASS #####

So your patch has triggered test failures\, albeit under these very obscure conditions. I'm going to revert your patch from blead and then re-apply it in a branch so that we can continue to gather smoke- test reports.

Thank you very much.

The smoke-test branch is​:

smoke-me/jkeenan/dom/134182-mb

-- James E Keenan (jkeenan@​cpan.org)

[p5pRT]

From @jmdh

On Sat\, Jun 08\, 2019 at 02​:43​:16PM -0700\, James E Keenan via RT wrote​:

Dom\,

Unfortunately I have to call your attention to 2 smoke-test failures in ext/POSIX/t/mb.t which were recorded *after* I applied your patch.

http​://perl5.test-smoke.org/report/89146 logs at​: http​://perl5.test-smoke.org/logfile/89146

http​://perl5.test-smoke.org/report/89211 logs at​: http​://perl5.test-smoke.org/logfile/89211

(These can be tracked via this search​: http​://perl5.test-smoke.org/submatrix?test=../ext/POSIX/t/mb.t&pversion=5.31.1)

In each case the failures in mb.t occurred when blead was configured as follows​:

[stdio] -Dcc=clang -Accflags="-Werror=declaration-after-statement -g -fno-omit-frame-pointer -fsanitize=address -fno-common -fsanitize-blacklist=`pwd`/asan_ignore" -Aldflags="-fsanitize=address"

With and without -DDEBUGGING.

Very curious\, this looks like the original bug that the test was written for. Niko\, do you understand what's happening here?

Cheers\, Dominic\,

[p5pRT]

From @ntyni

On Fri\, Jun 14\, 2019 at 06​:21​:16PM +0100\, Dominic Hargreaves wrote​:

On Sat\, Jun 08\, 2019 at 02​:43​:16PM -0700\, James E Keenan via RT wrote​:

Unfortunately I have to call your attention to 2 smoke-test failures in ext/POSIX/t/mb.t which were recorded *after* I applied your patch.

In each case the failures in mb.t occurred when blead was configured as follows​:

[stdio] -Dcc=clang -Accflags="-Werror=declaration-after-statement -g -fno-omit-frame-pointer -fsanitize=address -fno-common -fsanitize-blacklist=`pwd`/asan_ignore" -Aldflags="-fsanitize=address"

With and without -DDEBUGGING.

Very curious\, this looks like the original bug that the test was written for. Niko\, do you understand what's happening here?

It's a different thing that just happened to get triggered here; this is with non-threaded builds for starters.

I can reproduce it on 5.30.0. It seems to be related to version strings and LC_NUMERIC. I reduced it to this​:

$ LC_NUMERIC=C.UTF-8 ./perl -l -Ilib -e 'require 5.006;'

==21403==ERROR​: AddressSanitizer​: heap-use-after-free on address 0x602000000190 at pc 0x0000004813aa bp 0x7fff4f62ea90 sp 0x7fff4f62e230 READ of size 2 at 0x602000000190 thread T0   #0 0x4813a9 in __interceptor_setlocale (/tmp/perl-5.30.0/perl+0x4813a9)   #1 0x6d7feb in Perl_upg_version /tmp/perl-5.30.0/./vutil.c​:717​:17   #2 0x6d73bf in Perl_new_version /tmp/perl-5.30.0/./vutil.c​:551​:12   #3 0x8019a4 in S_require_version /tmp/perl-5.30.0/pp_ctl.c​:3719​:10   #4 0x8019a4 in Perl_pp_require /tmp/perl-5.30.0/pp_ctl.c​:4345   #5 0x725bf9 in Perl_runops_standard /tmp/perl-5.30.0/run.c​:41​:26   #6 0x588f71 in S_run_body /tmp/perl-5.30.0/perl.c   #7 0x588381 in perl_run /tmp/perl-5.30.0/perl.c​:2639​:2   #8 0x516e1c in main /tmp/perl-5.30.0/perlmain.c​:127​:9   #9 0x7f073082a09a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2409a)   #10 0x43fc49 in _start (/tmp/perl-5.30.0/perl+0x43fc49)

0x602000000190 is located 0 bytes inside of 8-byte region [0x602000000190\,0x602000000198) freed by thread T0 here​:   #0 0x4e7712 in __interceptor_free (/tmp/perl-5.30.0/perl+0x4e7712)   #1 0x7f0730833963 in setlocale (/lib/x86_64-linux-gnu/libc.so.6+0x2d963)

previously allocated by thread T0 here​:   #0 0x4e7a93 in malloc (/tmp/perl-5.30.0/perl+0x4e7a93)   #1 0x7f073088ddb9 in __strdup (/lib/x86_64-linux-gnu/libc.so.6+0x87db9)

SUMMARY​: AddressSanitizer​: heap-use-after-free (/tmp/perl-5.30.0/perl+0x4813a9) in __interceptor_setlocale Shadow bytes around the buggy address​:   0x0c047fff7fe0​: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   0x0c047fff7ff0​: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   0x0c047fff8000​: fa fa 01 fa fa fa 00 02 fa fa 00 02 fa fa 00 02   0x0c047fff8010​: fa fa 00 02 fa fa 00 02 fa fa 00 02 fa fa 06 fa   0x0c047fff8020​: fa fa 00 02 fa fa fd fa fa fa fd fa fa fa 00 fa =>0x0c047fff8030​: fa fa[fd]fa fa fa fd fa fa fa fd fa fa fa fd fa   0x0c047fff8040​: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa   0x0c047fff8050​: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa   0x0c047fff8060​: fa fa fd fa fa fa 02 fa fa fa 00 fa fa fa 02 fa   0x0c047fff8070​: fa fa fd fa fa fa fd fa fa fa 00 00 fa fa 00 06   0x0c047fff8080​: fa fa 00 03 fa fa 00 03 fa fa 00 fa fa fa 00 04 Shadow byte legend (one shadow byte represents 8 application bytes)​:   Addressable​: 00   Partially addressable​: 01 02 03 04 05 06 07   Heap left redzone​: fa   Freed heap region​: fd   Stack left redzone​: f1   Stack mid redzone​: f2   Stack right redzone​: f3   Stack after return​: f5   Stack use after scope​: f8   Global redzone​: f9   Global init order​: f6   Poisoned by user​: f7   Container overflow​: fc   Array cookie​: ac   Intra object redzone​: bb   ASan internal​: fe   Left alloca redzone​: ca   Right alloca redzone​: cb   Shadow gap​: cc ==21403==ABORTING

-- Niko Tyni ntyni@​debian.org

[p5pRT]

From @ntyni

On Mon\, Jun 17\, 2019 at 09​:47​:04AM +0300\, Niko Tyni wrote​:

I can reproduce it on 5.30.0. It seems to be related to version strings and LC_NUMERIC. I reduced it to this​:

$ LC_NUMERIC=C.UTF-8 ./perl -l -Ilib -e 'require 5.006;'

And further to this. It's not clear to me if this is a problem with asan or the code.

$ cat t.c; clang -g -fsanitize=address t.c; ./a.out #include \<locale.h> int main(void) {   char *l;   setlocale(LC_NUMERIC\, "C.UTF-8");   l = setlocale(LC_NUMERIC\, NULL);   setlocale(LC_NUMERIC\, "C");   setlocale(LC_NUMERIC\, l); }

==17625==ERROR​: AddressSanitizer​: heap-use-after-free on address 0x602000000070 at pc 0x00000045ea3a bp 0x7ffce1e85f70 sp 0x7ffce1e85710 READ of size 2 at 0x602000000070 thread T0   #0 0x45ea39 in __interceptor_setlocale (/home/ntyni/a.out+0x45ea39)   #1 0x4f4327 in main /home/ntyni/t.c​:7​:5   #2 0x7fd77885209a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2409a)   #3 0x41d2d9 in _start (/home/ntyni/a.out+0x41d2d9)

0x602000000070 is located 0 bytes inside of 8-byte region [0x602000000070\,0x602000000078) freed by thread T0 here​:   #0 0x4c4da2 in __interceptor_free (/home/ntyni/a.out+0x4c4da2)   #1 0x7fd77885b963 in setlocale (/lib/x86_64-linux-gnu/libc.so.6+0x2d963)

previously allocated by thread T0 here​:   #0 0x4c5123 in malloc (/home/ntyni/a.out+0x4c5123)   #1 0x7fd7788b5db9 in __strdup (/lib/x86_64-linux-gnu/libc.so.6+0x87db9)

SUMMARY​: AddressSanitizer​: heap-use-after-free (/home/ntyni/a.out+0x45ea39) in __interceptor_setlocale Shadow bytes around the buggy address​:   0x0c047fff7fb0​: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   0x0c047fff7fc0​: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   0x0c047fff7fd0​: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   0x0c047fff7fe0​: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   0x0c047fff7ff0​: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =>0x0c047fff8000​: fa fa fd fa fa fa fd fa fa fa 00 fa fa fa[fd]fa   0x0c047fff8010​: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa   0x0c047fff8020​: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa   0x0c047fff8030​: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa   0x0c047fff8040​: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa   0x0c047fff8050​: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa Shadow byte legend (one shadow byte represents 8 application bytes)​:   Addressable​: 00   Partially addressable​: 01 02 03 04 05 06 07   Heap left redzone​: fa   Freed heap region​: fd   Stack left redzone​: f1   Stack mid redzone​: f2   Stack right redzone​: f3   Stack after return​: f5   Stack use after scope​: f8   Global redzone​: f9   Global init order​: f6   Poisoned by user​: f7   Container overflow​: fc   Array cookie​: ac   Intra object redzone​: bb   ASan internal​: fe   Left alloca redzone​: ca   Right alloca redzone​: cb   Shadow gap​: cc ==17625==ABORTING

-- Niko

[p5pRT]

From @ntyni

On Mon\, Jun 17\, 2019 at 02​:49​:43PM +0300\, Niko Tyni wrote​:

On Mon\, Jun 17\, 2019 at 09​:47​:04AM +0300\, Niko Tyni wrote​:

I can reproduce it on 5.30.0. It seems to be related to version strings and LC_NUMERIC. I reduced it to this​:

$ LC_NUMERIC=C.UTF-8 ./perl -l -Ilib -e 'require 5.006;'

And further to this. It's not clear to me if this is a problem with asan or the code.

$ cat t.c; clang -g -fsanitize=address t.c; ./a.out #include \<locale.h> int main(void) { char *l; setlocale(LC_NUMERIC\, "C.UTF-8"); l = setlocale(LC_NUMERIC\, NULL); setlocale(LC_NUMERIC\, "C"); setlocale(LC_NUMERIC\, l); }

Presumably the intervening setlocale() call clobbers the buffer that l points to.

The attached patch to vutil.c seems to fix this issue for me\, but eyeballs appreciated of course. -- Niko

[p5pRT]

From @ntyni

0001-Copy-setlocale-return-value-in-case-it-gets-clobbere.patch

```diff From 2357c65fd9559dd0852d1cf3febb3a4e468151ed Mon Sep 17 00:00:00 2001 From: Niko Tyni Date: Mon, 17 Jun 2019 16:21:20 +0300 Subject: [PATCH] Copy setlocale() return value in case it gets clobbered by later calls Flagged by AddressSanitizer in [perl #134182] Quoting IEEE Std 1003.1, 2004 Edition https://pubs.opengroup.org/onlinepubs/009695399/functions/setlocale.html The string returned by setlocale() is such that a subsequent call with that string and its associated category shall restore that part of the program's locale. The application shall not modify the string returned which may be overwritten by a subsequent call to setlocale(). Bug: https://rt.perl.org/Public/Bug/Display.html?id=134182 --- vutil.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/vutil.c b/vutil.c index 236748915..6814e59b0 100644 --- a/vutil.c +++ b/vutil.c @@ -639,7 +639,7 @@ VER_NV: LC_NUMERIC_LOCK(0); /* Start critical section */ - locale_name_on_entry = setlocale(LC_NUMERIC, NULL); + locale_name_on_entry = savepv(setlocale(LC_NUMERIC, NULL)); if ( strNE(locale_name_on_entry, "C") && strNE(locale_name_on_entry, "POSIX")) { @@ -647,6 +647,7 @@ VER_NV: } else { /* This value indicates to the restore code that we didn't change the locale */ + Safefree(locale_name_on_entry); locale_name_on_entry = NULL; } @@ -715,6 +716,7 @@ VER_NV: if (locale_name_on_entry) { setlocale(LC_NUMERIC, locale_name_on_entry); + Safefree(locale_name_on_entry); } LC_NUMERIC_UNLOCK; /* End critical section */ -- 2.20.1 ```

[p5pRT]

From @ntyni

On Mon\, Jun 17\, 2019 at 07​:20​:32PM +0300\, Niko Tyni wrote​:

On Mon\, Jun 17\, 2019 at 02​:49​:43PM +0300\, Niko Tyni wrote​:

On Mon\, Jun 17\, 2019 at 09​:47​:04AM +0300\, Niko Tyni wrote​:

#include \<locale.h> int main(void) { char *l; setlocale(LC_NUMERIC\, "C.UTF-8"); l = setlocale(LC_NUMERIC\, NULL); setlocale(LC_NUMERIC\, "C"); setlocale(LC_NUMERIC\, l); }

Presumably the intervening setlocale() call clobbers the buffer that l points to.

The attached patch to vutil.c seems to fix this issue for me\, but eyeballs appreciated of course.

I see vutil.c comes from the version.pm distribution so I've submitted the patch there as

  https://github.com/Perl/version.pm/pull/7

-- Niko

[p5pRT]

From @tonycoz

On Mon\, 17 Jun 2019 09​:21​:04 -0700\, ntyni@​debian.org wrote​:

On Mon\, Jun 17\, 2019 at 02​:49​:43PM +0300\, Niko Tyni wrote​:

On Mon\, Jun 17\, 2019 at 09​:47​:04AM +0300\, Niko Tyni wrote​:

I can reproduce it on 5.30.0. It seems to be related to version strings and LC_NUMERIC. I reduced it to this​:

$ LC_NUMERIC=C.UTF-8 ./perl -l -Ilib -e 'require 5.006;'

And further to this. It's not clear to me if this is a problem with asan or the code.

$ cat t.c; clang -g -fsanitize=address t.c; ./a.out #include \<locale.h> int main(void) { char *l; setlocale(LC_NUMERIC\, "C.UTF-8"); l = setlocale(LC_NUMERIC\, NULL); setlocale(LC_NUMERIC\, "C"); setlocale(LC_NUMERIC\, l); }

Presumably the intervening setlocale() call clobbers the buffer that l points to.

The attached patch to vutil.c seems to fix this issue for me\, but eyeballs appreciated of course.

https://rt-archive.perl.org/perl5/Ticket/Display.html?id=134212 has a more complete fix (I didn't see this until I diagnosed it.)

Tony

[khwilliamson]

I believe this ticket can be closed, since the patch referenced above has been applied.

Any disagreement?

[ntyni]

Yes, looks good and closeable to me. Thanks!

[ntyni]

Uh, taking this back: the original issue @jmdh filed here about mb.t failing on semi-broken locales seems to be still present. @jkeenan reverted the proposed patch after smokers caught the separate issue with memory corruption around vutil.c that's now fixed, but the patch is not reinstated yet afaics.

[khwilliamson]

I rebased the patch, and am smoking it at https://git.io/Jvd6h @jkeenan could you see if it passes things the previous versions failed on; or were those irrelevant to this patch?

[jkeenan]

I built perl in the smoke-me/khw-mb branch at v5.31.10-26-ge37211489e with these config_args:

./perl -Ilib -V:config_args
config_args='-des -Dusedevel -Dcc=clang -Accflags=-Werror=declaration-after-statement -g -fno-omit-frame-pointer -fsanitize=address -fno-common -fsanitize-blacklist=/home/jkeenan/gitwork/perl/asan_ignore -Aldflags=-fsanitize=address';

I then ran:

$ cd t;./perl harness -v ../ext/POSIX/t/mb.t; cd -

ok 1 - mblen() works on ASCII input
ok 2 - ... and the 2nd parameter is optional
ok 3 - MB_CUR_MAX is at least 4 in a UTF-8 locale
ok 4 - mblen() recognizes invalid multibyte characters
ok 5 - mblen() works on UTF-8 characters
ok 6 - mblen() returns -1 when input length is too short
ok 7 - mbtowc() returns correct length on ASCII input
ok 8 - mbtowc() returns correct ordinal on ASCII input
ok 9 - mbtowc() recognizes invalid multibyte characters
ok 10 - mbtowc() works on UTF-8 characters
ok 11 - mbtowc() returns -1 when input length is too short
ok 12 - wctomb() works on ASCII input
ok 13 - wctomb() works on UTF-8 characters
ok
All tests successful.
Files=1, Tests=13,  1 wallclock secs ( 0.03 usr  0.01 sys +  0.77 cusr  0.20 csys =  1.01 CPU)
Result: PASS

That may resolve the test failures. But I can't be very confident of my results because (a) when I build perl with address sanitizer my computer slows to a halt during make test_harness, makes it impossible to toggle between programs and forces me to reboot; (b) my Linux kernel is well behind the one that is regularly used with these configure args.

Thank you very much. Jim Keenan

[khwilliamson]

@xsawyerx I would like permission to merge this patch for 5.32 It is very low risk, as it affects just one .t file that didn't even exist in 5.30, and makes life easier for our downstream Debian partners. And it fell through the cracks for months. I have tested that things fail before the patch is applied in the situation it applies to, and pass after it is applied.

[xsawyerx]

Approved!

[khwilliamson]

Fixed by https://github.com/Perl/perl5/commit/8f8f6a18de1ce640a226f841deebf018443f872f