cpan/Test-Simple/t/Legacy/More.t: frequent failures in non-threaded builds on FreeBSD-13

[p5pRT]

Migrated from rt.perl.org#133958 (status was 'resolved')

Searchable as RT133958$

[p5pRT]

From @jkeenan

We are experiencing massive test failures in our smoke-testing of the Perl 5 core distribution on FreeBSD-13-CURRENT. I am filing two separate bug tickets for these failures\, since one failure appeared long before the others. This is the first of the two tickets.

cpan/Test-Simple/t/Legacy/More.t is repeatedly failing during smoke-testing of non-threaded builds on FreeBSD-13. See\, e.g.​:

http​://perl5.test-smoke.org/report/82470 http​://perl5.test-smoke.org/report/82946

Test failures​: ~~ ../cpan/Test-Simple/t/Legacy/More.t ......................... FAILED 54 Non-zero exit status​: 1   [stdio] -Dcc=clang++ -Duse64bitall   [stdio] -Dcc=clang++   [stdio] -Dcc=clang++ DEBUGGING

We only have one regular smoke-testing rig set up for FreeBSD-13\, that of course being one of Carlos Guevara's rigs. These failures​:

i. Do not seem to occur on threaded builds.

ii. Seem to occur on most\, but not all\, non-threaded builds.

iii. Appear to have first reported during a smoke-test in http​://perl5.test-smoke.org/report/79958\, which suggests that the failures were introduced in\, or shortly before commit https://perl5.git.perl.org/perl.git/commitdiff/8021814b48.  That commit is in a deleted branch​: smoke-me/khw-vlb.

iv. Both Carlos and I have attempted to reproduce this failure manually -- but it only seems to crop up during smoke-testing.

v. Here's the end of the file which is failing​:

##### $ tail cpan/Test-Simple/t/Legacy/More.t   );

# rt.cpan.org 53469 is_deeply with regexes is_deeply( qr/a/\, qr/a/\, "same regex" );

# These two tests must remain at the end. is( $@​\, $Err\, '$@​ untouched' ); cmp_ok( $!\, '=='\, $Errno\, '$! untouched' ); #####

Thank you very much. Jim Keenan

perl perl perl

[p5pRT]

From @khwilliamson

On 3/25/19 6​:45 PM\, James E Keenan (via RT) wrote​:

# New Ticket Created by James E Keenan # Please include the string​: [perl #133958] # in the subject line of all future correspondence about this issue. # \<URL​: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=133958 >

We are experiencing massive test failures in our smoke-testing of the Perl 5 core distribution on FreeBSD-13-CURRENT. I am filing two separate bug tickets for these failures\, since one failure appeared long before the others. This is the first of the two tickets.

cpan/Test-Simple/t/Legacy/More.t is repeatedly failing during smoke-testing of non-threaded builds on FreeBSD-13. See\, e.g.​:

http​://perl5.test-smoke.org/report/82470 http​://perl5.test-smoke.org/report/82946

Test failures​: ~~ ../cpan/Test-Simple/t/Legacy/More.t ......................... FAILED 54 Non-zero exit status​: 1 [stdio] -Dcc=clang++ -Duse64bitall [stdio] -Dcc=clang++ [stdio] -Dcc=clang++ DEBUGGING

We only have one regular smoke-testing rig set up for FreeBSD-13\, that of course being one of Carlos Guevara's rigs. These failures​:

i. Do not seem to occur on threaded builds.

ii. Seem to occur on most\, but not all\, non-threaded builds.

iii. Appear to have first reported during a smoke-test in http​://perl5.test-smoke.org/report/79958\, which suggests that the failures were introduced in\, or shortly before commit https://perl5.git.perl.org/perl.git/commitdiff/8021814b48.&nbsp; That commit is in a deleted branch​: smoke-me/khw-vlb.

iv. Both Carlos and I have attempted to reproduce this failure manually -- but it only seems to crop up during smoke-testing.

v. Here's the end of the file which is failing​:

##### $ tail cpan/Test-Simple/t/Legacy/More.t );

# rt.cpan.org 53469 is_deeply with regexes is_deeply( qr/a/\, qr/a/\, "same regex" );

# These two tests must remain at the end. is( $@​\, $Err\, '$@​ untouched' ); cmp_ok( $!\, '=='\, $Errno\, '$! untouched' ); #####

Thank you very much. Jim Keenan

perl perl perl

Have you tried manually running this with address sanitizer?

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From @jkeenan

On 3/27/19 11​:15 PM\, karl williamson via RT wrote​:

On 3/25/19 6​:45 PM\, James E Keenan (via RT) wrote​:

# New Ticket Created by James E Keenan # Please include the string​: [perl #133958] # in the subject line of all future correspondence about this issue. # \<URL​: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=133958 >

We are experiencing massive test failures in our smoke-testing of the Perl 5 core distribution on FreeBSD-13-CURRENT. I am filing two separate bug tickets for these failures\, since one failure appeared long before the others. This is the first of the two tickets.

cpan/Test-Simple/t/Legacy/More.t is repeatedly failing during smoke-testing of non-threaded builds on FreeBSD-13. See\, e.g.​:

http​://perl5.test-smoke.org/report/82470 http​://perl5.test-smoke.org/report/82946

Test failures​: ~~ ../cpan/Test-Simple/t/Legacy/More.t ......................... FAILED 54 Non-zero exit status​: 1 [stdio] -Dcc=clang++ -Duse64bitall [stdio] -Dcc=clang++ [stdio] -Dcc=clang++ DEBUGGING

We only have one regular smoke-testing rig set up for FreeBSD-13\, that of course being one of Carlos Guevara's rigs. These failures​:

i. Do not seem to occur on threaded builds.

ii. Seem to occur on most\, but not all\, non-threaded builds.

iii. Appear to have first reported during a smoke-test in http​://perl5.test-smoke.org/report/79958\, which suggests that the failures were introduced in\, or shortly before commit https://perl5.git.perl.org/perl.git/commitdiff/8021814b48.&nbsp; That commit is in a deleted branch​: smoke-me/khw-vlb.

iv. Both Carlos and I have attempted to reproduce this failure manually -- but it only seems to crop up during smoke-testing.

v. Here's the end of the file which is failing​:

##### $ tail cpan/Test-Simple/t/Legacy/More.t );

# rt.cpan.org 53469 is_deeply with regexes is_deeply( qr/a/\, qr/a/\, "same regex" );

# These two tests must remain at the end. is( $@​\, $Err\, '$@​ untouched' ); cmp_ok( $!\, '=='\, $Errno\, '$! untouched' ); #####

Thank you very much. Jim Keenan

perl perl perl

Have you tried manually running this with address sanitizer?

1. I myself have never run address sanitizer. How would I do that?

2. Even if I did that\, how would that address the problem that we've only been able to observe this problem during smoke-testing?

Thank you very much. Jim Keenan

[p5pRT]

From @khwilliamson

On 3/28/19 6​:21 AM\, James E Keenan wrote​:

On 3/27/19 11​:15 PM\, karl williamson via RT wrote​:

On 3/25/19 6​:45 PM\, James E Keenan (via RT) wrote​:

# New Ticket Created by  James E Keenan # Please include the string​:  [perl #133958] # in the subject line of all future correspondence about this issue. # \<URL​: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=133958 >

We are experiencing massive test failures in our smoke-testing of the Perl 5 core distribution on FreeBSD-13-CURRENT.  I am filing two separate bug tickets for these failures\, since one failure appeared long before the others.  This is the first of the two tickets.

cpan/Test-Simple/t/Legacy/More.t is repeatedly failing during smoke-testing of non-threaded builds on FreeBSD-13.  See\, e.g.​:

http​://perl5.test-smoke.org/report/82470 http​://perl5.test-smoke.org/report/82946

Test failures​: ~~ ../cpan/Test-Simple/t/Legacy/More.t ......................... FAILED 54 Non-zero exit status​: 1      [stdio] -Dcc=clang++ -Duse64bitall      [stdio] -Dcc=clang++      [stdio] -Dcc=clang++ DEBUGGING

We only have one regular smoke-testing rig set up for FreeBSD-13\, that of course being one of Carlos Guevara's rigs.  These failures​:

i. Do not seem to occur on threaded builds.

ii. Seem to occur on most\, but not all\, non-threaded builds.

iii. Appear to have first reported during a smoke-test in http​://perl5.test-smoke.org/report/79958\, which suggests that the failures were introduced in\, or shortly before commit https://perl5.git.perl.org/perl.git/commitdiff/8021814b48.  That commit is in a deleted branch​: smoke-me/khw-vlb.

iv. Both Carlos and I have attempted to reproduce this failure manually -- but it only seems to crop up during smoke-testing.

v. Here's the end of the file which is failing​:

##### $ tail cpan/Test-Simple/t/Legacy/More.t             );

# rt.cpan.org 53469  is_deeply with regexes is_deeply( qr/a/\, qr/a/\, "same regex" );

# These two tests must remain at the end. is( $@​\, $Err\,               '$@​ untouched' ); cmp_ok( $!\, '=='\, $Errno\,   '$! untouched' ); #####

Thank you very much. Jim Keenan

perl perl perl

Have you tried manually running this with address sanitizer?

1. I myself have never run address sanitizer.  How would I do that?

Instructions are in perlhacktips. It's just some Configure options to clang\, which I believe you're already using. Then you run the test suite.

2. Even if I did that\, how would that address the problem that we've only been able to observe this problem during smoke-testing?

There are two likely causes for something showing up only in smoke testing that I can think of off-hand. One is timing\, with lots of parallel jobs. And the other is that there is a read of uninitialized memory. Address sanitizer will catch this latter cause. This is similar to valgrind on Linux. I have run valgrind on the test suite recently with no errors beyond the usual two in CPAN (and yes there is a ticket for these open)

Thank you very much. Jim Keenan

[p5pRT]

From @jkeenan

On 3/28/19 10​:16 AM\, Karl Williamson wrote​:

On 3/28/19 6​:21 AM\, James E Keenan wrote​:

On 3/27/19 11​:15 PM\, karl williamson via RT wrote​:

On 3/25/19 6​:45 PM\, James E Keenan (via RT) wrote​:

# New Ticket Created by  James E Keenan # Please include the string​:  [perl #133958] # in the subject line of all future correspondence about this issue. # \<URL​: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=133958 >

We are experiencing massive test failures in our smoke-testing of the Perl 5 core distribution on FreeBSD-13-CURRENT.  I am filing two separate bug tickets for these failures\, since one failure appeared long before the others.  This is the first of the two tickets.

cpan/Test-Simple/t/Legacy/More.t is repeatedly failing during smoke-testing of non-threaded builds on FreeBSD-13.  See\, e.g.​:

http​://perl5.test-smoke.org/report/82470 http​://perl5.test-smoke.org/report/82946

Test failures​: ~~ ../cpan/Test-Simple/t/Legacy/More.t ......................... FAILED 54 Non-zero exit status​: 1      [stdio] -Dcc=clang++ -Duse64bitall      [stdio] -Dcc=clang++      [stdio] -Dcc=clang++ DEBUGGING

We only have one regular smoke-testing rig set up for FreeBSD-13\, that of course being one of Carlos Guevara's rigs.  These failures​:

i. Do not seem to occur on threaded builds.

ii. Seem to occur on most\, but not all\, non-threaded builds.

iii. Appear to have first reported during a smoke-test in http​://perl5.test-smoke.org/report/79958\, which suggests that the failures were introduced in\, or shortly before commit https://perl5.git.perl.org/perl.git/commitdiff/8021814b48.  That commit is in a deleted branch​: smoke-me/khw-vlb.

iv. Both Carlos and I have attempted to reproduce this failure manually -- but it only seems to crop up during smoke-testing.

v. Here's the end of the file which is failing​:

##### $ tail cpan/Test-Simple/t/Legacy/More.t             );

# rt.cpan.org 53469  is_deeply with regexes is_deeply( qr/a/\, qr/a/\, "same regex" );

# These two tests must remain at the end. is( $@​\, $Err\,               '$@​ untouched' ); cmp_ok( $!\, '=='\, $Errno\,   '$! untouched' ); #####

Thank you very much. Jim Keenan

perl perl perl

Have you tried manually running this with address sanitizer?

1. I myself have never run address sanitizer.  How would I do that?

Instructions are in perlhacktips.  It's just some Configure options to clang\, which I believe you're already using.  Then you run the test suite.

2. Even if I did that\, how would that address the problem that we've only been able to observe this problem during smoke-testing?

There are two likely causes for something showing up only in smoke testing that I can think of off-hand.  One is timing\, with lots of parallel jobs.

I should note that in this environment $TEST_JOBS is set to 2. So (assuming I understand the test code correctly)\, only handy00.t and handy01.t really matter. I tried running those two in parallel by​:

$> cd t; ./perl harness ../ext/XS-APItest/t/handy00.t ../ext/XS-APItest/t/handy01.t; cd -

I could not reproduce the problem. But it seems to occur regularly inside Test-Smoke's harness.

And the other is that there is a read of uninitialized memory.  Address sanitizer will catch this latter cause.  This is similar to valgrind on Linux.

I will try to test with valgrind today or tomorrow.

I have run valgrind on the test suite recently with no errors beyond the usual two in CPAN (and yes there is a ticket for these open)

Thank you very much. Jim Keenan

[p5pRT]

From @jkeenan

On Thu\, 28 Mar 2019 15​:15​:50 GMT\, jkeenan@​pobox.com wrote​:

On 3/28/19 10​:16 AM\, Karl Williamson wrote​:

On 3/28/19 6​:21 AM\, James E Keenan wrote​:

On 3/27/19 11​:15 PM\, karl williamson via RT wrote​:

On 3/25/19 6​:45 PM\, James E Keenan (via RT) wrote​:

# New Ticket Created by  James E Keenan # Please include the string​:  [perl #133958] # in the subject line of all future correspondence about this issue. # \<URL​: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=133958 >

We are experiencing massive test failures in our smoke-testing of the Perl 5 core distribution on FreeBSD-13-CURRENT.  I am filing two separate bug tickets for these failures\, since one failure appeared long before the others.  This is the first of the two tickets.

cpan/Test-Simple/t/Legacy/More.t is repeatedly failing during smoke-testing of non-threaded builds on FreeBSD-13.  See\, e.g.​:

http​://perl5.test-smoke.org/report/82470 http​://perl5.test-smoke.org/report/82946

Test failures​: ~~ ../cpan/Test-Simple/t/Legacy/More.t ......................... FAILED 54 Non-zero exit status​: 1      [stdio] -Dcc=clang++ -Duse64bitall      [stdio] -Dcc=clang++      [stdio] -Dcc=clang++ DEBUGGING

We only have one regular smoke-testing rig set up for FreeBSD-13\, that of course being one of Carlos Guevara's rigs.  These failures​:

i. Do not seem to occur on threaded builds.

ii. Seem to occur on most\, but not all\, non-threaded builds.

iii. Appear to have first reported during a smoke-test in http​://perl5.test-smoke.org/report/79958\, which suggests that the failures were introduced in\, or shortly before commit https://perl5.git.perl.org/perl.git/commitdiff/8021814b48.  That commit is in a deleted branch​: smoke-me/khw-vlb.

iv. Both Carlos and I have attempted to reproduce this failure manually -- but it only seems to crop up during smoke-testing.

v. Here's the end of the file which is failing​:

##### $ tail cpan/Test-Simple/t/Legacy/More.t             );

# rt.cpan.org 53469  is_deeply with regexes is_deeply( qr/a/\, qr/a/\, "same regex" );

# These two tests must remain at the end. is( $@​\, $Err\,               '$@​ untouched' ); cmp_ok( $!\, '=='\, $Errno\,   '$! untouched' ); #####

Thank you very much. Jim Keenan

perl perl perl

Have you tried manually running this with address sanitizer?

1. I myself have never run address sanitizer.  How would I do that?

Instructions are in perlhacktips.  It's just some Configure options to clang\, which I believe you're already using.  Then you run the test suite.

2. Even if I did that\, how would that address the problem that we've only been able to observe this problem during smoke-testing?

There are two likely causes for something showing up only in smoke testing that I can think of off-hand.  One is timing\, with lots of parallel jobs.

I should note that in this environment $TEST_JOBS is set to 2. So (assuming I understand the test code correctly)\, only handy00.t and handy01.t really matter. I tried running those two in parallel by​:

$> cd t; ./perl harness ../ext/XS-APItest/t/handy00.t ../ext/XS-APItest/t/handy01.t; cd -

I could not reproduce the problem. But it seems to occur regularly inside Test-Smoke's harness.

And the other is that there is a read of uninitialized memory.  Address sanitizer will catch this latter cause.  This is similar to valgrind on Linux.

I will try to test with valgrind today or tomorrow.

I mistyped. I meant to say that I would try to build a perl with AddressSanitizer. See below.

I have run valgrind on the test suite recently with no errors beyond the usual two in CPAN (and yes there is a ticket for these open)

My efforts to build and test a perl at blead with AddressSanitizer in this FreeBSD-13 were largely unsuccessful.

After struggling with the syntax of the ./Configure invocation\, I was eventually able to get 'make' to complete successfully. See attachment with name starting with 'perl_V'.

However\, I could not run 'make test_harness' successfully. Normally\, in these VMs I have $TEST_JOBS set to 2. However\, 'make test_harness' was 'Killed' in the t/re/*.t tests. See attachment starting with 'test_harness'.

I then set TEST_JOBS=1 and ran 'make test' -- so no tests were running in parallel. I got only a little farther on. See attachment starting with 'make_test'.

Now\, I concede​: (a) I've never previously attempted to build with AddressSanitizer; (b) This VM may be too small to run a perl built with AddressSanitizer. But Carlos's FreeBSD-13 smoker\, while not large\, is larger than mine -- and we're still getting the failures on non-threaded builds in t/Legacy/More.t which are the subject of this RT. See\, e.g.\, http​://perl5.test-smoke.org/report/83974.

Thank you very much.

-- James E Keenan (jkeenan@​cpan.org)

[p5pRT]

From @jkeenan

t/re/no_utf8_pm ................................................ ok t/re/overload .................................................. ok t/re/pat ....................................................... FAILED--expected 860 tests\, saw 108 t/re/pat_advanced .............................................. ok t/re/pat_advanced_thr .......................................... ok t/re/pat_psycho ................................................ # Test process timed out - terminating FAILED--expected 15 tests\, saw 0 t/re/pat_psycho_thr ............................................ Killed *** Error code 137

Stop. make​: stopped in /home/jkeenan/gitwork/perl

[p5pRT]

From @jkeenan

Summary of my perl5 (revision 5 version 29 subversion 10) configuration​:   Commit id​: 8e48a48cc5006a06110dd20994b419be393e42ce   Platform​:   osname=freebsd   osvers=13.0-current   archname=amd64-freebsd-thread-multi   uname='freebsd perl-reporter-05 13.0-current freebsd 13.0-current r340361 generic amd64 '   config_args='-des -Dusedevel -Duseithreads -Dcc=clang -Accflags=-fsanitize=address -Aldflags=-fsanitize=address -Alddlflags=-shared -fsanitize=address'   hint=recommended   useposix=true   d_sigaction=define   useithreads=define   usemultiplicity=define   use64bitint=define   use64bitall=define   uselongdouble=undef   usemymalloc=n   default_inc_excludes_dot=define   bincompat5005=undef   Compiler​:   cc='clang'   ccflags ='-DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -fsanitize=address -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_FORTIFY_SOURCE=2'   optimize='-O2'   cppflags='-DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -fsanitize=address -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include'   ccversion=''   gccversion='4.2.1 Compatible FreeBSD Clang 6.0.1 (tags/RELEASE_601/final 335540)'   gccosandvers=''   intsize=4   longsize=8   ptrsize=8   doublesize=8   byteorder=12345678   doublekind=3   d_longlong=define   longlongsize=8   d_longdbl=define   longdblsize=16   longdblkind=3   ivtype='long'   ivsize=8   nvtype='double'   nvsize=8   Off_t='off_t'   lseeksize=8   alignbytes=8   prototype=define   Linker and Libraries​:   ld='clang'   ldflags ='-pthread -Wl\,-E -fsanitize=address -fstack-protector-strong -L/usr/local/lib'   libpth=/usr/lib /usr/local/lib /usr/lib/clang/6.0.1/lib /usr/lib   libs=-lpthread -lgdbm -ldl -lm -lcrypt -lutil   perllibs=-lpthread -ldl -lm -lcrypt -lutil   libc=   so=so   useshrplib=false   libperl=libperl.a   gnulibc_version=''   Dynamic Linking​:   dlsrc=dl_dlopen.xs   dlext=so   d_dlsymun=undef   ccdlflags=' '   cccdlflags='-DPIC -fPIC'   lddlflags='-shared -shared -fsanitize=address -L/usr/local/lib -fstack-protector-strong'

Characteristics of this binary (from libperl)​:   Compile-time options​:   HAS_TIMES   MULTIPLICITY   PERLIO_LAYERS   PERL_COPY_ON_WRITE   PERL_DONT_CREATE_GVSV   PERL_IMPLICIT_CONTEXT   PERL_MALLOC_WRAP   PERL_OP_PARENT   PERL_PRESERVE_IVUV   PERL_USE_DEVEL   USE_64_BIT_ALL   USE_64_BIT_INT   USE_ITHREADS   USE_LARGE_FILES   USE_LOCALE   USE_LOCALE_COLLATE   USE_LOCALE_CTYPE   USE_LOCALE_NUMERIC   USE_LOCALE_TIME   USE_PERLIO   USE_PERL_ATOF   USE_REENTRANT_API   USE_THREAD_SAFE_LOCALE   Built under freebsd   Compiled at Mar 29 2019 14​:44​:36   %ENV​:   PERL2DIR="/home/jkeenan/gitwork/perl2"   PERL_WORKDIR="/home/jkeenan/gitwork/perl"   @​INC​:   lib   /usr/local/lib/perl5/site_perl/5.29.10/amd64-freebsd-thread-multi   /usr/local/lib/perl5/site_perl/5.29.10   /usr/local/lib/perl5/5.29.10/amd64-freebsd-thread-multi   /usr/local/lib/perl5/5.29.10

[p5pRT]

From @jkeenan

./miniperl -Ilib make_ext.pl cpan/Archive-Tar/pm_to_blib MAKE="make" LIBPERL_A=libperl.a ./miniperl -Ilib make_ext.pl dist/Attribute-Handlers/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for dist/Attribute-Handlers directly ./miniperl -Ilib make_ext.pl cpan/AutoLoader/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for cpan/AutoLoader directly ./miniperl -Ilib make_ext.pl cpan/CPAN/pm_to_blib MAKE="make" LIBPERL_A=libperl.a ./miniperl -Ilib make_ext.pl cpan/CPAN-Meta/pm_to_blib MAKE="make" LIBPERL_A=libperl.a ./miniperl -Ilib make_ext.pl cpan/CPAN-Meta-Requirements/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for cpan/CPAN-Meta-Requirements directly ./miniperl -Ilib make_ext.pl cpan/CPAN-Meta-YAML/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for cpan/CPAN-Meta-YAML directly ./miniperl -Ilib make_ext.pl dist/Carp/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for dist/Carp directly ./miniperl -Ilib make_ext.pl cpan/Config-Perl-V/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for cpan/Config-Perl-V directly ./miniperl -Ilib make_ext.pl dist/Devel-SelfStubber/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for dist/Devel-SelfStubber directly ./miniperl -Ilib make_ext.pl cpan/Digest/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for cpan/Digest directly ./miniperl -Ilib make_ext.pl dist/Dumpvalue/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for dist/Dumpvalue directly ./miniperl -Ilib make_ext.pl dist/Env/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for dist/Env directly ./miniperl -Ilib make_ext.pl ext/Errno/pm_to_blib MAKE="make" LIBPERL_A=libperl.a ./miniperl -Ilib make_ext.pl dist/Exporter/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for dist/Exporter directly ./miniperl -Ilib make_ext.pl dist/ExtUtils-CBuilder/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for dist/ExtUtils-CBuilder directly ./miniperl -Ilib make_ext.pl cpan/ExtUtils-Constant/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for cpan/ExtUtils-Constant directly ./miniperl -Ilib make_ext.pl cpan/ExtUtils-Install/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for cpan/ExtUtils-Install directly ./miniperl -Ilib make_ext.pl cpan/ExtUtils-MakeMaker/pm_to_blib MAKE="make" LIBPERL_A=libperl.a ./miniperl -Ilib make_ext.pl cpan/ExtUtils-Manifest/pm_to_blib MAKE="make" LIBPERL_A=libperl.a ./miniperl -Ilib make_ext.pl ext/ExtUtils-Miniperl/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for ext/ExtUtils-Miniperl directly ./miniperl -Ilib make_ext.pl dist/ExtUtils-ParseXS/pm_to_blib MAKE="make" LIBPERL_A=libperl.a ./miniperl -Ilib make_ext.pl cpan/File-Fetch/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for cpan/File-Fetch directly ./miniperl -Ilib make_ext.pl ext/File-Find/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for ext/File-Find directly ./miniperl -Ilib make_ext.pl cpan/File-Path/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for cpan/File-Path directly ./miniperl -Ilib make_ext.pl cpan/File-Temp/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for cpan/File-Temp directly ./miniperl -Ilib make_ext.pl ext/FileCache/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for ext/FileCache directly ./miniperl -Ilib make_ext.pl dist/Filter-Simple/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for dist/Filter-Simple directly ./miniperl -Ilib make_ext.pl cpan/Getopt-Long/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for cpan/Getopt-Long directly ./miniperl -Ilib make_ext.pl cpan/HTTP-Tiny/pm_to_blib MAKE="make" LIBPERL_A=libperl.a ./miniperl -Ilib make_ext.pl dist/I18N-Collate/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for dist/I18N-Collate directly ./miniperl -Ilib make_ext.pl dist/I18N-LangTags/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for dist/I18N-LangTags directly ./miniperl -Ilib make_ext.pl dist/lib/pm_to_blib MAKE="make" LIBPERL_A=libperl.a ./miniperl -Ilib make_ext.pl cpan/IO-Compress/pm_to_blib MAKE="make" LIBPERL_A=libperl.a ./miniperl -Ilib make_ext.pl cpan/IO-Socket-IP/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for cpan/IO-Socket-IP directly ./miniperl -Ilib make_ext.pl cpan/IO-Zlib/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for cpan/IO-Zlib directly ./miniperl -Ilib make_ext.pl cpan/IPC-Cmd/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for cpan/IPC-Cmd directly ./miniperl -Ilib make_ext.pl ext/IPC-Open3/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for ext/IPC-Open3 directly ./miniperl -Ilib make_ext.pl cpan/JSON-PP/pm_to_blib MAKE="make" LIBPERL_A=libperl.a ./miniperl -Ilib make_ext.pl dist/Locale-Maketext/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for dist/Locale-Maketext directly ./miniperl -Ilib make_ext.pl cpan/Locale-Maketext-Simple/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for cpan/Locale-Maketext-Simple directly ./miniperl -Ilib make_ext.pl cpan/Math-BigInt/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for cpan/Math-BigInt directly ./miniperl -Ilib make_ext.pl cpan/Math-BigRat/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for cpan/Math-BigRat directly ./miniperl -Ilib make_ext.pl cpan/Math-Complex/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for cpan/Math-Complex directly ./miniperl -Ilib make_ext.pl cpan/Memoize/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for cpan/Memoize directly ./miniperl -Ilib make_ext.pl dist/Module-CoreList/pm_to_blib MAKE="make" LIBPERL_A=libperl.a ./miniperl -Ilib make_ext.pl cpan/Module-Load/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for cpan/Module-Load directly ./miniperl -Ilib make_ext.pl cpan/Module-Load-Conditional/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for cpan/Module-Load-Conditional directly ./miniperl -Ilib make_ext.pl cpan/Module-Loaded/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for cpan/Module-Loaded directly ./miniperl -Ilib make_ext.pl cpan/Module-Metadata/pm_to_blib MAKE="make" LIBPERL_A=libperl.a ./miniperl -Ilib make_ext.pl cpan/NEXT/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for cpan/NEXT directly ./miniperl -Ilib make_ext.pl dist/Net-Ping/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for dist/Net-Ping directly ./miniperl -Ilib make_ext.pl cpan/Params-Check/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for cpan/Params-Check directly ./miniperl -Ilib make_ext.pl cpan/Perl-OSType/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for cpan/Perl-OSType directly ./miniperl -Ilib make_ext.pl cpan/PerlIO-via-QuotedPrint/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for cpan/PerlIO-via-QuotedPrint directly ./miniperl -Ilib make_ext.pl cpan/Pod-Checker/pm_to_blib MAKE="make" LIBPERL_A=libperl.a ./miniperl -Ilib make_ext.pl cpan/Pod-Escapes/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for cpan/Pod-Escapes directly ./miniperl -Ilib make_ext.pl cpan/Pod-Simple/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for cpan/Pod-Simple directly ./miniperl -Ilib make_ext.pl ext/Pod-Functions/pm_to_blib MAKE="make" LIBPERL_A=libperl.a ./miniperl -Ilib make_ext.pl ext/Pod-Html/pm_to_blib MAKE="make" LIBPERL_A=libperl.a ./miniperl -Ilib make_ext.pl cpan/Pod-Parser/pm_to_blib MAKE="make" LIBPERL_A=libperl.a ./miniperl -Ilib make_ext.pl cpan/Pod-Perldoc/pm_to_blib MAKE="make" LIBPERL_A=libperl.a ./miniperl -Ilib make_ext.pl cpan/Pod-Usage/pm_to_blib MAKE="make" LIBPERL_A=libperl.a ./miniperl -Ilib make_ext.pl dist/Safe/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for dist/Safe directly ./miniperl -Ilib make_ext.pl dist/Search-Dict/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for dist/Search-Dict directly ./miniperl -Ilib make_ext.pl dist/SelfLoader/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for dist/SelfLoader directly ./miniperl -Ilib make_ext.pl cpan/Term-ANSIColor/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for cpan/Term-ANSIColor directly ./miniperl -Ilib make_ext.pl cpan/Term-Cap/pm_to_blib MAKE="make" LIBPERL_A=libperl.a ./miniperl -Ilib make_ext.pl dist/Term-Complete/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for dist/Term-Complete directly ./miniperl -Ilib make_ext.pl dist/Term-ReadLine/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for dist/Term-ReadLine directly ./miniperl -Ilib make_ext.pl dist/Test/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for dist/Test directly ./miniperl -Ilib make_ext.pl cpan/Test-Harness/pm_to_blib MAKE="make" LIBPERL_A=libperl.a ./miniperl -Ilib make_ext.pl cpan/Test-Simple/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for cpan/Test-Simple directly ./miniperl -Ilib make_ext.pl dist/Text-Abbrev/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for dist/Text-Abbrev directly ./miniperl -Ilib make_ext.pl cpan/Text-Balanced/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for cpan/Text-Balanced directly ./miniperl -Ilib make_ext.pl cpan/Text-ParseWords/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for cpan/Text-ParseWords directly ./miniperl -Ilib make_ext.pl cpan/Text-Tabs/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for cpan/Text-Tabs directly ./miniperl -Ilib make_ext.pl dist/Thread-Queue/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for dist/Thread-Queue directly ./miniperl -Ilib make_ext.pl dist/Thread-Semaphore/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for dist/Thread-Semaphore directly ./miniperl -Ilib make_ext.pl dist/Tie-File/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for dist/Tie-File directly ./miniperl -Ilib make_ext.pl ext/Tie-Memoize/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for ext/Tie-Memoize directly ./miniperl -Ilib make_ext.pl cpan/Tie-RefHash/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for cpan/Tie-RefHash directly ./miniperl -Ilib make_ext.pl cpan/Time-Local/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for cpan/Time-Local directly ./miniperl -Ilib make_ext.pl dist/XSLoader/pm_to_blib MAKE="make" LIBPERL_A=libperl.a ./miniperl -Ilib make_ext.pl cpan/autodie/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for cpan/autodie directly ./miniperl -Ilib make_ext.pl dist/autouse/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for dist/autouse directly ./miniperl -Ilib make_ext.pl dist/base/pm_to_blib MAKE="make" LIBPERL_A=libperl.a ./miniperl -Ilib make_ext.pl cpan/bignum/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for cpan/bignum directly ./miniperl -Ilib make_ext.pl dist/constant/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for dist/constant directly ./miniperl -Ilib make_ext.pl dist/encoding-warnings/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for dist/encoding-warnings directly ./miniperl -Ilib make_ext.pl cpan/experimental/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for cpan/experimental directly ./miniperl -Ilib make_ext.pl dist/if/pm_to_blib MAKE="make" LIBPERL_A=libperl.a ./miniperl -Ilib make_ext.pl cpan/libnet/pm_to_blib MAKE="make" LIBPERL_A=libperl.a ./miniperl -Ilib make_ext.pl cpan/parent/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for cpan/parent directly ./miniperl -Ilib make_ext.pl cpan/perlfaq/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for cpan/perlfaq directly ./miniperl -Ilib make_ext.pl cpan/podlators/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Manifying 2 pod documents ./miniperl -Ilib make_ext.pl cpan/version/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Running pm_to_blib for cpan/version directly ./miniperl -Ilib mkppport ppport.h in cpan/DB_File is up-to-date ppport.h in cpan/IPC-SysV is up-to-date ppport.h in cpan/Win32API-File is up-to-date ppport.h in dist/IO is up-to-date ppport.h in dist/Storable is up-to-date ./miniperl -Ilib make_ext.pl DynaLoader.o MAKE="make" LIBPERL_A=libperl.a LINKTYPE=static CCCDLFLAGS= ./miniperl -Ilib make_ext.pl lib/auto/Cwd/Cwd.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic /home/jkeenan/gitwork/perl/dist/PathTools/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Cwd.bs ../../lib/auto/Cwd/Cwd.bs 644

  Making utilities ./miniperl -Ilib lib/unicore/mktables -C lib/unicore -P pod -maketest -makelist -p lib/unicore/mktables​: Files seem to be ok\, not bothering to rebuild. Add '-w' option to force build ./miniperl -Ilib make_ext.pl lib/auto/B/B.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic /home/jkeenan/gitwork/perl/ext/B/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- B.bs ../../lib/auto/B/B.bs 644 ./miniperl -Ilib make_ext.pl lib/auto/Compress/Raw/Bzip2/Bzip2.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic /home/jkeenan/gitwork/perl/cpan/Compress-Raw-Bzip2/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Bzip2.bs ../../lib/auto/Compress/Raw/Bzip2/Bzip2.bs 644 ./miniperl -Ilib make_ext.pl lib/auto/Compress/Raw/Zlib/Zlib.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic /home/jkeenan/gitwork/perl/cpan/Compress-Raw-Zlib/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Zlib.bs ../../lib/auto/Compress/Raw/Zlib/Zlib.bs 644 ./miniperl -Ilib make_ext.pl lib/auto/DB_File/DB_File.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic /home/jkeenan/gitwork/perl/cpan/DB_File/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- DB_File.bs ../../lib/auto/DB_File/DB_File.bs 644 ./miniperl -Ilib make_ext.pl lib/auto/Data/Dumper/Dumper.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic /home/jkeenan/gitwork/perl/dist/Data-Dumper/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Dumper.bs ../../lib/auto/Data/Dumper/Dumper.bs 644 ./miniperl -Ilib make_ext.pl lib/auto/Devel/PPPort/PPPort.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic /home/jkeenan/gitwork/perl/dist/Devel-PPPort/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- PPPort.bs ../../lib/auto/Devel/PPPort/PPPort.bs 644 ./miniperl -Ilib make_ext.pl lib/auto/Devel/Peek/Peek.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic /home/jkeenan/gitwork/perl/ext/Devel-Peek/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Peek.bs ../../lib/auto/Devel/Peek/Peek.bs 644 ./miniperl -Ilib make_ext.pl lib/auto/Digest/MD5/MD5.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic /home/jkeenan/gitwork/perl/cpan/Digest-MD5/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- MD5.bs ../../lib/auto/Digest/MD5/MD5.bs 644 ./miniperl -Ilib make_ext.pl lib/auto/Digest/SHA/SHA.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic /home/jkeenan/gitwork/perl/cpan/Digest-SHA/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- SHA.bs ../../lib/auto/Digest/SHA/SHA.bs 644 ./miniperl -Ilib make_ext.pl lib/auto/Encode/Encode.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic /home/jkeenan/gitwork/perl/cpan/Encode/Byte/../../../miniperl "-I../../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Byte.bs ../../../lib/auto/Encode/Byte/Byte.bs 644 /home/jkeenan/gitwork/perl/cpan/Encode/CN/../../../miniperl "-I../../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- CN.bs ../../../lib/auto/Encode/CN/CN.bs 644 /home/jkeenan/gitwork/perl/cpan/Encode/EBCDIC/../../../miniperl "-I../../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- EBCDIC.bs ../../../lib/auto/Encode/EBCDIC/EBCDIC.bs 644 /home/jkeenan/gitwork/perl/cpan/Encode/JP/../../../miniperl "-I../../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- JP.bs ../../../lib/auto/Encode/JP/JP.bs 644 /home/jkeenan/gitwork/perl/cpan/Encode/KR/../../../miniperl "-I../../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- KR.bs ../../../lib/auto/Encode/KR/KR.bs 644 /home/jkeenan/gitwork/perl/cpan/Encode/Symbol/../../../miniperl "-I../../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Symbol.bs ../../../lib/auto/Encode/Symbol/Symbol.bs 644 /home/jkeenan/gitwork/perl/cpan/Encode/TW/../../../miniperl "-I../../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- TW.bs ../../../lib/auto/Encode/TW/TW.bs 644 /home/jkeenan/gitwork/perl/cpan/Encode/Unicode/../../../miniperl "-I../../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Unicode.bs ../../../lib/auto/Encode/Unicode/Unicode.bs 644 /home/jkeenan/gitwork/perl/cpan/Encode/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Encode.bs ../../lib/auto/Encode/Encode.bs 644 ./miniperl -Ilib make_ext.pl lib/auto/Fcntl/Fcntl.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic /home/jkeenan/gitwork/perl/ext/Fcntl/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Fcntl.bs ../../lib/auto/Fcntl/Fcntl.bs 644 ./miniperl -Ilib make_ext.pl lib/auto/File/DosGlob/DosGlob.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic /home/jkeenan/gitwork/perl/ext/File-DosGlob/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- DosGlob.bs ../../lib/auto/File/DosGlob/DosGlob.bs 644 ./miniperl -Ilib make_ext.pl lib/auto/File/Glob/Glob.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic /home/jkeenan/gitwork/perl/ext/File-Glob/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Glob.bs ../../lib/auto/File/Glob/Glob.bs 644 ./miniperl -Ilib make_ext.pl lib/auto/Filter/Util/Call/Call.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic /home/jkeenan/gitwork/perl/cpan/Filter-Util-Call/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Call.bs ../../lib/auto/Filter/Util/Call/Call.bs 644 ./miniperl -Ilib make_ext.pl lib/auto/GDBM_File/GDBM_File.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic /home/jkeenan/gitwork/perl/ext/GDBM_File/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- GDBM_File.bs ../../lib/auto/GDBM_File/GDBM_File.bs 644 ./miniperl -Ilib make_ext.pl lib/auto/Hash/Util/Util.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic /home/jkeenan/gitwork/perl/ext/Hash-Util/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Util.bs ../../lib/auto/Hash/Util/Util.bs 644 ./miniperl -Ilib make_ext.pl lib/auto/Hash/Util/FieldHash/FieldHash.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic /home/jkeenan/gitwork/perl/ext/Hash-Util-FieldHash/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- FieldHash.bs ../../lib/auto/Hash/Util/FieldHash/FieldHash.bs 644 ./miniperl -Ilib make_ext.pl lib/auto/I18N/Langinfo/Langinfo.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic /home/jkeenan/gitwork/perl/ext/I18N-Langinfo/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Langinfo.bs ../../lib/auto/I18N/Langinfo/Langinfo.bs 644 ./miniperl -Ilib make_ext.pl lib/auto/IO/IO.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic /home/jkeenan/gitwork/perl/dist/IO/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- IO.bs ../../lib/auto/IO/IO.bs 644 ./miniperl -Ilib make_ext.pl lib/auto/IPC/SysV/SysV.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic /home/jkeenan/gitwork/perl/cpan/IPC-SysV/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- SysV.bs ../../lib/auto/IPC/SysV/SysV.bs 644 ./miniperl -Ilib make_ext.pl lib/auto/List/Util/Util.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic /home/jkeenan/gitwork/perl/cpan/Scalar-List-Utils/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Util.bs ../../lib/auto/List/Util/Util.bs 644 ./miniperl -Ilib make_ext.pl lib/auto/MIME/Base64/Base64.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic /home/jkeenan/gitwork/perl/cpan/MIME-Base64/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Base64.bs ../../lib/auto/MIME/Base64/Base64.bs 644 ./miniperl -Ilib make_ext.pl lib/auto/Math/BigInt/FastCalc/FastCalc.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic /home/jkeenan/gitwork/perl/cpan/Math-BigInt-FastCalc/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- FastCalc.bs ../../lib/auto/Math/BigInt/FastCalc/FastCalc.bs 644 ./miniperl -Ilib make_ext.pl lib/auto/NDBM_File/NDBM_File.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic /home/jkeenan/gitwork/perl/ext/NDBM_File/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- NDBM_File.bs ../../lib/auto/NDBM_File/NDBM_File.bs 644 ./miniperl -Ilib make_ext.pl lib/auto/Opcode/Opcode.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic /home/jkeenan/gitwork/perl/ext/Opcode/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Opcode.bs ../../lib/auto/Opcode/Opcode.bs 644 ./miniperl -Ilib make_ext.pl lib/auto/POSIX/POSIX.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic /home/jkeenan/gitwork/perl/ext/POSIX/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- POSIX.bs ../../lib/auto/POSIX/POSIX.bs 644 ./miniperl -Ilib make_ext.pl lib/auto/PerlIO/encoding/encoding.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic /home/jkeenan/gitwork/perl/ext/PerlIO-encoding/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- encoding.bs ../../lib/auto/PerlIO/encoding/encoding.bs 644 ./miniperl -Ilib make_ext.pl lib/auto/PerlIO/mmap/mmap.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic /home/jkeenan/gitwork/perl/ext/PerlIO-mmap/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- mmap.bs ../../lib/auto/PerlIO/mmap/mmap.bs 644 ./miniperl -Ilib make_ext.pl lib/auto/PerlIO/scalar/scalar.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic /home/jkeenan/gitwork/perl/ext/PerlIO-scalar/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- scalar.bs ../../lib/auto/PerlIO/scalar/scalar.bs 644 ./miniperl -Ilib make_ext.pl lib/auto/PerlIO/via/via.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic /home/jkeenan/gitwork/perl/ext/PerlIO-via/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- via.bs ../../lib/auto/PerlIO/via/via.bs 644 ./miniperl -Ilib make_ext.pl lib/auto/SDBM_File/SDBM_File.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic /home/jkeenan/gitwork/perl/ext/SDBM_File/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- SDBM_File.bs ../../lib/auto/SDBM_File/SDBM_File.bs 644 ./miniperl -Ilib make_ext.pl lib/auto/Socket/Socket.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic /home/jkeenan/gitwork/perl/cpan/Socket/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Socket.bs ../../lib/auto/Socket/Socket.bs 644 ./miniperl -Ilib make_ext.pl lib/auto/Storable/Storable.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic /home/jkeenan/gitwork/perl/dist/Storable/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Storable.bs ../../lib/auto/Storable/Storable.bs 644 ./miniperl -Ilib make_ext.pl lib/auto/Sys/Hostname/Hostname.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic /home/jkeenan/gitwork/perl/ext/Sys-Hostname/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Hostname.bs ../../lib/auto/Sys/Hostname/Hostname.bs 644 ./miniperl -Ilib make_ext.pl lib/auto/Sys/Syslog/Syslog.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic /home/jkeenan/gitwork/perl/cpan/Sys-Syslog/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Syslog.bs ../../lib/auto/Sys/Syslog/Syslog.bs 644 ./miniperl -Ilib make_ext.pl lib/auto/Tie/Hash/NamedCapture/NamedCapture.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic /home/jkeenan/gitwork/perl/ext/Tie-Hash-NamedCapture/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- NamedCapture.bs ../../lib/auto/Tie/Hash/NamedCapture/NamedCapture.bs 644 ./miniperl -Ilib make_ext.pl lib/auto/Time/HiRes/HiRes.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic /home/jkeenan/gitwork/perl/dist/Time-HiRes/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- HiRes.bs ../../lib/auto/Time/HiRes/HiRes.bs 644 ./miniperl -Ilib make_ext.pl lib/auto/Time/Piece/Piece.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic /home/jkeenan/gitwork/perl/cpan/Time-Piece/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Piece.bs ../../lib/auto/Time/Piece/Piece.bs 644 ./miniperl -Ilib make_ext.pl lib/auto/Unicode/Collate/Collate.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic /home/jkeenan/gitwork/perl/cpan/Unicode-Collate/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Collate.bs ../../lib/auto/Unicode/Collate/Collate.bs 644 ./miniperl -Ilib make_ext.pl lib/auto/Unicode/Normalize/Normalize.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic /home/jkeenan/gitwork/perl/dist/Unicode-Normalize/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Normalize.bs ../../lib/auto/Unicode/Normalize/Normalize.bs 644 ./miniperl -Ilib make_ext.pl lib/auto/XS/APItest/APItest.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic ./miniperl -Ilib make_ext.pl lib/auto/XS/Typemap/Typemap.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic /home/jkeenan/gitwork/perl/ext/XS-Typemap/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- Typemap.bs ../../lib/auto/XS/Typemap/Typemap.bs 644 ./miniperl -Ilib make_ext.pl lib/auto/attributes/attributes.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic /home/jkeenan/gitwork/perl/ext/attributes/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- attributes.bs ../../lib/auto/attributes/attributes.bs 644 ./miniperl -Ilib make_ext.pl lib/auto/mro/mro.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic /home/jkeenan/gitwork/perl/ext/mro/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- mro.bs ../../lib/auto/mro/mro.bs 644 ./miniperl -Ilib make_ext.pl lib/auto/re/re.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic /home/jkeenan/gitwork/perl/ext/re/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- re.bs ../../lib/auto/re/re.bs 644 ./miniperl -Ilib make_ext.pl lib/auto/threads/threads.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic /home/jkeenan/gitwork/perl/dist/threads/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- threads.bs ../../lib/auto/threads/threads.bs 644 ./miniperl -Ilib make_ext.pl lib/auto/threads/shared/shared.so MAKE="make" LIBPERL_A=libperl.a LINKTYPE=dynamic /home/jkeenan/gitwork/perl/dist/threads-shared/../../miniperl "-I../../lib" -MExtUtils​::Command​::MM -e 'cp_nonempty' -- shared.bs ../../lib/auto/threads/shared/shared.bs 644 ./perl -Ilib -I. -f pod/buildtoc -q cd t && (rm -f perl; /bin/ln -s ../perl perl) TESTFILE=harness ./runtests choose No saved state\, selection will be empty base/cond.t ........................................................ ok base/if.t .......................................................... ok base/lex.t ......................................................... ok base/num.t ......................................................... ok base/pat.t ......................................................... ok base/rs.t .......................................................... ok base/term.t ........................................................ ok base/translate.t ................................................... ok base/while.t ....................................................... ok comp/bproto.t ...................................................... ok comp/cmdopt.t ...................................................... ok comp/colon.t ....................................................... ok comp/decl.t ........................................................ ok comp/filter_exception.t ............................................ ok comp/final_line_num.t .............................................. ok comp/form_scope.t .................................................. ok comp/fold.t ........................................................ ok comp/line_debug.t .................................................. ok comp/hints.t ....................................................... ok comp/multiline.t ................................................... ok comp/opsubs.t ...................................................... ok comp/our.t ......................................................... ok comp/package.t ..................................................... ok comp/package_block.t ............................................... ok comp/parser.t ...................................................... ok comp/parser_run.t .................................................. ok comp/redef.t ....................................................... ok comp/proto.t ....................................................... ok comp/retainedlines.t ............................................... ok comp/require.t ..................................................... ok comp/term.t ........................................................ ok comp/uproto.t ...................................................... ok comp/use.t ......................................................... ok comp/utf.t ......................................................... ok run/cloexec.t ...................................................... ok run/dtrace.t ....................................................... skipped​: no dtrace run/exit.t ......................................................... ok run/fresh_perl.t ................................................... ok run/noswitch.t ..................................................... ok run/runenv.t ....................................................... ok run/locale.t ....................................................... ok run/switch0.t ...................................................... ok run/script.t ....................................................... ok run/switcha.t ...................................................... ok run/switchC.t ...................................................... ok run/switchd-78586.t ................................................ ok run/switchd.t ...................................................... ok run/switchDx.t ..................................................... skipped​: DEBUGGING build required run/switchF.t ...................................................... ok run/switches.t ..................................................... ok run/switchF1.t ..................................................... ok run/switchF2.t ..................................................... ok run/switchI.t ...................................................... ok run/switchM.t ...................................................... ok run/switchn.t ...................................................... ok run/switchp.t ...................................................... ok run/switcht.t ...................................................... ok run/switchx.t ...................................................... ok cmd/elsif.t ........................................................ ok cmd/for.t .......................................................... ok cmd/mod.t .......................................................... ok cmd/subval.t ....................................................... ok cmd/switch.t ....................................................... ok io/argv.t .......................................................... ok io/binmode.t ....................................................... ok io/bom.t ........................................................... ok io/closepid.t ...................................................... ok io/crlf.t .......................................................... ok io/data.t .......................................................... ok io/defout.t ........................................................ ok io/crlf_through.t .................................................. ok io/eintr.t ......................................................... skipped​: various portability issues io/dup.t ........................................................... ok io/eintr_print.t ................................................... skipped​: various portability issues io/errno.t ......................................................... ok io/fflush.t ........................................................ ok io/fs.t ............................................................ ok io/errnosig.t ...................................................... ok io/inplace.t ....................................................... ok io/iofile.t ........................................................ ok io/layers.t ........................................................ ok io/iprefix.t ....................................................... ok io/nargv.t ......................................................... ok io/openpid.t ....................................................... ok io/open.t .......................................................... ok io/paragraph_mode.t ................................................ ok io/perlio.t ........................................................ ok io/perlio_fail.t ................................................... ok io/perlio_leaks.t .................................................. ok io/perlio_open.t ................................................... ok io/pipe.t .......................................................... ok io/print.t ......................................................... ok io/pvbm.t .......................................................... ok io/read.t .......................................................... ok io/say.t ........................................................... ok io/sem.t ........................................................... ok io/semctl.t ........................................................ ok io/shm.t ........................................................... ok io/socket.t ........................................................ ok io/tell.t .......................................................... ok io/socketpair.t .................................................... ok io/utf8.t .......................................................... ok re/alpha_assertions.t .............................................. ok io/through.t ....................................................... ok re/charset.t ....................................................... ok # Test process timed out - terminating re/fold_grind_8.t .................................................. All 9 subtests passed # Test process timed out - terminating re/fold_grind_a.t .................................................. All 9 subtests passed # Test process timed out - terminating re/fold_grind_aa.t ................................................. All 9 subtests passed # Test process timed out - terminating re/fold_grind_d.t .................................................. All 9 subtests passed # Test process timed out - terminating re/fold_grind_l.t .................................................. All 12 subtests passed re/fold_grind_T.t .................................................. skipped​: Couldn't find a UTF-8 turkic locale # Test process timed out - terminating re/fold_grind_u.t .................................................. All 9 subtests passed re/keep_tabs.t ..................................................... ok re/no_utf8_pm.t .................................................... ok re/overload.t ...................................................... ok Killed *** Error code 137

Stop. make​: stopped in /home/jkeenan/gitwork/perl

[p5pRT]

From @iabyn

On Tue\, Apr 02\, 2019 at 02​:31​:16PM -0700\, James E Keenan via RT wrote​:

Now\, I concede​: (a) I've never previously attempted to build with AddressSanitizer; (b) This VM may be too small to run a perl built with AddressSanitizer.

Address Sanitizer needs lots of memory. My 16Gb system struggles to run 10 tests in parallel when compiled with ASan.

Just t/re/pat_psycho.t consumes about 1.7Gb on a threaded debugging build.

-- 31 December 1661​: "I have newly taken a solemne oath about abstaining from plays". 1 January 1662​: "And after ... we went by coach to the play".   -- The Diary of Samuel Pepys

[p5pRT]

From @jkeenan

On Tue\, 02 Apr 2019 21​:31​:16 GMT\, jkeenan wrote​:

On Thu\, 28 Mar 2019 15​:15​:50 GMT\, jkeenan@​pobox.com wrote​:

On 3/28/19 10​:16 AM\, Karl Williamson wrote​:

On 3/28/19 6​:21 AM\, James E Keenan wrote​:

On 3/27/19 11​:15 PM\, karl williamson via RT wrote​:

On 3/25/19 6​:45 PM\, James E Keenan (via RT) wrote​:

# New Ticket Created by  James E Keenan # Please include the string​:  [perl #133958] # in the subject line of all future correspondence about this issue. # \<URL​: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=133958 >

We are experiencing massive test failures in our smoke-testing of the Perl 5 core distribution on FreeBSD-13-CURRENT.  I am filing two separate bug tickets for these failures\, since one failure appeared long before the others.  This is the first of the two tickets.

cpan/Test-Simple/t/Legacy/More.t is repeatedly failing during smoke-testing of non-threaded builds on FreeBSD-13.  See\, e.g.​:

http​://perl5.test-smoke.org/report/82470 http​://perl5.test-smoke.org/report/82946

Test failures​: ~~ ../cpan/Test-Simple/t/Legacy/More.t ......................... FAILED 54 Non-zero exit status​: 1      [stdio] -Dcc=clang++ -Duse64bitall      [stdio] -Dcc=clang++      [stdio] -Dcc=clang++ DEBUGGING

We only have one regular smoke-testing rig set up for FreeBSD- 13\, that of course being one of Carlos Guevara's rigs.  These failures​:

i. Do not seem to occur on threaded builds.

ii. Seem to occur on most\, but not all\, non-threaded builds.

iii. Appear to have first reported during a smoke-test in http​://perl5.test-smoke.org/report/79958\, which suggests that the failures were introduced in\, or shortly before commit https://perl5.git.perl.org/perl.git/commitdiff/8021814b48.  That commit is in a deleted branch​: smoke-me/khw-vlb.

iv. Both Carlos and I have attempted to reproduce this failure manually -- but it only seems to crop up during smoke-testing.

v. Here's the end of the file which is failing​:

##### $ tail cpan/Test-Simple/t/Legacy/More.t             );

# rt.cpan.org 53469  is_deeply with regexes is_deeply( qr/a/\, qr/a/\, "same regex" );

# These two tests must remain at the end. is( $@​\, $Err\,               '$@​ untouched' ); cmp_ok( $!\, '=='\, $Errno\,   '$! untouched' ); #####

I have since built perl5 blead on FreeBSD-13 (i) unthreaded; (ii) without AddressSanitizer; (iii) $TEST_JOBS=1. In this manner​:

1. In 'make test_harness' the last test in ../cpan/Test-Simple/t/Legacy/More.t FAILs.

##### ../cpan/Test-Simple/t/Legacy/More.t (Wstat​: 256 Tests​: 54 Failed​: 1)   Failed test​: 54   Non-zero exit status​: 1 #####

2. However\, when\, in the same build\, I run that file manually\, that file PASSes.

##### $ cd t;./perl harness ../cpan/Test-Simple/t/Legacy/More.t; cd - ../cpan/Test-Simple/t/Legacy/More.t .. ok
All tests successful. Files=1\, Tests=54\, 0 wallclock secs ( 0.02 usr 0.01 sys + 0.09 cusr 0.04 csys = 0.15 CPU) Result​: PASS #####

The above was built/tested at​:

##### commit a719c52dc174d7df309b8c7a4bf318e60fc704e0 (HEAD -> blead\, origin/blead\, origin/HEAD) Author​: Tomasz Konojacki \me@&#8203;xenu\.pl AuthorDate​: Tue Apr 9 23​:15​:41 2019 +0200 Commit​: Steve Hay \steve\.m\.hay@&#8203;googlemail\.com CommitDate​: Wed Apr 10 22​:52​:21 2019 +0100 #####

But when I build *threaded*\, the test passes during 'make test_harness'.

-- James E Keenan (jkeenan@​cpan.org)

[p5pRT]

From @khwilliamson

On 4/11/19 7​:57 PM\, James E Keenan via RT wrote​:

On Tue\, 02 Apr 2019 21​:31​:16 GMT\, jkeenan wrote​:

On Thu\, 28 Mar 2019 15​:15​:50 GMT\, jkeenan@​pobox.com wrote​:

On 3/28/19 10​:16 AM\, Karl Williamson wrote​:

On 3/28/19 6​:21 AM\, James E Keenan wrote​:

On 3/27/19 11​:15 PM\, karl williamson via RT wrote​:

On 3/25/19 6​:45 PM\, James E Keenan (via RT) wrote​:

# New Ticket Created by  James E Keenan # Please include the string​:  [perl #133958] # in the subject line of all future correspondence about this issue. # \<URL​: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=133958 >

We are experiencing massive test failures in our smoke-testing of the Perl 5 core distribution on FreeBSD-13-CURRENT.  I am filing two separate bug tickets for these failures\, since one failure appeared long before the others.  This is the first of the two tickets.

cpan/Test-Simple/t/Legacy/More.t is repeatedly failing during smoke-testing of non-threaded builds on FreeBSD-13.  See\, e.g.​:

http​://perl5.test-smoke.org/report/82470 http​://perl5.test-smoke.org/report/82946

Test failures​: ~~ ../cpan/Test-Simple/t/Legacy/More.t ......................... FAILED 54 Non-zero exit status​: 1      [stdio] -Dcc=clang++ -Duse64bitall      [stdio] -Dcc=clang++      [stdio] -Dcc=clang++ DEBUGGING

We only have one regular smoke-testing rig set up for FreeBSD- 13\, that of course being one of Carlos Guevara's rigs.  These failures​:

i. Do not seem to occur on threaded builds.

ii. Seem to occur on most\, but not all\, non-threaded builds.

iii. Appear to have first reported during a smoke-test in http​://perl5.test-smoke.org/report/79958\, which suggests that the failures were introduced in\, or shortly before commit https://perl5.git.perl.org/perl.git/commitdiff/8021814b48.  That commit is in a deleted branch​: smoke-me/khw-vlb.

iv. Both Carlos and I have attempted to reproduce this failure manually -- but it only seems to crop up during smoke-testing.

v. Here's the end of the file which is failing​:

##### $ tail cpan/Test-Simple/t/Legacy/More.t             );

# rt.cpan.org 53469  is_deeply with regexes is_deeply( qr/a/\, qr/a/\, "same regex" );

# These two tests must remain at the end. is( $@​\, $Err\,               '$@​ untouched' ); cmp_ok( $!\, '=='\, $Errno\,   '$! untouched' ); #####

I have since built perl5 blead on FreeBSD-13 (i) unthreaded; (ii) without AddressSanitizer; (iii) $TEST_JOBS=1. In this manner​:

1. In 'make test_harness' the last test in ../cpan/Test-Simple/t/Legacy/More.t FAILs.

##### ../cpan/Test-Simple/t/Legacy/More.t (Wstat​: 256 Tests​: 54 Failed​: 1) Failed test​: 54 Non-zero exit status​: 1 #####

2. However\, when\, in the same build\, I run that file manually\, that file PASSes.

##### $ cd t;./perl harness ../cpan/Test-Simple/t/Legacy/More.t; cd - ../cpan/Test-Simple/t/Legacy/More.t .. ok All tests successful. Files=1\, Tests=54\, 0 wallclock secs ( 0.02 usr 0.01 sys + 0.09 cusr 0.04 csys = 0.15 CPU) Result​: PASS #####

The above was built/tested at​:

##### commit a719c52dc174d7df309b8c7a4bf318e60fc704e0 (HEAD -> blead\, origin/blead\, origin/HEAD) Author​: Tomasz Konojacki \me@&#8203;xenu\.pl AuthorDate​: Tue Apr 9 23​:15​:41 2019 +0200 Commit​: Steve Hay \steve\.m\.hay@&#8203;googlemail\.com CommitDate​: Wed Apr 10 22​:52​:21 2019 +0100 #####

But when I build *threaded*\, the test passes during 'make test_harness'.

One possibility is to bisect this. You don't have that big a range of commits that it could be IIUC. It should be possible to construct a bisect that runs a whole 'make test'

[p5pRT]

From @jkeenan

On Fri\, 12 Apr 2019 15​:52​:53 GMT\, public@​khwilliamson.com wrote​:

One possibility is to bisect this. You don't have that big a range of commits that it could be IIUC. It should be possible to construct a bisect that runs a whole 'make test'

We don't have a value for the '--target' switch to Porting/bisect.pl that would evaluate the overall result of 'make test' or 'make test_harness'. I suppose that we could create such a target\, but that's going to require many hours of work and more brain power than I currently have.

I did\, however\, try to bisect manually -- but before I get to that\, let's recall the original problem. We have​:

* A new test failure in a very old\, cpan-upstream\, test. * The error only occurs on the development version of one   operating system. * The error only occurs on unthreaded builds. * The error never seems to appear when the test file is run   by itself. * The error appears frequently\, but nonetheless   intermittently\, when run in test harnesses\, such as during   smoke-testing.

In addition\, it appears that when the error does appear\, it looks like this​:

##### cpan/Test-Simple/t/Legacy/More ...# Failed test '$! untouched' # at t/Legacy/More.t line 184. # got​: 12 # expected​: 42 # Looks like you failed 1 test of 54. FAILED at test 54 #####

Since the error is intermittent\, it's quite likely that a single bisection run will be unable to pinpoint the first bad commit in a reproducible manner. Hence\, it's quite likely that the best we can do is to identify a short range of commits in which the error first becomes apparent.

So\, over the past few days\, I have built and run the test suite on an unthreaded perl on FreeBSD-13 several dozen times. I identified all the commits between v5.29.8 and v5.29.9. To make a long story short\, I believe that the FAIL first appeared in one of these two adjacent commits​:

##### commit 3b89859ad83deaeff7c1ee7911d181ef10236879 Author​: Karl Williamson \khw@&#8203;cpan\.org AuthorDate​: Sun Mar 17 21​:59​:52 2019 -0600 Commit​: Karl Williamson \khw@&#8203;cpan\.org CommitDate​: Sun Mar 17 22​:17​:28 2019 -0600

regcomp.c​: Use mnemonic for flag parameter

### commit 6ef7fe531911e0b41ffcc04c1d6b6ec25a8b1bc9 Author​: Karl Williamson \khw@&#8203;cpan\.org AuthorDate​: Sun Mar 17 22​:11​:04 2019 -0600 Commit​: Karl Williamson \khw@&#8203;cpan\.org CommitDate​: Sun Mar 17 22​:17​:29 2019 -0600

PATCH​: [perl #131551] Too deep regex compilation recursion

This patch\, started by Yves Orton\, and refined in consultation with Tony Cook\, imposes a maximum depth of unclosed left parentheses\, at which point it croaks. This is to prevent the segfault in the ticket.

The patch adds a variable that can be set to increase or decrease this limit at run time (actually regex compilation time) should this be desired\, and hence our pre-determined limit of 1000 can be changed if necessary. #####

Why cannot I not say which of these was the first bad commit? My bisection process first identified 6ef7fe531911e0b41ffcc04c1d6b6ec25a8b1bc9 as the first bad commit. Then I said to $self\, let's verify this by running 'make test' at the previous commit and this one. I started to get the failure at the first of the two commits. (I should note that the two commits entered the repository only a second apart\, so they were probably part of the same 'git push'.)

I don't understand the internals that were modified in those two commits\, so I'm still at a loss as to explain these intermittent failures -- especially the fact that they only occur on non-threaded builds.

The problem persists. See\, e.g.\, http​://perl5.test-smoke.org/report/85075.

Thank you very much.

-- James E Keenan (jkeenan@​cpan.org)

[p5pRT]

From @khwilliamson

On 4/14/19 1​:59 PM\, James E Keenan via RT wrote​:

On Fri\, 12 Apr 2019 15​:52​:53 GMT\, public@​khwilliamson.com wrote​:

One possibility is to bisect this. You don't have that big a range of commits that it could be IIUC. It should be possible to construct a bisect that runs a whole 'make test'

We don't have a value for the '--target' switch to Porting/bisect.pl that would evaluate the overall result of 'make test' or 'make test_harness'. I suppose that we could create such a target\, but that's going to require many hours of work and more brain power than I currently have.

We could make a shell script that called make test and then returned appropriately. I could help out devising that should it become necessary.

I did\, however\, try to bisect manually -- but before I get to that\, let's recall the original problem. We have​:

* A new test failure in a very old\, cpan-upstream\, test. * The error only occurs on the development version of one operating system. * The error only occurs on unthreaded builds. * The error never seems to appear when the test file is run by itself. * The error appears frequently\, but nonetheless intermittently\, when run in test harnesses\, such as during smoke-testing.

In addition\, it appears that when the error does appear\, it looks like this​:

##### cpan/Test-Simple/t/Legacy/More ...# Failed test '$! untouched' # at t/Legacy/More.t line 184. # got​: 12 # expected​: 42 # Looks like you failed 1 test of 54. FAILED at test 54 #####

Since the error is intermittent\, it's quite likely that a single bisection run will be unable to pinpoint the first bad commit in a reproducible manner. Hence\, it's quite likely that the best we can do is to identify a short range of commits in which the error first becomes apparent.

So\, over the past few days\, I have built and run the test suite on an unthreaded perl on FreeBSD-13 several dozen times. I identified all the commits between v5.29.8 and v5.29.9. To make a long story short\, I believe that the FAIL first appeared in one of these two adjacent commits​:

##### commit 3b89859ad83deaeff7c1ee7911d181ef10236879 Author​: Karl Williamson \khw@&#8203;cpan\.org AuthorDate​: Sun Mar 17 21​:59​:52 2019 -0600 Commit​: Karl Williamson \khw@&#8203;cpan\.org CommitDate​: Sun Mar 17 22​:17​:28 2019 -0600

regcomp.c​: Use mnemonic for flag parameter

### commit 6ef7fe531911e0b41ffcc04c1d6b6ec25a8b1bc9 Author​: Karl Williamson \khw@&#8203;cpan\.org AuthorDate​: Sun Mar 17 22​:11​:04 2019 -0600 Commit​: Karl Williamson \khw@&#8203;cpan\.org CommitDate​: Sun Mar 17 22​:17​:29 2019 -0600

PATCH​: [perl #131551] Too deep regex compilation recursion

This patch\, started by Yves Orton\, and refined in consultation with Tony Cook\, imposes a maximum depth of unclosed left parentheses\, at which point it croaks. This is to prevent the segfault in the ticket.

The patch adds a variable that can be set to increase or decrease this limit at run time (actually regex compilation time) should this be desired\, and hence our pre-determined limit of 1000 can be changed if necessary. #####

Why cannot I not say which of these was the first bad commit? My bisection process first identified 6ef7fe531911e0b41ffcc04c1d6b6ec25a8b1bc9 as the first bad commit. Then I said to $self\, let's verify this by running 'make test' at the previous commit and this one. I started to get the failure at the first of the two commits. (I should note that the two commits entered the repository only a second apart\, so they were probably part of the same 'git push'.)

I don't understand the internals that were modified in those two commits\, so I'm still at a loss as to explain these intermittent failures -- especially the fact that they only occur on non-threaded builds.

The problem persists. See\, e.g.\, http​://perl5.test-smoke.org/report/85075.

It's much more likely to be 6ef7fe531911e0b41ffcc04c1d6b6ec25a8b1bc9. It seems to me that we should revert it\, and see if the problem goes away. If it doesn't revert cleanly\, let me know\, and I'll set up a branch for you with it reverted. And if the bug persists revert the other one. That should tell us if it's really one of these two

Thank you very much.

[p5pRT]

From @jkeenan

On Sun\, 14 Apr 2019 20​:22​:59 GMT\, public@​khwilliamson.com wrote​: [snip]

It's much more likely to be 6ef7fe531911e0b41ffcc04c1d6b6ec25a8b1bc9. It seems to me that we should revert it\, and see if the problem goes away. If it doesn't revert cleanly\, let me know\, and I'll set up a branch for you with it reverted. And if the bug persists revert the other one. That should tell us if it's really one of these two

The branch idea sounds good\, as Carlos's smokers test branches. But\, as you anticipated\, a simple 'git revert' yields conflicts. See http​://paste.scsys.co.uk/583957. If you let me know how the conflicts should be resolved\, I'll push the branch.

Thank you very much.

-- James E Keenan (jkeenan@​cpan.org)

[p5pRT]

From @jkeenan

On Sun\, 14 Apr 2019 20​:22​:59 GMT\, public@​khwilliamson.com wrote​: [snip]

It's much more likely to be 6ef7fe531911e0b41ffcc04c1d6b6ec25a8b1bc9. It seems to me that we should revert it\, and see if the problem goes away. If it doesn't revert cleanly\, let me know\, and I'll set up a branch for you with it reverted. And if the bug persists revert the other one. That should tell us if it's really one of these two

Unfortunately\, reverting the second commit did not clear up the FAIL​:

##### smoke-me/jkeenan/rt133958-revert-6ef7fe53 ##### ../cpan/Test-Simple/t/Legacy/Bugs/600.t ............................ ok ===( 157250;770 1/54 0/? )========================================== # Failed test '$! untouched' # at t/Legacy/More.t line 184. # got​: 12 # expected​: 42 # Looks like you failed 1 test of 54. ../cpan/Test-Simple/t/Legacy/More.t ................................ Dubious\, test returned 1 (wstat 256\, 0x100) Failed 1/54 subtests #####

I will now try to revert only the first of the two commits in question.

Thank you very much. -- James E Keenan (jkeenan@​cpan.org)

[p5pRT]

From @iabyn

On Sun\, Apr 14\, 2019 at 03​:49​:22PM -0700\, James E Keenan via RT wrote​:

Unfortunately\, reverting the second commit did not clear up the FAIL​:

##### smoke-me/jkeenan/rt133958-revert-6ef7fe53 ##### ../cpan/Test-Simple/t/Legacy/Bugs/600.t ............................ ok ===( 157250;770 1/54 0/? )========================================== # Failed test '$! untouched' # at t/Legacy/More.t line 184. # got​: 12 # expected​: 42 # Looks like you failed 1 test of 54. ../cpan/Test-Simple/t/Legacy/More.t ................................ Dubious\, test returned 1 (wstat 256\, 0x100) Failed 1/54 subtests #####

The test is failing because the process's errno ($!) has not being restored to its original value at the end of test script. It would be interesting to see what the error code means. Can you compile and execute this program on the failing host?

  #include \<stdio.h>   #include \<errno.h>

  int main(int argc\, char**argv)   {

  errno = 12;   perror("this is the error​:");   return 0;   }

Failing that\, you should be able to find error code 12 in the file /usr/include/errno.h\, or in one of the files '#include'd from there. For example on linux\, I end up at /usr/include/asm-generic/errno-base.h with the line

  #define ENOMEM 12 /* Out of memory */

It's possible that linux and BSD share the same basic error code numbers\, in which case the smoke may be intermittently failing due to running out of memory. Which is odd\, as that test only takes 16Mb resident on my linux DEBUGGING build\, compared with 34Mb resident for t/re/pat_psycho.t for example (which doesn't appear to be frequently failing).

-- O Unicef Clearasil! Gibberish and Drivel!   -- "Bored of the Rings"

[p5pRT]

From @jkeenan

On Wed\, 24 Apr 2019 14​:23​:06 GMT\, davem wrote​:

On Sun\, Apr 14\, 2019 at 03​:49​:22PM -0700\, James E Keenan via RT wrote​:

Unfortunately\, reverting the second commit did not clear up the FAIL​:

##### smoke-me/jkeenan/rt133958-revert-6ef7fe53 ##### ../cpan/Test-Simple/t/Legacy/Bugs/600.t ............................ ok ===( 157250;770 1/54 0/? )========================================== # Failed test '$! untouched' # at t/Legacy/More.t line 184. # got​: 12 # expected​: 42 # Looks like you failed 1 test of 54. ../cpan/Test-Simple/t/Legacy/More.t ................................ Dubious\, test returned 1 (wstat 256\, 0x100) Failed 1/54 subtests #####

The test is failing because the process's errno ($!) has not being restored to its original value at the end of test script. It would be interesting to see what the error code means. Can you compile and execute this program on the failing host?

#include \<stdio.h> #include \<errno.h>

int main(int argc\, char**argv) {

errno = 12; perror("this is the error​:"); return 0; }

Failing that\, you should be able to find error code 12 in the file /usr/include/errno.h\, or in one of the files '#include'd from there. For example on linux\, I end up at /usr/include/asm-generic/errno- base.h with the line

#define ENOMEM 12 /* Out of memory */

It's possible that linux and BSD share the same basic error code numbers\, in which case the smoke may be intermittently failing due to running out of memory. Which is odd\, as that test only takes 16Mb resident on my linux DEBUGGING build\, compared with 34Mb resident for t/re/pat_psycho.t for example (which doesn't appear to be frequently failing).

On FreeBSD-13​:

##### [c] $ uname -mrs FreeBSD 13.0-CURRENT amd64

[c] $ cat 133958-errno.c #include \<stdio.h> #include \<errno.h>

int main(int argc\, char**argv) {

  errno = 12;   perror("this is the error​:");   return 0; }

[c] $ cc --version FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on LLVM 6.0.1) Target​: x86_64-unknown-freebsd13.0 Thread model​: posix InstalledDir​: /usr/bin

[c] $ cc 133958-errno.c -o 133958-errno

[c] $ ./133958-errno this is the error​:: Cannot allocate memory

[c] $ cd /usr/local/include

[include] $ ack '\t(12|42)\t' /usr/include/sys/errno.h #define ENOMEM 12 /* Cannot allocate memory */ #define ENOPROTOOPT 42 /* Protocol not available */ #####

On Linux​:

##### $ ack '\t(12|42)\t' /usr/include/asm-generic/errno-base.h #define ENOMEM 12 /* Out of memory */ #####

So there is no error '42' on Linux -- or at least not in /usr/include/asm-generic/errno-base.h.

I wonder if\, when cpan/Test-Simple/t/Legacy/More.t\, was added back in 2002\, '42' was considered to be a sufficiently "high" error number as to be suitable for use in testing as an impossible-to-achieve value.

Note also that error number 42 is not a new one for FreeBSD-13. On FreeBSD-11.2​:

##### [jkeenan] $ uname -mrs FreeBSD 11.2-STABLE amd64 [jkeenan] $ ack '\t(12|42)\t' /usr/include/sys/errno.h #define ENOMEM 12 /* Cannot allocate memory */ #define ENOPROTOOPT 42 /* Protocol not available */ ##### -- James E Keenan (jkeenan@​cpan.org)

[p5pRT]

From @iabyn

On Wed\, Apr 24\, 2019 at 08​:10​:19AM -0700\, James E Keenan via RT wrote​:

So there is no error '42' on Linux -- or at least not in /usr/include/asm-generic/errno-base.h.

I wonder if\, when cpan/Test-Simple/t/Legacy/More.t\, was added back in 2002\, '42' was considered to be a sufficiently "high" error number as to be suitable for use in testing as an impossible-to-achieve value.

Note also that error number 42 is not a new one for FreeBSD-13. On FreeBSD-11.2​:

The error number 42 is irrelevant. It's just an arbitrary but humorous number that $! is set to at the start of the script\, in order to see that it doesn't get changed during the course of the script (i.e. test that Test​::More etc localise $! when doing their stuff.)

The fact that its getting changed to ENOMEM is more interesting. This might indicate a lack of resources on the testing machine\, and would also explain its intermittency.

However\, experimenting with 'ulimit -d'\, as I decrease the available memory\, I see the script usually dying after test 51\, never 54. Which might instead indicate something off with BSD and the setting of errno.

I'm only seeing this test script fail on one smoker\,   freebsd 13.0-CURRENT (amd64/1 cpu) and its only been failing since around v5.29.8-61-gb5e20476d6

Test-Simple was upgraded in v5.29.7-102-ga6afdf72cd

[p5pRT]

From @jkeenan

On Thu\, 25 Apr 2019 08​:13​:03 GMT\, davem wrote​:

On Wed\, Apr 24\, 2019 at 08​:10​:19AM -0700\, James E Keenan via RT wrote​:

So there is no error '42' on Linux -- or at least not in /usr/include/asm-generic/errno-base.h.

I wonder if\, when cpan/Test-Simple/t/Legacy/More.t\, was added back in 2002\, '42' was considered to be a sufficiently "high" error number as to be suitable for use in testing as an impossible-to-achieve value.

Note also that error number 42 is not a new one for FreeBSD-13. On FreeBSD-11.2​:

The error number 42 is irrelevant. It's just an arbitrary but humorous number that $! is set to at the start of the script\, in order to see that it doesn't get changed during the course of the script (i.e. test that Test​::More etc localise $! when doing their stuff.)

The fact that its getting changed to ENOMEM is more interesting. This might indicate a lack of resources on the testing machine\, and would also explain its intermittency.

However\, experimenting with 'ulimit -d'\, as I decrease the available memory\, I see the script usually dying after test 51\, never 54. Which might instead indicate something off with BSD and the setting of errno.

I'm only seeing this test script fail on one smoker\, freebsd 13.0-CURRENT (amd64/1 cpu) and its only been failing since around v5.29.8-61-gb5e20476d6

Test-Simple was upgraded in v5.29.7-102-ga6afdf72cd

As an experiment\, I created a branch in which I reverted that upgrade to Test-Simple. I didn't really expect it to clear up the anomalous test failures on non-threaded builds on FreeBSD-13\, and it did not. See http​://perl.develop-help.com/?b=smoke-me%2Fjkeenan%2Frt133958-revert-a6afdf72cd.

So we can rule out Test-Simple as the cause of the problem.

Thank you very much.

-- James E Keenan (jkeenan@​cpan.org)

[p5pRT]

From @iabyn

On Sat\, Apr 27\, 2019 at 06​:02​:20AM -0700\, James E Keenan via RT wrote​:

As an experiment\, I created a branch in which I reverted that upgrade to Test-Simple. I didn't really expect it to clear up the anomalous test failures on non-threaded builds on FreeBSD-13\, and it did not. See http​://perl.develop-help.com/?b=smoke-me%2Fjkeenan%2Frt133958-revert-a6afdf72cd.

So we can rule out Test-Simple as the cause of the problem.

Are you able to increase the amount of virtual memory available on the problematic machine when running more More.t? To confirm whether or not its an issue with limited resources? For that matter\, how much VM *is* available to run the test?

-- The Enterprise's efficient long-range scanners detect a temporal vortex distortion in good time\, allowing it to be safely avoided via a minor course correction.   -- Things That Never Happen in "Star Trek" #21

[p5pRT]

From @jkeenan

On Mon\, 29 Apr 2019 08​:46​:09 GMT\, davem wrote​:

On Sat\, Apr 27\, 2019 at 06​:02​:20AM -0700\, James E Keenan via RT wrote​:

As an experiment\, I created a branch in which I reverted that upgrade to Test-Simple. I didn't really expect it to clear up the anomalous test failures on non-threaded builds on FreeBSD-13\, and it did not. See http​://perl.develop-help.com/?b=smoke-me%2Fjkeenan%2Frt133958- revert-a6afdf72cd.

So we can rule out Test-Simple as the cause of the problem.

Are you able to increase the amount of virtual memory available on the problematic machine when running more More.t?

There is a complex procedure for increasing the size of a VM -- complex enough that I think it can only be done when setting up a VM in the first place (and complex enough that I have somebody else do it).

I doubt there's a procedure for increasing the amount of memory available for just one test. And\, in any event\, this is a problem that is intermittent (but frequent) and appears when tests are being run in parallel.

To confirm whether or not its an issue with limited resources? For that matter\, how much VM *is* available to run the test?

On my VM​:

##### $ VBoxManage showvminfo freebsd-13-current-201811161731_default_1542414238270_9060 | grep 'Memory size' Memory size​: 512MB #####

IIRC on the FreeBSD-13 VM Carlos uses for smoke-testing\, it's 2048MB.

-- James E Keenan (jkeenan@​cpan.org)

[p5pRT]

From @khwilliamson

On 4/29/19 5​:57 AM\, James E Keenan via RT wrote​:

On Mon\, 29 Apr 2019 08​:46​:09 GMT\, davem wrote​:

On Sat\, Apr 27\, 2019 at 06​:02​:20AM -0700\, James E Keenan via RT wrote​:

As an experiment\, I created a branch in which I reverted that upgrade to Test-Simple. I didn't really expect it to clear up the anomalous test failures on non-threaded builds on FreeBSD-13\, and it did not. See http​://perl.develop-help.com/?b=smoke-me%2Fjkeenan%2Frt133958- revert-a6afdf72cd.

So we can rule out Test-Simple as the cause of the problem.

Are you able to increase the amount of virtual memory available on the problematic machine when running more More.t?

There is a complex procedure for increasing the size of a VM -- complex enough that I think it can only be done when setting up a VM in the first place (and complex enough that I have somebody else do it).

I doubt there's a procedure for increasing the amount of memory available for just one test. And\, in any event\, this is a problem that is intermittent (but frequent) and appears when tests are being run in parallel.

To confirm whether or not its an issue with limited resources? For that matter\, how much VM *is* available to run the test?

On my VM​:

##### $ VBoxManage showvminfo freebsd-13-current-201811161731_default_1542414238270_9060 | grep 'Memory size' Memory size​: 512MB #####

IIRC on the FreeBSD-13 VM Carlos uses for smoke-testing\, it's 2048MB.

You'd think 2G would be enough.

Would using $^M help in debugging this situation at all?

[p5pRT]

From @khwilliamson

On 4/29/19 5​:57 AM\, James E Keenan via RT wrote​:

On Mon\, 29 Apr 2019 08​:46​:09 GMT\, davem wrote​:

On Sat\, Apr 27\, 2019 at 06​:02​:20AM -0700\, James E Keenan via RT wrote​:

As an experiment\, I created a branch in which I reverted that upgrade to Test-Simple. I didn't really expect it to clear up the anomalous test failures on non-threaded builds on FreeBSD-13\, and it did not. See http​://perl.develop-help.com/?b=smoke-me%2Fjkeenan%2Frt133958- revert-a6afdf72cd.

So we can rule out Test-Simple as the cause of the problem.

Are you able to increase the amount of virtual memory available on the problematic machine when running more More.t?

There is a complex procedure for increasing the size of a VM -- complex enough that I think it can only be done when setting up a VM in the first place (and complex enough that I have somebody else do it).

I doubt there's a procedure for increasing the amount of memory available for just one test. And\, in any event\, this is a problem that is intermittent (but frequent) and appears when tests are being run in parallel.

To confirm whether or not its an issue with limited resources? For that matter\, how much VM *is* available to run the test?

On my VM​:

##### $ VBoxManage showvminfo freebsd-13-current-201811161731_default_1542414238270_9060 | grep 'Memory size' Memory size​: 512MB #####

IIRC on the FreeBSD-13 VM Carlos uses for smoke-testing\, it's 2048MB.

Another option to rule out that its locale handling\, is to add -Accflags='-DUSE_THREAD_SAFE_LOCALE' to your Configure options.

The thread safe locale handling is usually only selected on threaded builds\, and is very different from the non-thread safe. It seems unlikely that the newer handling would work when the old tried and true doesn't\, but it could be a bug in this unstable OS version that gets bypassed. As this seems pretty easy to try\, I think it might be worth a try.

[p5pRT]

From @khwilliamson

On 4/29/19 5​:57 AM\, James E Keenan via RT wrote​:

On Mon\, 29 Apr 2019 08​:46​:09 GMT\, davem wrote​:

On Sat\, Apr 27\, 2019 at 06​:02​:20AM -0700\, James E Keenan via RT wrote​:

As an experiment\, I created a branch in which I reverted that upgrade to Test-Simple. I didn't really expect it to clear up the anomalous test failures on non-threaded builds on FreeBSD-13\, and it did not. See http​://perl.develop-help.com/?b=smoke-me%2Fjkeenan%2Frt133958- revert-a6afdf72cd.

So we can rule out Test-Simple as the cause of the problem.

Are you able to increase the amount of virtual memory available on the problematic machine when running more More.t?

There is a complex procedure for increasing the size of a VM -- complex enough that I think it can only be done when setting up a VM in the first place (and complex enough that I have somebody else do it).

I doubt there's a procedure for increasing the amount of memory available for just one test. And\, in any event\, this is a problem that is intermittent (but frequent) and appears when tests are being run in parallel.

To confirm whether or not its an issue with limited resources? For that matter\, how much VM *is* available to run the test?

On my VM​:

##### $ VBoxManage showvminfo freebsd-13-current-201811161731_default_1542414238270_9060 | grep 'Memory size' Memory size​: 512MB #####

IIRC on the FreeBSD-13 VM Carlos uses for smoke-testing\, it's 2048MB.

Tony Cook set me up a VM of FreeBSD 13 with 4096MB. I ran the test suite several times\, and it did not fail for me\, but look at this excerpt from the log

io/through.t ....................................................... ok Cannot allocate memory re/fold_grind_a.t .................................................. ok Cannot allocate memory re/fold_grind_8.t .................................................. ok

plus several more like that. Yet all tests passed.

So something is getting errno 12\, and printing out the associated English text\, but it doesn't affect the pass/fail of these tests.

That makes it seem likely that its the smaller memory size available in these other smokers is so small that it can't recover as it does here. Perhaps someone can shed light on how the recovery is done.

But the question becomes\, why is this happening now\, and not earlier\, and why only threaded builds? I presume the freebsd smokers of lower version numbers of the OS have the same amount of vm as this. Is this correct?

I ran through the test suite with clang asan enabled\, and got no errors\, besides undefined behavior. One of the header files on the platform had code that left shifts a negative integer\, and it showed up ubiquitously until I silenced it. The path to the hdr includes 'machine' as a directory\, so it's quite likely that the behavior it gets is as expected.

I tried to do leak detection\, but it says​:

==64995==AddressSanitizer​: detect_leaks is not supported on this platform

[p5pRT]

From @tonycoz

On Mon\, 25 Mar 2019 17​:45​:17 -0700\, jkeenan@​pobox.com wrote​:

We are experiencing massive test failures in our smoke-testing of the Perl 5 core distribution on FreeBSD-13-CURRENT. I am filing two separate bug tickets for these failures\, since one failure appeared long before the others. This is the first of the two tickets.

Does the attached fix this for you?

I was only able to reproduce the messages from re/fold_grind_a.t\, but from the mechanism of the failure (unexpected changes to errno) I'm pretty sure this will solve it for More.t too.

Tony

[p5pRT]

From @tonycoz

0001-perl-133958-preserve-errno-on-successful-malloc-real.patch

```diff From 30dd168e6e502186d2de002acaf77323162c97ff Mon Sep 17 00:00:00 2001 From: Tony Cook Date: Fri, 3 May 2019 14:49:50 +1000 Subject: (perl #133958) preserve errno on successful malloc/realloc In general perl doesn't try to preserve errno (aka $!) since we're aiming at the same behaviour as for C code - errno is only meaningful if a function returned an error. The exception to that is when perl is working without an explicit request from the perl programmer. When code is performing assignments, concatenating strings, pushing on arrays etc, perl is exercising the memory allocation machinery, calling malloc() and realloc(). It turns out that at least on one platform, realloc() can modify errno on success. It appears to be happening when jemalloc (the malloc() implementation used on FreeBSD) tries to extend a memory arena and fails, leaving the error number from that failure in errno, from truss: mmap(0x80142f000,32768,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_ANON|MAP_EXCL,-1,0x0) ERR#12 'Cannot allocate memory' This magic call appears to be a FreeBSD specific mechanism to resize the anonymous mapping. On Linux the equivalent seems to be calling mremap(). In each case for the test code mmap() is successfully called immediately afterwards: mmap(0x0,69632,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 34390323200 (0x801d2b000) and realloc() succeeds. glibc() realloc seems to be simpler, AFAICT from reading the code it only uses mremap() when the memory block is the entire mapping, ie. for large blocks rather than for memory arenas, and it doesn't request the same address, so it doesn't fail. For blocks that are part of arenas, glibc tries to expand in-place within the current arena (with no extending the arena itself) or falls back to malloc, so there's no chance for errno to be changed on a successful realloc(). --- util.c | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/util.c b/util.c index 7276fd901d..27b221837c 100644 --- a/util.c +++ b/util.c @@ -132,6 +132,7 @@ Perl_safesysmalloc(MEM_SIZE size) dTHX; #endif Malloc_t ptr; + dSAVEDERRNO; #ifdef USE_MDH if (size + PERL_MEMORY_DEBUG_HEADER_SIZE < size) @@ -143,6 +144,7 @@ Perl_safesysmalloc(MEM_SIZE size) Perl_croak_nocontext("panic: malloc, size=%" UVuf, (UV) size); #endif if (!size) size = 1; /* malloc(0) is NASTY on our system */ + SAVE_ERRNO; #ifdef PERL_DEBUG_READONLY_COW if ((ptr = mmap(0, size, PROT_READ|PROT_WRITE, MAP_ANON|MAP_PRIVATE, -1, 0)) == MAP_FAILED) { @@ -154,6 +156,11 @@ Perl_safesysmalloc(MEM_SIZE size) #endif PERL_ALLOC_CHECK(ptr); if (ptr != NULL) { + /* malloc() can modify errno() even on success, but since someone + writing perl code doesn't have any control over when perl calls + malloc() we need to hide that. + */ + RESTORE_ERRNO; #ifdef USE_MDH struct perl_memory_debug_header *const header = (struct perl_memory_debug_header *)ptr; @@ -223,6 +230,7 @@ Perl_safesysrealloc(Malloc_t where,MEM_SIZE size) ptr = safesysmalloc(size); } else { + dSAVE_ERRNO; #ifdef USE_MDH where = (Malloc_t)((char*)where-PERL_MEMORY_DEBUG_HEADER_SIZE); if (size + PERL_MEMORY_DEBUG_HEADER_SIZE < size) @@ -276,6 +284,11 @@ Perl_safesysrealloc(Malloc_t where,MEM_SIZE size) might allocate memory/free/move memory, and until we do the fixup, it may well be chasing (and writing to) free memory. */ if (ptr != NULL) { + /* realloc() can modify errno() even on success, but since someone + writing perl code doesn't have any control over when perl calls + realloc() we need to hide that. + */ + RESTORE_ERRNO; #ifdef PERL_TRACK_MEMPOOL struct perl_memory_debug_header *const header = (struct perl_memory_debug_header *)ptr; -- 2.21.0 ```

[p5pRT]

From @tonycoz

On Thu\, 02 May 2019 22​:11​:19 -0700\, tonyc wrote​:

On Mon\, 25 Mar 2019 17​:45​:17 -0700\, jkeenan@​pobox.com wrote​:

We are experiencing massive test failures in our smoke-testing of the Perl 5 core distribution on FreeBSD-13-CURRENT. I am filing two separate bug tickets for these failures\, since one failure appeared long before the others. This is the first of the two tickets.

Does the attached fix this for you?

I was only able to reproduce the messages from re/fold_grind_a.t\, but from the mechanism of the failure (unexpected changes to errno) I'm pretty sure this will solve it for More.t too.

Karl did most of the work in tracking this down.

Tony

[p5pRT]

From @hvds

On Thu\, 02 May 2019 22​:11​:19 -0700\, tonyc wrote​:

+ RESTORE_ERRNO; #ifdef USE_MDH struct perl_memory_debug_header *const header = (struct perl_memory_debug_header *)ptr;

I think you'll need to protect the 'header' declaration from the non-declaration above it\, and similarly in the second RESTORE case.

Hugo

[p5pRT]

From @tonycoz

On Fri\, May 03\, 2019 at 02​:57​:20AM -0700\, Hugo van der Sanden via RT wrote​:

On Thu\, 02 May 2019 22​:11​:19 -0700\, tonyc wrote​:

+ RESTORE_ERRNO; #ifdef USE_MDH struct perl_memory_debug_header *const header = (struct perl_memory_debug_header *)ptr;

I think you'll need to protect the 'header' declaration from the non-declaration above it\, and similarly in the second RESTORE case.

Oops\, yep.

Tony

[p5pRT]

From @jkeenan

On Fri\, 03 May 2019 10​:02​:35 GMT\, tonyc wrote​:

On Fri\, May 03\, 2019 at 02​:57​:20AM -0700\, Hugo van der Sanden via RT wrote​:

On Thu\, 02 May 2019 22​:11​:19 -0700\, tonyc wrote​:

+ RESTORE_ERRNO; #ifdef USE_MDH struct perl_memory_debug_header *const header = (struct perl_memory_debug_header *)ptr;

I think you'll need to protect the 'header' declaration from the non- declaration above it\, and similarly in the second RESTORE case.

Oops\, yep.

Tony

Tony\, when you revise your patch to reflect Hugo's comments above\, can you push it to a smoke-me branch?

That will be easier for me to test in my FreeBSD-13 VM.

Thank you very much.

-- James E Keenan (jkeenan@​cpan.org)

[p5pRT]

From @khwilliamson

On 5/3/19 5​:40 AM\, James E Keenan via RT wrote​:

On Fri\, 03 May 2019 10​:02​:35 GMT\, tonyc wrote​:

On Fri\, May 03\, 2019 at 02​:57​:20AM -0700\, Hugo van der Sanden via RT wrote​:

On Thu\, 02 May 2019 22​:11​:19 -0700\, tonyc wrote​:

+ RESTORE_ERRNO; #ifdef USE_MDH struct perl_memory_debug_header *const header = (struct perl_memory_debug_header *)ptr;

I think you'll need to protect the 'header' declaration from the non- declaration above it\, and similarly in the second RESTORE case.

Oops\, yep.

Tony

Tony\, when you revise your patch to reflect Hugo's comments above\, can you push it to a smoke-me branch?

That will be easier for me to test in my FreeBSD-13 VM.

Thank you very much.

I made the changes suggested by Hugo\, and pushed the result to smoke-me/khw-freebsd

I can confirm that this getst rid of the spurious messages in #134076

[p5pRT]

From @dur-randir

On Thu\, 02 May 2019 22​:11​:19 -0700\, tonyc wrote​:

Does the attached fix this for you?

Shouldn't there be a new configure probe for a broken malloc\, so systems without such behaviour do not suffer performance penalty?

[p5pRT]

From @jkeenan

On Fri\, 03 May 2019 15​:08​:59 GMT\, public@​khwilliamson.com wrote​:

On 5/3/19 5​:40 AM\, James E Keenan via RT wrote​:

On Fri\, 03 May 2019 10​:02​:35 GMT\, tonyc wrote​:

On Fri\, May 03\, 2019 at 02​:57​:20AM -0700\, Hugo van der Sanden via RT wrote​:

On Thu\, 02 May 2019 22​:11​:19 -0700\, tonyc wrote​:

+ RESTORE_ERRNO; #ifdef USE_MDH struct perl_memory_debug_header *const header = (struct perl_memory_debug_header *)ptr;

I think you'll need to protect the 'header' declaration from the non- declaration above it\, and similarly in the second RESTORE case.

Oops\, yep.

Tony

Tony\, when you revise your patch to reflect Hugo's comments above\, can you push it to a smoke-me branch?

That will be easier for me to test in my FreeBSD-13 VM.

Thank you very much.

I made the changes suggested by Hugo\, and pushed the result to smoke-me/khw-freebsd

I can confirm that this getst rid of the spurious messages in #134076

Now I'm confused. I've already started smoking the smoke-me/tonyc/133958-realloc-errno-success branch. How does that differ from the smoke-me/khw-freebsd branch?

Thank you very much.

-- James E Keenan (jkeenan@​cpan.org)

[p5pRT]

From @khwilliamson

On 5/3/19 9​:24 AM\, James E Keenan via RT wrote​:

On Fri\, 03 May 2019 15​:08​:59 GMT\, public@​khwilliamson.com wrote​:

On 5/3/19 5​:40 AM\, James E Keenan via RT wrote​:

On Fri\, 03 May 2019 10​:02​:35 GMT\, tonyc wrote​:

On Fri\, May 03\, 2019 at 02​:57​:20AM -0700\, Hugo van der Sanden via RT wrote​:

On Thu\, 02 May 2019 22​:11​:19 -0700\, tonyc wrote​:

+ RESTORE_ERRNO; #ifdef USE_MDH struct perl_memory_debug_header *const header = (struct perl_memory_debug_header *)ptr;

I think you'll need to protect the 'header' declaration from the non- declaration above it\, and similarly in the second RESTORE case.

Oops\, yep.

Tony

Tony\, when you revise your patch to reflect Hugo's comments above\, can you push it to a smoke-me branch?

That will be easier for me to test in my FreeBSD-13 VM.

Thank you very much.

I made the changes suggested by Hugo\, and pushed the result to smoke-me/khw-freebsd

I can confirm that this getst rid of the spurious messages in #134076

Now I'm confused. I've already started smoking the smoke-me/tonyc/133958-realloc-errno-success branch. How does that differ from the smoke-me/khw-freebsd branch?

Thank you very much.

I didn't know Tony had pushed a new branch. There wasn't a reply on the list\, so I thought your request to him had come too late in his day. But my guess is that his branch and mine identical in functionality\, so continue smoking his.

[p5pRT]

From @tonycoz

On Fri\, 03 May 2019 04​:40​:11 -0700\, jkeenan wrote​:

On Fri\, 03 May 2019 10​:02​:35 GMT\, tonyc wrote​:

On Fri\, May 03\, 2019 at 02​:57​:20AM -0700\, Hugo van der Sanden via RT wrote​:

On Thu\, 02 May 2019 22​:11​:19 -0700\, tonyc wrote​:

+ RESTORE_ERRNO; #ifdef USE_MDH struct perl_memory_debug_header *const header = (struct perl_memory_debug_header *)ptr;

I think you'll need to protect the 'header' declaration from the non- declaration above it\, and similarly in the second RESTORE case.

Oops\, yep.

Tony

Tony\, when you revise your patch to reflect Hugo's comments above\, can you push it to a smoke-me branch?

That will be easier for me to test in my FreeBSD-13 VM.

Thank you very much.

As you saw\, I pushed it last night and this morning here's the patch for the record.

Tony

[p5pRT]

From @tonycoz

0001-perl-133958-preserve-errno-on-successful-malloc-real.patch

```diff From 373d744e311ee3721ab7a4ee2bd8a33b88a7f387 Mon Sep 17 00:00:00 2001 From: Tony Cook Date: Fri, 3 May 2019 14:49:50 +1000 Subject: (perl #133958) preserve errno on successful malloc/realloc In general perl doesn't try to preserve errno (aka $!) since we're aiming at the same behaviour as for C code - errno is only meaningful if a function returned an error. The exception to that is when perl is working without an explicit request from the perl programmer. When code is performing assignments, concatenating strings, pushing on arrays etc, perl is exercising the memory allocation machinery, calling malloc() and realloc(). It turns out that at least on one platform, realloc() can modify errno on success. It appears to be happening when jemalloc (the malloc() implementation used on FreeBSD) tries to extend a memory arena and fails, leaving the error number from that failure in errno, from truss: mmap(0x80142f000,32768,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_ANON|MAP_EXCL,-1,0x0) ERR#12 'Cannot allocate memory' This magic call appears to be a FreeBSD specific mechanism to resize the anonymous mapping. On Linux the equivalent seems to be calling mremap(). In each case for the test code mmap() is successfully called immediately afterwards: mmap(0x0,69632,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 34390323200 (0x801d2b000) and realloc() succeeds. glibc() realloc seems to be simpler, AFAICT from reading the code it only uses mremap() when the memory block is the entire mapping, ie. for large blocks rather than for memory arenas, and it doesn't request the same address, so it doesn't fail. For blocks that are part of arenas, glibc tries to expand in-place within the current arena (with no extending the arena itself) or falls back to malloc, so there's no chance for errno to be changed on a successful realloc(). --- util.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/util.c b/util.c index 7276fd901d..e5320c8b8c 100644 --- a/util.c +++ b/util.c @@ -132,6 +132,7 @@ Perl_safesysmalloc(MEM_SIZE size) dTHX; #endif Malloc_t ptr; + dSAVEDERRNO; #ifdef USE_MDH if (size + PERL_MEMORY_DEBUG_HEADER_SIZE < size) @@ -143,6 +144,7 @@ Perl_safesysmalloc(MEM_SIZE size) Perl_croak_nocontext("panic: malloc, size=%" UVuf, (UV) size); #endif if (!size) size = 1; /* malloc(0) is NASTY on our system */ + SAVE_ERRNO; #ifdef PERL_DEBUG_READONLY_COW if ((ptr = mmap(0, size, PROT_READ|PROT_WRITE, MAP_ANON|MAP_PRIVATE, -1, 0)) == MAP_FAILED) { @@ -182,6 +184,11 @@ Perl_safesysmalloc(MEM_SIZE size) ptr = (Malloc_t)((char*)ptr+PERL_MEMORY_DEBUG_HEADER_SIZE); DEBUG_m(PerlIO_printf(Perl_debug_log, "0x%" UVxf ": (%05ld) malloc %ld bytes\n",PTR2UV(ptr),(long)PL_an++,(long)size)); + /* malloc() can modify errno() even on success, but since someone + writing perl code doesn't have any control over when perl calls + malloc() we need to hide that. + */ + RESTORE_ERRNO; } else { #ifdef USE_MDH @@ -223,6 +230,7 @@ Perl_safesysrealloc(Malloc_t where,MEM_SIZE size) ptr = safesysmalloc(size); } else { + dSAVE_ERRNO; #ifdef USE_MDH where = (Malloc_t)((char*)where-PERL_MEMORY_DEBUG_HEADER_SIZE); if (size + PERL_MEMORY_DEBUG_HEADER_SIZE < size) @@ -296,6 +304,12 @@ Perl_safesysrealloc(Malloc_t where,MEM_SIZE size) maybe_protect_ro(header->prev); #endif ptr = (Malloc_t)((char*)ptr+PERL_MEMORY_DEBUG_HEADER_SIZE); + + /* realloc() can modify errno() even on success, but since someone + writing perl code doesn't have any control over when perl calls + realloc() we need to hide that. + */ + RESTORE_ERRNO; } /* In particular, must do that fixup above before logging anything via -- 2.21.0 ```

[p5pRT]

From @tonycoz

On Fri\, 03 May 2019 08​:20​:06 -0700\, randir wrote​:

On Thu\, 02 May 2019 22​:11​:19 -0700\, tonyc wrote​:

Does the attached fix this for you?

Shouldn't there be a new configure probe for a broken malloc\, so systems without such behaviour do not suffer performance penalty?

Such behaviour is permitted by the C standard.

From the specification for errno (C11)​:

The value of errno may be set to nonzero by a library function call whether or not there is an error\, provided the use of errno is not documented in the description of the function in this International Standard.

and the standard doesn't document any behaviour for errno for realloc()\, malloc()\, so the behaviour is permitted.

Similarly POSIX says for errno​:

The setting of errno after a successful call to a function is unspecified unless the description of that function specifies that errno shall not be modified.

and there's nothing in the documentation of realloc() guaranteeing errno is unchanged on success.

Tony

[p5pRT]

From @tonycoz

On Fri\, 03 May 2019 17​:57​:22 -0700\, tonyc wrote​:

As you saw\, I pushed it last night and this morning here's the patch for the record.

Applied as 9f300641fc6fb15749f287d7ac7fd6f312b3edd8.

Tony

[p5pRT]

@tonycoz - Status changed from 'open' to 'pending release'

[p5pRT]

From @khwilliamson

Thank you for filing this report. You have helped make Perl better.

With the release today of Perl 5.30.0\, this and 160 other issues have been resolved.

Perl 5.30.0 may be downloaded via​: https://metacpan.org/release/XSAWYERX/perl-5.30.0

If you find that the problem persists\, feel free to reopen this ticket.

[p5pRT]

@khwilliamson - Status changed from 'pending release' to 'resolved'