Perl / perl5

🐪 The Perl programming language
https://dev.perl.org/perl5/
Other
1.91k stars 542 forks source link

XS::APItest::test_coplabel fails on HP-UX with aCC +O2 optimization enabled #11748

Closed p5pRT closed 12 years ago

p5pRT commented 12 years ago

Migrated from rt.perl.org#103668 (status was 'resolved')

Searchable as RT103668$

p5pRT commented 12 years ago

From @icerider70

Created by philip.monsen@pobox.com

Under +O1 compiler optimization\, the test passes fine​:

$ ./perl -I./lib ext/XS-APItest/t/coplabel.t 1..1 ok 1

But when +O2 is used\, not so much​:

$ ./perl -I./lib ext/XS-APItest/t/coplabel.t 1..1 fail # cop_fetch_label label at ext/XS-APItest/t/coplabel.t line 7.

I've filed this as a bug because the default -Doptimize value when Configure is run is '+O2 +Onolimit'.

I believe the issue is partly due to reordering of instructions. Here's the C code generated from APItest.xs​:

XS_EUPXS(XS_XS__APItest_test_coplabel) {   dVAR; dXSARGS;   if (items != 0)   croak_xs_usage(cv\, "");   { #line 2643 "APItest.xs"   COP *cop;   const char *label;   STRLEN len;   U32 utf8; #line 4476 "APItest.c" #line 2648 "APItest.xs"   cop = &PL_compiling;   Perl_cop_store_label(aTHX_ cop\, "foo"\, 3\, 0);   label = Perl_cop_fetch_label(aTHX_ cop\, &len\, &utf8);   if (strcmp(label\,"foo")) croak("fail # cop_fetch_label label");   if (len != 3) croak("fail # cop_fetch_label len");   if (utf8) croak("fail # cop_fetch_label utf8");   /* SMALL GERMAN UMLAUT A */   Perl_cop_store_label(aTHX_ cop\, "foC$"\, 4\, SVf_UTF8);   label = Perl_cop_fetch_label(aTHX_ cop\, &len\, &utf8);   if (strcmp(label\,"foC$")) croak("fail # cop_fetch_label label");   if (len != 4) croak("fail # cop_fetch_label len");   if (!utf8) croak("fail # cop_fetch_label utf8"); #line 4490 "APItest.c"   }   XSRETURN_EMPTY; }

Here's a snippet from a gdb debugging session under +O1​:

2649 Perl_cop_store_label(aTHX_ cop\, "foo"\, 3\, 0); (gdb) n 2650 label = Perl_cop_fetch_label(aTHX_ cop\, &len\, &utf8); (gdb) n 2651 if (strcmp(label\,"foo")) croak("fail # cop_fetch_label label"); (gdb) n 2652 if (len != 3) croak("fail # cop_fetch_label len"); (gdb) n 2653 if (utf8) croak("fail # cop_fetch_label utf8"); (gdb) n 2655 Perl_cop_store_label(aTHX_ cop\, "foä"\, 4\, SVf_UTF8); (gdb) n 2656 label = Perl_cop_fetch_label(aTHX_ cop\, &len\, &utf8); (gdb) n 2657 if (strcmp(label\,"foä")) croak("fail # cop_fetch_label label"); (gdb) p label $1 = 0x600000000051ac3d "fo\303\244" (gdb) p len $2 = 4 (gdb) p utf8 $3 = 536870912 (gdb) n 2658 if (len != 4) croak("fail # cop_fetch_label len"); (gdb) n 2659 if (!utf8) croak("fail # cop_fetch_label utf8"); (gdb) n 4491 XSRETURN_EMPTY;

Note the contrast when +O2 is in force​:

2650 label = Perl_cop_fetch_label(aTHX_ cop\, &len\, &utf8); (gdb) n 2651 if (strcmp(label\,"foo")) croak("fail # cop_fetch_label label"); (gdb) p label $1 = \ (gdb) p len $2 = 3 (gdb) p utf8 $3 = 0 (gdb) n 0x9fffffffbeb76080​:0 2659 if (!utf8) croak("fail # cop_fetch_label utf8"); (gdb) n 2652 if (len != 3) croak("fail # cop_fetch_label len"); (gdb) n 2653 if (utf8) croak("fail # cop_fetch_label utf8"); (gdb) n 2655 Perl_cop_store_label(aTHX_ cop\, "foä"\, 4\, SVf_UTF8); (gdb) n 2656 label = Perl_cop_fetch_label(aTHX_ cop\, &len\, &utf8); (gdb) n 2657 if (strcmp(label\,"foä")) croak("fail # cop_fetch_label label"); (gdb) p label $4 = \ (gdb) p len $5 = 4 (gdb) p utf8 $6 = 536870912 (gdb) n 0x9fffffffbeb76050​:0 2659 if (!utf8) croak("fail # cop_fetch_label utf8"); (gdb) n 0x9fffffffbf00adf0​:0 in siglongjmp+0 () from /usr/lib/hpux64/libc.so.1

Due to the opaqueness imposed by the +O2 optimizations\, it's hard to tell exactly what happened\, but somehow the optimizations are inducing a false exit path.

Reordering the tests within this function\, and changing strcmp() to memcmp()\, appears to resolve the issue (i.e. we can compile with the default of -Doptimize='+O2 +Onolimit' and this test passes).

Patch forthcoming.

Perl Info ``` Flags: category=core severity=low Site configuration information for perl 5.15.4: Configured by monsp at Tue Nov 15 06:26:35 CST 2011. Summary of my perl5 (revision 5 version 15 subversion 4) configuration: Derived from: 691f175822a5edda9dc7ba243d6c774254b9e225 Platform: osname=hpux, osvers=11.23, archname=IA64.ARCHREV_0-LP64 uname='hp-ux rp5 b.11.23 u ia64 3731604725 unlimited-user license ' config_args='-des -Uuseithreads -Uglibpth -Ulocincpth -Uloclibpth -Dusedevel -Duse64bitall -Dcc=aCC -Dccflags=+DD64 -Ae +DSnative -D_XOPEN_SOURCE -D_XOPEN_SOURCE_EXTENDED -D_HPUX_ALT_XOPEN_SOCKET_API -D_HPUX_SOURCE -Wl,+vnocompatwarnings -Dld=/usr/bin/ld -Dldflags=+DD64 -L/usr/lib/hpux64 -Dlibpth=/lib/hpux64 /usr/lib/hpux64 /usr/ccs/lib -Dlibs=-lnsl -lnm -ldl -ldld -lm -lsec -lc -DDEBUGGING -Doptimize=+O2 +Onolimit -Dprefix=/home/monsp/open-source/perl-install/blead' hint=recommended, useposix=true, d_sigaction=define useithreads=undef, usemultiplicity=undef useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef use64bitint=define, use64bitall=define, uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='aCC', ccflags =' -Ae +DD64 +DSnative -D_XOPEN_SOURCE -D_XOPEN_SOURCE_EXTENDED -D_HPUX_ALT_XOPEN_SOCKET_API -Wl,+vnocompatwarnings -D_HPUX_SOURCE -Wl,+vnocompatwarnings +DD64 -DDEBUGGING -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 ', optimize='+O2 +Onolimit -g', cppflags='-Aa -D__STDC_EXT__ +DD64 +DSnative -D_XOPEN_SOURCE -D_XOPEN_SOURCE_EXTENDED -D_HPUX_ALT_XOPEN_SOCKET_API -Wl,+vnocompatwarnings -D_HPUX_SOURCE -Ae +DD64 +DSnative -D_XOPEN_SOURCE -D_XOPEN_SOURCE_EXTENDED -D_HPUX_ALT_XOPEN_SOCKET_API -Wl,+vnocompatwarnings -D_HPUX_SOURCE -Wl,+vnocompatwarnings +DD64 -DDEBUGGING' ccversion='B3910B', gccversion='', gccosandvers='' intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=87654321 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16 ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8 alignbytes=8, prototype=define Linker and Libraries: ld='/usr/bin/ld', ldflags ='+DD64 -L/usr/lib/hpux64 +DD64' libpth=/lib/hpux64 /usr/lib/hpux64 /usr/ccs/lib libs=-lnsl -lnm -ldl -ldld -lm -lsec -lc perllibs=-lnsl -lnm -ldl -ldld -lm -lsec -lc libc=/usr/lib/hpux64/libc.so, so=so, useshrplib=false, libperl=libperl.a gnulibc_version='' Dynamic Linking: dlsrc=dl_hpux.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E -Wl,-B,deferred ' cccdlflags='+Z', lddlflags='-b +vnocompatwarnings -L/usr/lib/hpux64' Locally applied patches: @INC for perl 5.15.4: lib /home/monsp/open-source/perl-install/blead/lib/site_perl/5.15.4/IA64.ARCHREV_0-LP64 /home/monsp/open-source/perl-install/blead/lib/site_perl/5.15.4 /home/monsp/open-source/perl-install/blead/lib/5.15.4/IA64.ARCHREV_0-LP64 /home/monsp/open-source/perl-install/blead/lib/5.15.4 . Environment for perl 5.15.4: HOME=/users/apps/ssd/monsp LANG (unset) LANGUAGE (unset) LD_LIBRARY_PATH (unset) LOGDIR (unset) PATH=/apps/util/tools/xxxx/opt/subversion/bin:/apps/util/tools/xxxx/opt/git/bin:/apps/util/tools/xxxx/usr/bin:/sysapps/bin:/sysapps/bin2:/opt/ssh/bin:/usr/bin:/usr/ccs/bin:/usr/contrib/bin:/usr/bin/X11:/opt/perf/bin:/usr/contrib/bin/X11:/opt/aCC/bin:/opt/langtools/bin PERL_BADLANG (unset) SHELL=/usr/bin/ksh ```
p5pRT commented 12 years ago

From zefram@fysh.org

Philip Monsen wrote​:

Reordering the tests within this function\, and changing strcmp() to memcmp()\, appears to resolve the issue (i.e. we can compile with the default of -Doptimize='+O2 +Onolimit' and this test passes).

Presumably the same optimisation bug will strike in other places that we don't test. Can we resolve it instead by modifying the optimisation flags?

-zefram

p5pRT commented 12 years ago

The RT System itself - Status changed from 'new' to 'open'

p5pRT commented 12 years ago

From @icerider70

On Tue\, Nov 15\, 2011 at 10​:43 AM\, Zefram \zefram@​fysh\.org wrote​:

Presumably the same optimisation bug will strike in other places that we don't test. Can we resolve it instead by modifying the optimisation flags?

I thought about that\, but it seemed like overkill considering that all other tests in the test suite pass\, and given that this default build option hasn't changed for a long time. +O2 alone is sufficient to trigger the issue\, and downgrading optimization from that level for a production build doesn't seem appropriate just to fix one test.

Someone more familiar with the core code being tested could probably revise the test_coplabel() code (and/or its pure-Perl test invocation code) to insulate more generally against this bug. My patch is just one way to do it.

--Phil

p5pRT commented 12 years ago

From @icerider70

The attached patch corrects the issue. To illustrate the difference between this and what's currently in blead\, here's a snippet from a gdb session with the patch applied and +O2 in force​:

2650 label = Perl_cop_fetch_label(aTHX_ cop\, &len\, &utf8); (gdb) n 2651 if (len != 3) croak("fail # test1 cop_fetch_label len"); (gdb) n 2652 if (memcmp(label\,"foo"\,len)) (gdb) n 2654 if (utf8) croak("fail # test1 cop_fetch_label utf8"); (gdb) n 2656 Perl_cop_store_label(aTHX_ cop\, "foä"\, 4\, SVf_UTF8); (gdb) n 2657 label = Perl_cop_fetch_label(aTHX_ cop\, &len\, &utf8); (gdb) n 2658 if (len != 4) croak("fail # test2 cop_fetch_label len"); (gdb) n 2659 if (memcmp(label\,"foä"\,len)) (gdb) n 2661 if (!utf8) croak("fail # test2 cop_fetch_label utf8"); (gdb) n Perl_pp_entersub () at pp_hot.c​:2762

Note how the optimization no longer induces a bogus failure and early exit.

p5pRT commented 12 years ago

From @icerider70

0001-Make-test_coplabel-more-optimization-safe.patch ```diff From 009dcc517b2bf746c1a65640a1c89d80ca4589f2 Mon Sep 17 00:00:00 2001 From: Philip Monsen Date: Tue, 15 Nov 2011 09:51:52 -0600 Subject: [PATCH] Make test_coplabel() more optimization safe MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="------------1.7.7.3" This is a multi-part message in MIME format. --------------1.7.7.3 Content-Type: text/plain; charset=UTF-8; format=fixed Content-Transfer-Encoding: 8bit --- ext/XS-APItest/APItest.xs | 15 ++++++++------- 1 files changed, 8 insertions(+), 7 deletions(-) --------------1.7.7.3 Content-Type: text/x-patch; name="0001-Make-test_coplabel-more-optimization-safe.patch" Content-Transfer-Encoding: 8bit Content-Disposition: attachment; filename="0001-Make-test_coplabel-more-optimization-safe.patch" diff --git a/ext/XS-APItest/APItest.xs b/ext/XS-APItest/APItest.xs index 646d821..68ce3f5 100644 --- a/ext/XS-APItest/APItest.xs +++ b/ext/XS-APItest/APItest.xs @@ -2648,16 +2648,17 @@ test_coplabel() cop = &PL_compiling; Perl_cop_store_label(aTHX_ cop, "foo", 3, 0); label = Perl_cop_fetch_label(aTHX_ cop, &len, &utf8); - if (strcmp(label,"foo")) croak("fail # cop_fetch_label label"); - if (len != 3) croak("fail # cop_fetch_label len"); - if (utf8) croak("fail # cop_fetch_label utf8"); + if (len != 3) croak("fail # test1 cop_fetch_label len"); + if (memcmp(label,"foo",len)) + croak("fail # test1 cop_fetch_label label"); + if (utf8) croak("fail # test1 cop_fetch_label utf8"); /* SMALL GERMAN UMLAUT A */ Perl_cop_store_label(aTHX_ cop, "fo��", 4, SVf_UTF8); label = Perl_cop_fetch_label(aTHX_ cop, &len, &utf8); - if (strcmp(label,"fo��")) croak("fail # cop_fetch_label label"); - if (len != 4) croak("fail # cop_fetch_label len"); - if (!utf8) croak("fail # cop_fetch_label utf8"); - + if (len != 4) croak("fail # test2 cop_fetch_label len"); + if (memcmp(label,"fo��",len)) + croak("fail # test2 cop_fetch_label label"); + if (!utf8) croak("fail # test2 cop_fetch_label utf8"); HV * example_cophh_2hv() --------------1.7.7.3-- ```
p5pRT commented 12 years ago

From zefram@fysh.org

Philip Monsen wrote​:

The attached patch corrects the issue.

The patch has a small fault\, that the new code with memcmp() isn't testing for the nul terminator on the strings. That's easily fixed by incrementing the block length parameter.

However\, looking at the change\, I am even less happy about patching it here than I was previously. The kind of operations that are evidently triggering the compiler bug are very ordinary\, and we do that sort of thing all over the place. I am not satisfied that the rest of the test suite can be relied upon to catch it\, if a bug of this kind is affecting core behaviour. This test is the canary that warns us that this compiler is broken with the options we give it; nailing the canary to its perch is not the appropriate response.

We need to turn off the broken optimisation. If that means dropping all the way from +O2 to +O1 then so be it. I prefer to get the right answer slowly.

-zefram

p5pRT commented 12 years ago

From @icerider70

On Wed\, Nov 16\, 2011 at 7​:41 AM\, Zefram \zefram@​fysh\.org wrote​:

We need to turn off the broken optimisation. If that means dropping all the way from +O2 to +O1 then so be it. I prefer to get the right answer slowly.

Fair enough. I'll try using +O2 as a base and pulling specific optimizations down to what they would be at +O1 and see if I can find the culprit that way.

--Phil

p5pRT commented 12 years ago

From @icerider70

On Wed\, Nov 16\, 2011 at 8​:19 AM\, Philip Monsen \philip\.monsen@​pobox\.comwrote​:

On Wed\, Nov 16\, 2011 at 7​:41 AM\, Zefram \zefram@​fysh\.org wrote​:

We need to turn off the broken optimisation. If that means dropping all the way from +O2 to +O1 then so be it. I prefer to get the right answer slowly.

Fair enough. I'll try using +O2 as a base and pulling specific optimizations down to what they would be at +O1 and see if I can find the culprit that way.

I added +Oinfo to see what optimizations are actually occurring. Looks like inlining may be the culprit (these are the only messages for the test_coplabel() function)​:

"APItest.c"\, line 2651\, procedure XS_XS__APItest_test_coplabel​: info #20055-D​: Inlined call to strcmp

"APItest.c"\, line 2657\, procedure XS_XS__APItest_test_coplabel​: info #20055-D​: Inlined call to strcmp

Default for +O2 is '+inline_level 2' and for +O1 is '+inline_level 1'. Inline levels can be specified in increments of 0.1 between 1 and 9. Will see if tweaking this down corrects the issue\, and if so\, the maximum setting for which all tests still pass.

--Phil

p5pRT commented 12 years ago

From @icerider70

On Wed\, Nov 16\, 2011 at 9​:30 AM\, Philip Monsen \philip\.monsen@​pobox\.comwrote​:

I added +Oinfo to see what optimizations are actually occurring. Looks like inlining may be the culprit (these are the only messages for the test_coplabel() function)​:

"APItest.c"\, line 2651\, procedure XS_XS__APItest_test_coplabel​: info #20055-D​: Inlined call to strcmp

"APItest.c"\, line 2657\, procedure XS_XS__APItest_test_coplabel​: info #20055-D​: Inlined call to strcmp

Default for +O2 is '+inline_level 2' and for +O1 is '+inline_level 1'. Inline levels can be specified in increments of 0.1 between 1 and 9. Will see if tweaking this down corrects the issue\, and if so\, the maximum setting for which all tests still pass.

This wasn't quite the right tweak\, but the +Oinfo report was helpful. Looking across the whole build\, the only str* function being inlined was strcmp(). Various attempts to squash this inlining did not succeed\, to include​:

+O2 +Onoinline +O2 +Onoinline=strcmp

However\, using '+O2 +Onolibcalls=strcmp' results in this test passing. Here's the gdb session (nice and clean)​:

2650 label = Perl_cop_fetch_label(aTHX_ cop\, &len\, &utf8); (gdb) n 2651 if (strcmp(label\,"foo")) croak("fail # cop_fetch_label label"); (gdb) n 2652 if (len != 3) croak("fail # cop_fetch_label len"); (gdb) n 2653 if (utf8) croak("fail # cop_fetch_label utf8"); (gdb) n 2655 Perl_cop_store_label(aTHX_ cop\, "foä"\, 4\, SVf_UTF8); (gdb) n 2656 label = Perl_cop_fetch_label(aTHX_ cop\, &len\, &utf8); (gdb) n 2657 if (strcmp(label\,"foä")) croak("fail # cop_fetch_label label"); (gdb) n 2658 if (len != 4) croak("fail # cop_fetch_label len"); (gdb) n 2659 if (!utf8) croak("fail # cop_fetch_label utf8"); (gdb) n Perl_pp_entersub () at pp_hot.c​:2762

I'm going to rebuild once more with '+O2 +Onolimit +Olibcalls=strcmp'\, and assuming I get a clean test run\, I'll post a fresh patch\, this time to hints/hpux.sh

--Phil

p5pRT commented 12 years ago

From @icerider70

On Wed\, Nov 16\, 2011 at 11​:08 AM\, Philip Monsen \philip\.monsen@​pobox\.comwrote​:

I'm going to rebuild once more with '+O2 +Onolimit +Onolibcalls=strcmp'\, and assuming I get a clean test run\, I'll post a fresh patch\, this time to hints/hpux.sh

Promised patch (to hints/hpux.sh) against blead attached. This one also corrects a couple of incidental things I ran across in that file while preparing the patch\, but most importantly\, fixes the default optimization flags for my narrow combination of platform + compiler version. I think the version coverage (i.e. versions beyond B3910B A.06.15) could/should be expanded if/when other folks could confirm that the bug is also triggered with such compiler versions.

I added some additional relaxations to the optimize string\, as the +Onolibcalls=strcmp by itself corrected the failing test but appeared to induce another failure (in t/op/state.t). The resulting collection of flags​:

+O2 +Onolimit +Onoprocelim +Ostore_ordering +Onolibcalls=strcmp

also seems reasonable to me given the initial observed behavior\, and avoids having to punt back to +O1.

With this patch applied\, all tests in the test suite pass when the (updated) default optimization string is used. The earlier patch I submitted\, to the xs/XS-APItest/APItest.xs file\, should be disregarded.

--Phil

p5pRT commented 12 years ago

From @icerider70

0001-Relax-level-2-optimizations-to-fix-test-failures-on-.patch ```diff From f4ccbf7554962be4d37838734f7d6c842e12691b Mon Sep 17 00:00:00 2001 From: Philip Monsen Date: Thu, 17 Nov 2011 11:22:17 -0600 Subject: [PATCH] Relax level 2 optimizations to fix test failures on HP-UX MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="------------1.7.7.3" This is a multi-part message in MIME format. --------------1.7.7.3 Content-Type: text/plain; charset=UTF-8; format=fixed Content-Transfer-Encoding: 8bit --- hints/hpux.sh | 17 +++++++++++++++-- 1 files changed, 15 insertions(+), 2 deletions(-) --------------1.7.7.3 Content-Type: text/x-patch; name="0001-Relax-level-2-optimizations-to-fix-test-failures-on-.patch" Content-Transfer-Encoding: 8bit Content-Disposition: attachment; filename="0001-Relax-level-2-optimizations-to-fix-test-failures-on-.patch" diff --git a/hints/hpux.sh b/hints/hpux.sh index be6c1fd..64de1bd 100644 --- a/hints/hpux.sh +++ b/hints/hpux.sh @@ -157,7 +157,7 @@ case `$cc -v 2>&1`"" in done [ -z "$cc_found" ] && cc_found=`which cc` what $cc_found >&4 - ccversion=`what $cc_found | awk '/Compiler/{print $2}/Itanium/{print $6,$7}/for Integrity/{print $6}'` + ccversion=`what $cc_found | awk '/Compiler/{print $2}/Itanium/{print $6,$7}/for Integrity/{print $6,$7}'` case "$ccflags" in "-Ae "*) ;; *) ccflags="-Ae $cc_cppflags" @@ -414,7 +414,7 @@ case "$ccisgcc" in fi ;; - *) # HP's compiler cannot combine -g and -O + *) case "$optimize" in "") optimize="+O2 +Onolimit" ;; *O[3456789]*) optimize=`echo "$optimize" | sed -e 's/O[3-9]/O2/'` ;; @@ -436,6 +436,19 @@ case "$ccisgcc" in # maint (5.8.8+) and blead (5.9.3+) # -O1/+O1 passed all tests (m)'05 [ 10 Jan 2005 ] optimize="$opt" ;; + B3910B*A.06.15) + # > cc --version + # cc: HP C/aC++ B3910B A.06.15 [May 16 2007] + # Has optimizing problems with +O2 for blead (5.15.4), + # see https://rt.perl.org:443/rt3/Ticket/Display.html?id=103668. + # + # +O2 +Onolimit +Onoprocelim +Ostore_ordering \ + # +Onolibcalls=strcmp + # passes all tests (with/without -DDEBUGGING) [Nov 17 2011] + case "$optimize" in + *O2*) optimize="$optimize +Onoprocelim +Ostore_ordering +Onolibcalls=strcmp" ;; + esac + ;; *) doop_cflags="optimize=\"$opt\"" op_cflags="optimize=\"$opt\"" ;; esac --------------1.7.7.3-- ```
p5pRT commented 12 years ago

From zefram@fysh.org

Philip Monsen wrote​:

Promised patch (to hints/hpux.sh) against blead attached.

Thanks\, applied as 2ba3ed0.

-zefram

p5pRT commented 12 years ago

@cpansprout - Status changed from 'open' to 'resolved'