-DXvt crashes in strict.pm

Description Under -DDEBUGGING, the debug flags Xvt (-DXvt or $^D = "Xvt" at runtime) crashes when strict.pm is loaded via require/use.

Steps to Reproduce Perl must be built with -DDEBUGGING perl -DXvt -Mstrict -e 'print "Hello\n"'

Expected behavior The requested debug output (Xvt traces execution and dumps PADs and reports when CVs are cloned) alongside executed script behavior.

Actual results The last few lines of output are as such:

(15244:/usr/share/perl/5.32/strict.pm:28)       nextstate
(15244:/usr/share/perl/5.32/strict.pm:30)       padsv($bits)
(15244:/usr/share/perl/5.32/strict.pm:30)       padsv($inline_all_bits)
(15244:/usr/share/perl/5.32/strict.pm:30)       sassign
(15244:/usr/share/perl/5.32/strict.pm:30)       nextstate
(15244:/usr/share/perl/5.32/strict.pm:31)       anoncode
Pad 0x558289c25a40[0x558289c3fb18] sv:      17 sv=0x558289c25e48
Pad 0x558289c25fe0[0x558289c37438] new:       compcv=0x558289c25fc8 name=0x558289c391c8 flags=0x3
CV undef: cv=0x558289c25fc8 comppad=0x558289c25a40
Pad set_null
Pad undef: cv=0x558289c25fc8 padlist=0x558289c3cc38 comppad=0x558289c25a40

Pad CV clone
  Proto: CV=0x558289c25e48 (ANON), OUTSIDE=0x558289c25a28 (UNIQUE)
    PADLIST = 0x558289c37288
    PADNAME = 0x558289c391c8(0x558289c3c358) PAD = 0x558289c25e60(0x558289c38f08)
         1. 0x0<0> FAKE "$inline_all_bits" flags=0x0 index=16
debugperl: pad.c:1855: S_cv_dump: Assertion `!(((XPVCV*)({ void *_p = (((CV*)(cv))->sv_any); _p; }))->xcv_flags & 0x0008)' failed.
Aborted (core dumped)

The line 31 from strict.pm in question is this:

*all_bits = sub () { $inline_all_bits };

This is inside a BEGIN {} block so it is occurring as soon as strict.pm is require()d (including via 'use').

Core was generated by `debugperl -DXvt -Mstrict -e print "Hello\n"'.
Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:49
49      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:49
#1  0x00007fabaeb3a864 in __GI_abort () at abort.c:79
#2  0x00007fabaeb3a749 in __assert_fail_base (fmt=0x7fabaecc3f78 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x5582898a0610 "!(((XPVCV*)({ void *_p = (((CV*)(cv))->sv_any); _p; }))->xcv_flags & 0x0008)",
    file=0x5582898ce650 "pad.c", line=1855, function=<optimized out>) at assert.c:92
#3  0x00007fabaeb4c3d6 in __GI___assert_fail (assertion=assertion@entry=0x5582898a0610 "!(((XPVCV*)({ void *_p = (((CV*)(cv))->sv_any); _p; }))->xcv_flags & 0x0008)", file=file@entry=0x5582898ce650 "pad.c", line=line@entry=1855,
    function=function@entry=0x5582898cfa60 <__PRETTY_FUNCTION__.14> "S_cv_dump") at assert.c:101
#4  0x00005582896a9531 in S_cv_dump (my_perl=<optimized out>, cv=<optimized out>, title=0x5582898d4860 "To") at pad.c:1855
#5  0x00005582896aaf70 in S_cv_clone (my_perl=<optimized out>, proto=<optimized out>, cv=0x558289c25fc8, outside=<optimized out>, cloned=<optimized out>) at pad.c:2226
#6  0x00005582897884eb in Perl_pp_anoncode (my_perl=0x558289bf02a0) at pp.c:410
#7  0x00005582897024a2 in Perl_runops_debug (my_perl=0x558289bf02a0) at dump.c:2574
#8  0x000055828965aee7 in Perl_call_sv (my_perl=0x558289bf02a0, sv=<optimized out>, flags=<optimized out>) at perl.c:3092
#9  0x000055828966395b in Perl_call_list (my_perl=0x558289bf02a0, oldscope=6, paramList=0x558289c258f0) at perl.c:5193
#10 0x000055828963ee34 in S_process_special_blocks (my_perl=my_perl@entry=0x558289bf02a0, floor=floor@entry=159, fullname=<optimized out>, fullname@entry=0x558289c35428 "ESSAGES/libc.mo", gv=0x558289c25ef0, cv=cv@entry=0x558289c25a28)
    at op.c:11749
#11 0x000055828964083d in Perl_newATTRSUB_x (my_perl=0x558289bf02a0, floor=<optimized out>, o=<optimized out>, proto=<optimized out>, attrs=<optimized out>, block=0x558289c3ad48, o_is_gv=<optimized out>) at op.c:11675
#12 0x00005582896a7fc4 in Perl_yyparse (my_perl=0x558289bf02a0, gramtype=<optimized out>) at perly.y:307
#13 0x00005582897c5489 in S_doeval_compile (my_perl=my_perl@entry=0x558289bf02a0, gimme=gimme@entry=2 '\002', outside=outside@entry=0x0, seq=<optimized out>, hh=hh@entry=0x0) at pp_ctl.c:3544
#14 0x00005582897c84df in S_require_file (sv=<optimized out>, my_perl=0x558289bf02a0) at pp_ctl.c:4367
#15 Perl_pp_require (my_perl=0x558289bf02a0) at pp_ctl.c:4391
#16 0x00005582897024a2 in Perl_runops_debug (my_perl=0x558289bf02a0) at dump.c:2574
#17 0x000055828965aee7 in Perl_call_sv (my_perl=0x558289bf02a0, sv=<optimized out>, flags=<optimized out>) at perl.c:3092
#18 0x000055828966395b in Perl_call_list (my_perl=0x558289bf02a0, oldscope=2, paramList=0x558289c25710) at perl.c:5193
#19 0x000055828963ee34 in S_process_special_blocks (my_perl=my_perl@entry=0x558289bf02a0, floor=floor@entry=45, fullname=<optimized out>, fullname@entry=0x558289c2de08 "BEGIN", gv=0x558289c25728, cv=cv@entry=0x558289c25698) at op.c:11749
#20 0x000055828964083d in Perl_newATTRSUB_x (my_perl=0x558289bf02a0, floor=<optimized out>, o=<optimized out>, proto=<optimized out>, attrs=<optimized out>, block=0x558289c2db88, o_is_gv=<optimized out>) at op.c:11675
#21 0x000055828963813b in Perl_utilize (my_perl=0x558289bf02a0, aver=<optimized out>, floor=45, version=<optimized out>, idop=0x558289c2d360, arg=<optimized out>) at op.c:8837
#22 0x00005582896a6443 in Perl_yyparse (my_perl=0x558289bf02a0, gramtype=<optimized out>) at perly.y:347
#23 0x00005582896605f7 in S_parse_body (xsinit=0x558289624c40 <xs_init>, env=0x0, my_perl=0x558289bf02a0) at perl.c:2577
#24 perl_parse (my_perl=0x558289bf02a0, xsinit=0x558289624c40 <xs_init>, argc=<optimized out>, argv=<optimized out>, env=0x0) at perl.c:1872
#25 0x00005582896243b5 in main (argc=<optimized out>, argv=<optimized out>, env=<optimized out>) at perlmain.c:126

Workaround Code wishing to use strict (including indirectly via other modules) and also utilize -DXvt must instead use the following before any use strict statements occur:

BEGIN {
require strict; # Preload strict.pm, bypassing the -DXvt crash
$^D = "Xvt";
}

Perl configuration

$ debugperl -V
Summary of my perl5 (revision 5 version 32 subversion 1) configuration:

  Platform:
    osname=linux
    osvers=4.19.0
    archname=x86_64-linux-gnu-thread-multi
    uname='linux localhost 4.19.0 #1 smp debian 4.19.0 x86_64 gnulinux '
    config_args='-Dmksymlinks -Dusethreads -Duselargefiles -Dcc=x86_64-linux-gnu-gcc -Dcpp=x86_64-linux-gnu-cpp -Dld=x86_64-linux-gnu-gcc -Dccflags=-DDEBIAN -Wdate-time -D_FORTIFY_SOURCE=2 -g -O2 -ffile-prefix-map=/dummy/build/dir=. -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -Dldflags= -Wl,-Bsymbolic-functions -flto=auto -Wl,-z,relro -Dlddlflags=-shared -Wl,-Bsymbolic-functions -flto=auto -Wl,-z,relro -Dcccdlflags=-fPIC -Darchname=x86_64-linux-gnu -Dprefix=/usr -Dprivlib=/usr/share/perl/5.32 -Darchlib=/usr/lib/x86_64-linux-gnu/perl/5.32 -Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5 -Dvendorarch=/usr/lib/x86_64-linux-gnu/perl5/5.32 -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl/5.32.1 -Dsitearch=/usr/local/lib/x86_64-linux-gnu/perl/5.32.1 -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3 -Dsiteman1dir=/usr/local/man/man1 -Dsiteman3dir=/usr/local/man/man3 -Duse64bitint -Dman1ext=1 -Dman3ext=3perl -Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Ud_ualarm -Uusesfio -Uusenm -Ui_libutil -Ui_xlocale -Uversiononly -DDEBUGGING=-g -Doptimize=-O2 -dEs -Duseshrplib -Dlibperl=libperl.so.5.32.1'
    hint=recommended
    useposix=true
    d_sigaction=define
    useithreads=define
    usemultiplicity=define
    use64bitint=define
    use64bitall=define
    uselongdouble=undef
    usemymalloc=n
    default_inc_excludes_dot=define
    bincompat5005=undef
  Compiler:
    cc='x86_64-linux-gnu-gcc'
    ccflags ='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fwrapv -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64'
    optimize='-O2 -g'
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fwrapv -fno-strict-aliasing -pipe -I/usr/local/include'
    ccversion=''
    gccversion='10.3.0'
    gccosandvers=''
    intsize=4
    longsize=8
    ptrsize=8
    doublesize=8
    byteorder=12345678
    doublekind=3
    d_longlong=define
    longlongsize=8
    d_longdbl=define
    longdblsize=16
    longdblkind=3
    ivtype='long'
    ivsize=8
    nvtype='double'
    nvsize=8
    Off_t='off_t'
    lseeksize=8
    alignbytes=8
    prototype=define
  Linker and Libraries:
    ld='x86_64-linux-gnu-gcc'
    ldflags =' -fstack-protector-strong -L/usr/local/lib'
    libpth=/usr/local/lib /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib /lib64 /usr/lib64
    libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt
    perllibs=-ldl -lm -lpthread -lc -lcrypt
    libc=libc-2.33.so
    so=so
    useshrplib=true
    libperl=libperl.so.5.32
    gnulibc_version='2.33'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs
    dlext=so
    d_dlsymun=undef
    ccdlflags='-Wl,-E'
    cccdlflags='-fPIC'
    lddlflags='-shared -L/usr/local/lib -fstack-protector-strong'

Characteristics of this binary (from libperl):
  Compile-time options:
    DEBUGGING
    HAS_TIMES
    MULTIPLICITY
    PERLIO_LAYERS
    PERL_COPY_ON_WRITE
    PERL_DONT_CREATE_GVSV
    PERL_IMPLICIT_CONTEXT
    PERL_MALLOC_WRAP
    PERL_OP_PARENT
    PERL_PRESERVE_IVUV
    PERL_TRACK_MEMPOOL
    USE_64_BIT_ALL
    USE_64_BIT_INT
    USE_ITHREADS
    USE_LARGE_FILES
    USE_LOCALE
    USE_LOCALE_COLLATE
    USE_LOCALE_CTYPE
    USE_LOCALE_NUMERIC
    USE_LOCALE_TIME
    USE_PERLIO
    USE_PERL_ATOF
    USE_REENTRANT_API
    USE_THREAD_SAFE_LOCALE
  Locally applied patches:
    DEBPKG:debian/cpan_definstalldirs - Provide a sensible INSTALLDIRS default for modules installed from CPAN.
    DEBPKG:debian/db_file_ver - https://bugs.debian.org/340047 Remove overly restrictive DB_File version check.
    DEBPKG:debian/doc_info - Replace generic man(1) instructions with Debian-specific information.
    DEBPKG:debian/enc2xs_inc - https://bugs.debian.org/290336 Tweak enc2xs to follow symlinks and ignore missing @INC directories.
    DEBPKG:debian/errno_ver - https://bugs.debian.org/343351 Remove Errno version check due to upgrade problems with long-running processes.
    DEBPKG:debian/libperl_embed_doc - https://bugs.debian.org/186778 Note that libperl-dev package is required for embedded linking
    DEBPKG:fixes/respect_umask - Respect umask during installation
    DEBPKG:debian/writable_site_dirs - Set umask approproately for site install directories
    DEBPKG:debian/extutils_set_libperl_path - EU:MM: set location of libperl.a under /usr/lib
    DEBPKG:debian/no_packlist_perllocal - Don't install .packlist or perllocal.pod for perl or vendor
    DEBPKG:debian/fakeroot - Postpone LD_LIBRARY_PATH evaluation to the binary targets.
    DEBPKG:debian/instmodsh_doc - Debian policy doesn't install .packlist files for core or vendor.
    DEBPKG:debian/ld_run_path - Remove standard libs from LD_RUN_PATH as per Debian policy.
    DEBPKG:debian/libnet_config_path - Set location of libnet.cfg to /etc/perl/Net as /usr may not be writable.
    DEBPKG:debian/perlivp - https://bugs.debian.org/510895 Make perlivp skip include directories in /usr/local
    DEBPKG:debian/squelch-locale-warnings - https://bugs.debian.org/508764 Squelch locale warnings in Debian package maintainer scripts
    DEBPKG:debian/patchlevel - https://bugs.debian.org/567489 List packaged patches for 5.32.1-3ubuntu2.1 in patchlevel.h
    DEBPKG:fixes/document_makemaker_ccflags - https://bugs.debian.org/628522 [rt.cpan.org #68613] Document that CCFLAGS should include $Config{ccflags}
    DEBPKG:debian/find_html2text - https://bugs.debian.org/640479 Configure CPAN::Distribution with correct name of html2text
    DEBPKG:debian/perl5db-x-terminal-emulator.patch - https://bugs.debian.org/668490 Invoke x-terminal-emulator rather than xterm in perl5db.pl
    DEBPKG:debian/cpan-missing-site-dirs - https://bugs.debian.org/688842 Fix CPAN::FirstTime defaults with nonexisting site dirs if a parent is writable
    DEBPKG:fixes/memoize_storable_nstore - [rt.cpan.org #77790] https://bugs.debian.org/587650 Memoize::Storable: respect 'nstore' option not respected
    DEBPKG:debian/makemaker-pasthru - https://bugs.debian.org/758471 Pass LD settings through to subdirectories
    DEBPKG:debian/makemaker-manext - https://bugs.debian.org/247370 Make EU::MakeMaker honour MANnEXT settings in generated manpage headers
    DEBPKG:debian/kfreebsd-softupdates - https://bugs.debian.org/796798 Work around Debian Bug#796798
    DEBPKG:fixes/memoize-pod - [rt.cpan.org #89441] Fix POD errors in Memoize
    DEBPKG:debian/hurd-softupdates - https://bugs.debian.org/822735 Fix t/op/stat.t failures on hurd
    DEBPKG:fixes/math_complex_doc_great_circle - https://bugs.debian.org/697567 [rt.cpan.org #114104] Math::Trig: clarify definition of great_circle_midpoint
    DEBPKG:fixes/math_complex_doc_see_also - https://bugs.debian.org/697568 [rt.cpan.org #114105] Math::Trig: add missing SEE ALSO
    DEBPKG:fixes/math_complex_doc_angle_units - https://bugs.debian.org/731505 [rt.cpan.org #114106] Math::Trig: document angle units
    DEBPKG:fixes/cpan_web_link - https://bugs.debian.org/367291 CPAN: Add link to main CPAN web site
    DEBPKG:debian/hppa_op_optimize_workaround - https://bugs.debian.org/838613 Temporarily lower the optimization of op.c on hppa due to gcc-6 problems
    DEBPKG:debian/installman-utf8 - https://bugs.debian.org/840211 Generate man pages with UTF-8 characters
    DEBPKG:debian/hppa_opmini_optimize_workaround - https://bugs.debian.org/869122 Lower the optimization level of opmini.c on hppa
    DEBPKG:debian/sh4_op_optimize_workaround - https://bugs.debian.org/869373 Also lower the optimization level of op.c and opmini.c on sh4
    DEBPKG:debian/perldoc-pager - https://bugs.debian.org/870340 [rt.cpan.org #120229] Fix perldoc terminal escapes when sensible-pager is less
    DEBPKG:debian/prune_libs - https://bugs.debian.org/128355 Prune the list of libraries wanted to what we actually need.
    DEBPKG:debian/mod_paths - Tweak @INC ordering for Debian
    DEBPKG:debian/configure-regen - https://bugs.debian.org/762638 Regenerate Configure et al. after probe unit changes
    DEBPKG:debian/deprecate-with-apt - https://bugs.debian.org/747628 Point users to Debian packages of deprecated core modules
    DEBPKG:debian/disable-stack-check - https://bugs.debian.org/902779 [GH #16607] Disable debugperl stack extension checks for binary compatibility with perl
    DEBPKG:debian/perlbug-editor - https://bugs.debian.org/922609 Use "editor" as the default perlbug editor, as per Debian policy
    DEBPKG:debian/eu-mm-perl-base - https://bugs.debian.org/962138 Suppress an ExtUtils::MakeMaker warning about our non-default @INC
    DEBPKG:fixes/hurd-cachepropagate-test-fix - https://bugs.debian.org/963214 GNU/Hurd doesn't support SO_PROTOCOL
    DEBPKG:fixes/io_socket_ip_ipv6 - Disable getaddrinfo(3) AI_ADDRCONFIG for localhost and IPv4 numeric addresses
    DEBPKG:disable-libperl-tests -
    DEBPKG:CVE-2021-36770.patch - [PATCH] mitigate @INC pollution when loading ConfigLocal
  Built under linux
  Compiled at Aug  2 2021 12:24:15
  %ENV:
    PERL5LIB="/home/aquanight/perl5/lib/perl5"
    PERL_LOCAL_LIB_ROOT="/home/aquanight/perl5"
    PERL_MB_OPT="--install_base "/home/aquanight/perl5""
    PERL_MM_OPT="INSTALL_BASE=/home/aquanight/perl5"
  @INC:
    /home/aquanight/perl5/lib/perl5/5.32.1/x86_64-linux-gnu-thread-multi
    /home/aquanight/perl5/lib/perl5/5.32.1
    /home/aquanight/perl5/lib/perl5/x86_64-linux-gnu-thread-multi
    /home/aquanight/perl5/lib/perl5
    /etc/perl
    /usr/local/lib/x86_64-linux-gnu/perl/5.32.1
    /usr/local/share/perl/5.32.1
    /usr/lib/x86_64-linux-gnu/perl5/5.32
    /usr/share/perl5
    /usr/lib/x86_64-linux-gnu/perl/5.32
    /usr/share/perl/5.32
    /home/aquanight/perl5/lib/perl5/5.32.0
    /home/aquanight/perl5/lib/perl5/5.32.0/x86_64-linux-gnu-thread-multi
    /usr/local/lib/site_perl

Description Under -DDEBUGGING, the debug flags Xvt (-DXvt or $^D = "Xvt" at runtime) crashes when strict.pm is loaded via require/use.

Steps to Reproduce Perl must be built with -DDEBUGGING perl -DXvt -Mstrict -e 'print "Hello\n"'

Problem reproduced on blead on FreeBSD-12:

$ uname -mrs
FreeBSD 12.3-RELEASE amd64

$ ./perl -v | head -2 | tail -1
This is perl 5, version 35, subversion 10 (v5.35.10 (v5.35.9-18-g714a0a851b)) built for amd64-freebsd-thread-multi

Config args:
$ sh ./Configure -des -Dusedevel -Duseithreads -Dcc=clang10 -DDEBUGGING

Bisecting with:

./Porting/bisect.pl --start v5.20.0 --end v5.22.1 -DDEBUGGING --target miniperl \
  --crash -- ./miniperl -Ilib -DXvt -Mstrict -e 1

points to commit ed958fa315. It looks like the crash is triggered by the attempt to inject a couple of constant subs in a BEGIN block: removing the prototype from all_bits and all_explicit_bits is enough to stop it crashing.

I'll try to look further tomorrow if I can make time, but please don't let that stop anyone else from investigating. :)

It seems the const sub needs to close over a lexical variable that has ??gone out of scope - if I move the declarations of my($inline_all_bits, $inline_all_explicit_bits) above the BEGIN block the crash disappears, but I can reproduce the crash with eg:

./miniperl -DXv -e 'BEGIN { my $c = 1; *subc = sub () { $c }; }'

If I recall correctly, isn't the sub () { $lex } pattern for capturing a lexical as a constant? Devel::Peek shows the resulting CV of that pattern is flagged as both CONST and XSUB:

$ debugperl -MDevel::Peek -E 'BEGIN { my $lex = 42; *foo = sub () { $lex } } Dump(\&foo); say foo;'
SV = IV(0x55c798161510) at 0x55c798161520
  REFCNT = 1
  FLAGS = (TEMP,ROK)
  RV = 0x55c7981f0990
  SV = PVCV(0x55c7981c1b18) at 0x55c7981f0990
    REFCNT = 2
    FLAGS = (POK,pPOK,ANON,CONST,CVGV_RC,DYNFILE,ISXSUB)
    PROTOTYPE = ""
    COMP_STASH = 0x0
    XSUB = 0x55c797322000
    XSUBANY = 0x55c7981f09a8 (CONST SV)
    SV = IV(0x55c7981f0998) at 0x55c7981f09a8
      REFCNT = 2
      FLAGS = (PADTMP,IOK,READONLY,PROTECT,pIOK)
      IV = 42
    GVGV::GV = 0x55c7981f08d0   "main" :: "__ANON__"
    FILE = "-e"
    DEPTH = 0
    FLAGS = 0x148c
    OUTSIDE_SEQ = 0
    HSCXT = 0xefefefefefefefef
    OUTSIDE = 0x0 (null)
42

The XSUB flag appears to be responsible for triggering the assert in S_cv_dump (called during Perl_cv_clone under -DXv), due to this:

    const CV * const outside = CvOUTSIDE(cv);
    PADLIST* const padlist = CvPADLIST(cv);

The CvPADLIST macro asserts because cv is an XSUB.

On Sat, 26 Feb 2022 at 03:41, Hugo van der Sanden @.***> wrote:

Bisecting with:

./Porting/bisect.pl --start v5.20.0 --end v5.22.1 -DDEBUGGING --target miniperl \ --crash -- ./miniperl -Ilib -DXvt -Mstrict -e 1

How/why did you decide to bisect that version range? I mean that patch is OLD.

points to commit ed958fa https://github.com/Perl/perl5/commit/ed958fa3156084f3cf4d8c4768716d9e1a11ce91. It looks like the crash is triggered by the attempt to inject a couple of constant subs in a BEGIN block: removing the prototype from all_bits and all_explicit_bits is enough to stop it crashing.

I'll try to look further tomorrow if I can make time, but please don't let that stop anyone else from investigating. :)

Well if it werent for the version range and bisect results you posted here I would have guessed this was related to some recent work on subroutine stubs.

Ill take a look tomorrow.

Yves

-- perl -Mre=debug -e "/just|another|perl|hacker/"

How/why did you decide to bisect that version range? I mean that patch is OLD.

Before bisecting, I tested with an assortment of debugging perls I happened to have installed. But note that the OP was reporting against 5.32.1.

I'll take a look tomorrow.

Ok, I'll leave it to you.

On Sat, 26 Feb 2022 at 17:13, Hugo van der Sanden @.***> wrote:

How/why did you decide to bisect that version range? I mean that patch is OLD.

Before bisecting, I tested with an assortment of debugging perls I happened to have installed. But note that the OP was reporting against 5.32.1.

I'll take a look tomorrow.

Ok, I'll leave it to you.

I was able to minimize the failure to:

./perl -Ilib -DXv -e '{ my $n=1; *foo= sub () { $n }; }'

and I pushed a fix for this in

https://github.com/Perl/perl5/pull/19466

I haven't set up a bisect to determine exactly when this broke yet, but I will leave one running while I go to breakfast and maybe update the PR to reflect that info.

Thanks for the report aquanight, our testing of -D switches is pretty impoverished, we should fix that.

Cheers, Yves

-- perl -Mre=debug -e "/just|another|perl|hacker/"

On Sun, 27 Feb 2022 at 03:52, demerphq @.***> wrote:

I haven't set up a bisect to determine exactly when this broke yet, but I will leave one running while I go to breakfast and maybe update the PR to reflect that info.

Bisect says it was 5.21.6, commit 1567c65ac069266bfe65959430c185babd476538.

../perl2/Porting/bisect.pl --start v5.18.4 --end ed958fa3156084f3cf4d8c4768716d9e1a11ce91 -DDEBUGGING --target miniperl --crash -- ./miniperl -Ilib -DXv -e '{ my $n=1; *foo= sub () { $n }; }'

I pushed an update to the PR to reflect this.

Hugo, thanks for the example of how to use bisect for stuff like this, it is very useful!

cheers, yves

-- perl -Mre=debug -e "/just|another|perl|hacker/"

On Sun, 27 Feb 2022 at 03:52, demerphq @.**> wrote: I haven't set up a bisect to determine exactly when this broke yet, but I will leave one running while I go to breakfast and maybe update the PR to reflect that info. Bisect says it was 5.21.6, commit 1567c65. ../perl2/Porting/bisect.pl --start v5.18.4 --end ed958fa -DDEBUGGING --target miniperl --crash -- ./miniperl -Ilib -DXv -e '{ my $n=1; foo= sub () { $n }; }' I pushed an update to the PR to reflect this. Hugo, thanks for the example of how to use bisect for stuff like this, it is very useful! cheers, yves … -- perl -Mre=debug -e "/just|another|perl|hacker/"

Two-stage bisection! This deserves entry as an example in the documentation for bisection. See https://github.com/Perl/perl5/pull/19468.

[snip]

I was able to minimize the failure to: ./perl -Ilib -DXv -e '{ my $n=1; *foo= sub () { $n }; }'

How you were able to achieve this reduction would be a good subject for a pull request. It's more general than bisection, so perhaps a modification to something like pod/perldebtut.pod.

[snip]

I was able to minimize the failure to: ./perl -Ilib -DXv -e '{ my $n=1; *foo= sub () { $n }; }'

How you were able to achieve this reduction would be a good subject for a pull request. It's more general than bisection, so perhaps a modification to something like pod/perldebtut.pod.

Well I posted much of the reduction the previous day, it's not clear if Yves saw that and refined from there.

Generally, such reduction is always a mix of art and science. The difficult part is usually finding the patience and perseverance to proceed one step at a time.

In this case, the first part of the process was instinct: I saw that the code change found by the first bisect introduced some const subs, and having some awareness of the special handling they involve I immediately focused on them as a possible source of the problem.

Step 1 was to validate that guess. I know that the prototype is the trigger for making a sub constant, and removing the prototype should leave the code acting the same except without the const sub optimization. So I tested that, found that on removing the prototype it no longer crashed, and thus confirmed the focus.

If a crash is reproducible with miniperl, that's always a helpful reduction in itself. So I tried that, and found that it was indeed reproducible.

Step 3 was to see if I could short-circuit the minimization process by jumping straight to a guess:

  % ./miniperl -DXv -e 'my $foo = 1; sub bar () { $foo }'

.. but that did not crash. So I needed to proceed in smaller steps.

The next target was to get the sub declaration outside the BEGIN block, to remove another source of potential obfuscation. But the subs were using lexical variables declared and assigned inside the block. So before I could move the sub declarations, I first had to move the variable declarations, in this case to before the block.

So I tried just moving the variable declarations - but again that change was enough to stop it crashing. So I reasoned that maybe the original variable going out of scope was also critical to the crash, tried:

  % ./miniperl -DXv -e 'BEGIN { my $c = 1; *subc = sub () { $c }; }'

.. found that that was enough to reproduce the crash, decided it was getting late, and posted it. :)

I really have no idea though how to turn a description of reducing a specific case like this into something more general-purpose like a tutorial.

On Mon, 28 Feb 2022, 00:17 Hugo van der Sanden, @.***> wrote:

[snip]

I was able to minimize the failure to: ./perl -Ilib -DXv -e '{ my $n=1; *foo= sub () { $n }; }'

How you were able to achieve this reduction would be a good subject for a pull request. It's more general than bisection, so perhaps a modification to something like pod/perldebtut.pod.

Well I posted much of the reduction the previous day, it's not clear if Yves saw that and refined from there.

Yeah I saw it, I just ground it down a bit more by removing the BEGIN, and removing the t option from the -D flags.

Generally, such reduction is always a mix of art and science. The difficult

part is usually finding the patience and perseverance to proceed one step at a time.

Agreed and well put.

In this case, the first part of the process was instinct: I saw that the code change found by the first bisect introduced some const subs, and having some awareness of the special handling they involve I immediately focused on them as a possible source of the problem.

Step 1 was to validate that guess. I know that the prototype is the trigger for making a sub constant, and removing the prototype should leave the code acting the same except without the const sub optimization. So I tested that, found that on removing the prototype it no longer crashed, and thus confirmed the focus.

Minor note, with the case of a closure it's a combo of refcount==1 AND the prototype. That is why hoisting the var to package level "fixes" the problem, the refcount ends up at 2 ( one from the enclosing sub and one from the file level my declaration) thus the resulting sub is no longer marked CONST and relevant to this case no longer an XSUB.

If a crash is reproducible with miniperl, that's always a helpful reduction

in itself. So I tried that, and found that it was indeed reproducible.

I normally haven't used miniperl as a lot of regex bugs aren't revealed easily with it, whereas full perl basically won't build if the regex engine is even a little bit broken. The build process itself utilizes a wide range of regex features so if it builds usually any bugs left are esoteric or minor.

But working recently with Hugo, on this case and others, and especially seeing the speed impact on this bisect has taught me a new and much deeper appreciation for using it, and I now see it as a new tool in my toolbox. Thanks Hugo.

Step 3 was to see if I could short-circuit the minimization process by

jumping straight to a guess:

% ./miniperl -DXv -e 'my $foo = 1; sub bar () { $foo }'

.. but that did not crash. So I needed to proceed in smaller steps.

The next target was to get the sub declaration outside the BEGIN block, to remove another source of potential obfuscation. But the subs were using lexical variables declared and assigned inside the block. So before I could move the sub declarations, I first had to move the variable declarations, in this case to before the block.

So I tried just moving the variable declarations - but again that change was enough to stop it crashing. So I reasoned that maybe the original variable going out of scope was also critical to the crash, tried:

% ./miniperl -DXv -e 'BEGIN { my $c = 1; *subc = sub () { $c }; }'

.. found that that was enough to reproduce the crash, decided it was getting late, and posted it. :)

And I picked up from there... BEGIN blocks are special blocks so I tried various things to verify if they were relevant, including duplicating some of the steps that Hugo outlines above, but especially removing the BEGIN, and trying variants the code with and without BEGIN and with and without the -D switches and with and without Devel::Peek assistance (it loads strict so it was not possible to use it to debug some variants as it would trigfer the assert when it was used). I also checked to see if all the switches provided originaly were required. I also checked if this happens with a BEGIN or block wrapping a named sub with a true constant inside triggers the issue, (it doesn't) etc.

At some point I was able to determine that this happens when the anonymous sub created during compilation is cloned and assigned to the glob.

I then ran the minimized code under gdb, after the assert triggered typed 'bt' to get a back trace (c level stack trace) which told me the bug came from an assert at a specific line in cv_dump() in pad.c, inspecting this line I found CvPADLIST() which is a macro. Inspecting the macro definition I found the assert was to ensure that the macro shouldn't be used on CVs marked as XSUB, along with a comment that maybe the assert can be removed one day.

I then spent some time working out why this sub is marked as an XSUB even though it started off as perl, the XSUB flag means a XS sub so it's a bit weird a perl sub gets marked that way. My thought was that setting the flag might be an error so I tried commenting it out. That didn't help, removing it just changed the assert fail to a SEGV, suggesting the flag was likely correctly applied. I knew at some point FC had worked on optimizing constant subs, and he was the author of the commit that broke so I decided that probably the two were related and likely he had just never tested the -DXv switch on his patches; the -D switches have very little tests, and personally I have never used most of them, and never ever used this particular combination, so it made sense to me that FC might have missed them too.

So I then inspected cv_dump() and realized that the bulk of the code in it didn't need the padlist data, so I decide to move the CvPADLIST() call and the logic that used it into an if block guarded by the CvISXSUB() check, verified that it fixed the bug and didn't create any others and the adjusted debug output was reasonable and then finally created the PR.

Note that at this time I had not run the second bisect, I had put that off on the assumption it would take a long time. But once I had created the PR I wanted to improve the commit message and verify my assumptions about the XSUB flag. In order to do this I needed to find the change that actually broke the test case. Hugo's bisect was about when strict started dying, not for the change that caused that change to die so it gave me an end point but not a start point. I happened to have 5.18.4 handy to test and it did not die so that gave me a start point to start the second bisect.

So I then kicked off the bisect. What I did not expect was that it would run as quickly as it did, I expected that it would take a while and started getting ready to go out for breakfast with my wife with the expectation I would not see results until I returned sometime later. However Hugo's trick of using miniperl in the bisect meant it had completed before I had even left the house, IOW in a matter of minutes! That result confirmed my initial analysis, I updated the PR before we left and the task was done.

I really have no idea though how to turn a description of reducing a

specific case like this into something more general-purpose like a tutorial.

Maybe James can take what we both have written here and put something together, after all he has a bit more distance from it than we do.

I think some key points here are (in no specific order, every situation is different and depends on the bug and your personal prior knowledge).

Minimize the test case. Try to eliminate everything not essential replicating the error. This is a process of trial and error, removing things, checking for the error, adding them back if necessary, etc.
Determine the scope of the bug, this is similar to minimizing but involves more addition and substitution. Eg, in perl there are a lot of ways to do things, try as many variants as possible to see which are affected and which not. In this case the scope was narrow, nearly any chanfe to the minimized case made the assert not fire.
Use bisect to find the commit that caused the failure. This may be done first as in this case, or after minimization depending on the bug.
Use gdb to get a back trace to find where in the C code the error is triggered.
Use Devel::Peek::Dump to get a better picture of what is going on. Eg. In this case this was very helpful in inspecting what perl was doing when this code NOT run under the -DXv flag, allowing us to see the CONST and XSUB flags on the sub(s) we were debugging.
Do code analysis of what failed. Poke and prod. Try various tweaks of the C code to learn more about the failure, eg commenting out the assert to see what happened. Maybe add trace output to see what is happening during execution (not needed here, but often needed in other bugs) possibly with the view to make such trace permanent for future devs. (Hence the plethora of -D switches and debug options for the regex engine).
Use git blame to step through the history of the code involved. The -w and -L$line,+$num_lines flags, combined with the caret commit notation are very helpful at times. (although this was not needed for this case)
Read the docs, either perls or for various standards (eg Unicode specs), or for other tools (regex bugs especially), or even sometimes Wikipedia pages (very helpful for some things, like how utf8 encoding works or whatnot)
Possibly repeat previous steps depending on intuition and the results seen, combined with running tests. Maybe a tweak fixes the immediate bug but breaks something else.
Ask for help or guidance on #p5p or the porters mailing list. More eyes often makes things easier and faster. Likely someone knows something about the matter at hand.
Be persistent. It is just code, eventually if you are stubborn enough you will win. Don't be intimidated that you don't know what is needed, you can always learn new things to compensate, so just dive in and see where you end up.

One thing I really like about digging into Perl bug reports is I almost always learn something new, be it about perl itself, C, gdb, or just general coding or debugging techniques or a combination. The result I feel is every bug I investigate, particularly when I find a fix but even sometimes when I don't, makes me a better programmer. If I concede defeat then I watch what Nicholas or Dave M or Karl (or whoever, but its usually one of them) do to fix the bug and learn from that. Fixing Perl bugs is not just a matter of grinding off an esoteric rough edge in perl itself, I find that I make myself better, and more useful, to this project and future projects and future employers. In this case I learned:

Bisect with miniperl, especially using Porting/bisect.pl is a much more powerful and speedy tool than I had previously understood. (Thanks again Hugo!)
Perl constant subs are magically replaced with special XS equivalents. I knew they were optimized but now I know how.
That the -DXvt switches exist and what they do. Including the related logic of cv_dump() and friends, including various debugging utility subs I was previously unaware of. Prior to this bug I knew next to nothing about pads beyond their higher level use and existence. Now I know a bit more about them.
That our tests for -D switches can at best be described as incomplete if not outright nonexistent. I had thought the regex debugging options were poorly tested, but the -D switches are worse and in comparison the regex engine ones are quite a bit better.
That pad.c contains a ton of code that I should learn more about someday.
What the assert_() macro means and does. (Note trailing undebar).

So for me personally this bug was fun and educational even if it only fixes a very esoteric part of perl and I think it was worth my time and effort to dig into it.

HTH, Cheers Yves

Status of ticket: waiting for code review of p.r. in https://github.com/Perl/perl5/pull/19466

On Wed, 2 Mar 2022, 21:25 James E Keenan, @.***> wrote:

Status of ticket: waiting for code review of p.r. in #19466 https://github.com/Perl/perl5/pull/19466

I was going to merge it today. I don't think there is much to review, the patch is trivial. But I'll give it a few more days if you want?

Yves

On 3/2/22 09:23, Yves Orton wrote:

On Wed, 2 Mar 2022, 21:25 James E Keenan, @.***> wrote:

Status of ticket: waiting for code review of p.r. in #19466 https://github.com/Perl/perl5/pull/19466

I was going to merge it today. I don't think there is much to review, the patch is trivial. But I'll give it a few more days if you want?

As we're nearing code freeze, the value of a code review for anything touching the internals increases significantly. That's why I've requested that it be reviewed by people familiar with the internals.

On Wed, 2 Mar 2022 at 16:30, James E Keenan @.***> wrote:

On 3/2/22 09:23, Yves Orton wrote:

On Wed, 2 Mar 2022, 21:25 James E Keenan, @.***> wrote:

Status of ticket: waiting for code review of p.r. in #19466 https://github.com/Perl/perl5/pull/19466

I was going to merge it today. I don't think there is much to review, the patch is trivial. But I'll give it a few more days if you want?

As we're nearing code freeze, the value of a code review for anything touching the internals increases significantly. That's why I've requested that it be reviewed by people familiar with the internals.

It's debug code, so i wouldn't worry about that, especially as if it doesn't get patched we either ship an assert fail or a segv and -DXv will be useless. But it's been useless since 5.20 so I guess it's not used much.

Plus the patch was trivial. But sure, I added Karl too, as a C expert if not an expert on this part of the internals. :-)

cheers, Yves

Thanks for the report, this will be included in 5.36.0

Perl / perl5

-DXvt crashes in strict.pm #19463