Perl / perl5

🐪 The Perl programming language
https://dev.perl.org/perl5/
Other
1.96k stars 555 forks source link

eval in package DB can't see caller's lexicals in certain cases #16230

Open p5pRT opened 7 years ago

p5pRT commented 7 years ago

Migrated from rt.perl.org#132414 (status was 'open')

Searchable as RT132414$

p5pRT commented 7 years ago

From @jimav

This is a bug report for perl from jim.avera@​gmail.com\, generated with the help of perlbug 1.40 running under perl 5.24.1.


I have a debugging library which uses eval in package DB to access variables in the caller's context (per perldoc -f eval).

However lexicals are sometimes not accessible\, and it seems to depend on "irrelevant" details such as whether some not-yet-executed code exists following the call in the same block.   I don't know if this is a regression\, but I noticed it shortly after upgrading to perl 5.24.1

In the example below\, if there is any statement following the call into package DB\, then it works as expected\, i.e.\, the lexicals $key and $value are visible.  But as shown the code dies with an eval error.

Can someone shed light on this? Is there anything my code inside the DB function can do to avoid this problem?

#!/usr/bin/perl use strict; use warnings;

package DB;

sub my_interpolate($) {   my($s) = @​_;   my $result = eval "$s"; die "eval '$s' Failed​: $@​" if $@​;   print "eval '$s' => $result\n"; }

package main;

my %hash = (A => 100\, B => 200);

while (my ($key\, $value) = each %hash) {   # **DIES HERE** with   #    eval '$key' Failed​: Global symbol "$key" requires explicit package name...   DB​::my_interpolate '$key';

  #1;    # un-comment this and the problem vanishes! }



Flags​:     category=core     severity=low


Site configuration information for perl 5.24.1​:

Configured by Debian Project at Sat Mar 18 17​:00​:39 UTC 2017.

Summary of my perl5 (revision 5 version 24 subversion 1) configuration​:

  Platform​:     osname=linux\, osvers=3.16.0\, archname=x86_64-linux-gnu-thread-multi     uname='linux localhost 3.16.0 #1 smp debian 3.16.0 x86_64 gnulinux '     config_args='-Dusethreads -Duselargefiles -Dcc=x86_64-linux-gnu-gcc -Dcpp=x86_64-linux-gnu-cpp -Dld=x86_64-linux-gnu-gcc -Dccflags=-DDEBIAN -Wdate-time -D_FORTIFY_SOURCE=2 -g -O2 -fdebug-prefix-map=/build/perl-gaNj0t/perl-5.24.1=. -fstack-protector-strong -Wformat -Werror=format-security -Dldflags= -Wl\,-Bsymbolic-functions -Wl\,-z\,relro -Dlddlflags=-shared -Wl\,-Bsymbolic-functions -Wl\,-z\,relro -Dcccdlflags=-fPIC -Darchname=x86_64-linux-gnu -Dprefix=/usr -Dprivlib=/usr/share/perl/5.24 -Darchlib=/usr/lib/x86_64-linux-gnu/perl/5.24 -Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5 -Dvendorarch=/usr/lib/x86_64-linux-gnu/perl5/5.24 -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl/5.24.1 -Dsitearch=/usr/local/lib/x86_64-linux-gnu/perl/5.24.1 -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3 -Dsiteman1dir=/usr/local/man/man1 -Dsiteman3dir=/usr/local/man/man3 -Dusesitecustomize -Duse64bitint -Dman1ext=1 -Dman3ext=3perl -Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Ud_ualarm -Uusesfio -Uusenm -Ui_libutil -Uversiononly -DDEBUGGING=-g -Doptimize=-O2 -dEs -Duseshrplib -Dlibperl=libperl.so.5.24.1'     hint=recommended\, useposix=true\, d_sigaction=define     useithreads=define\, usemultiplicity=define     use64bitint=define\, use64bitall=define\, uselongdouble=undef     usemymalloc=n\, bincompat5005=undef   Compiler​:     cc='x86_64-linux-gnu-gcc'\, ccflags ='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fwrapv -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64'\,     optimize='-O2 -g'\,     cppflags='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fwrapv -fno-strict-aliasing -pipe -I/usr/local/include'     ccversion=''\, gccversion='6.3.0 20170316'\, gccosandvers=''     intsize=4\, longsize=8\, ptrsize=8\, doublesize=8\, byteorder=12345678\, doublekind=3     d_longlong=define\, longlongsize=8\, d_longdbl=define\, longdblsize=16\, longdblkind=3     ivtype='long'\, ivsize=8\, nvtype='double'\, nvsize=8\, Off_t='off_t'\, lseeksize=8     alignbytes=8\, prototype=define   Linker and Libraries​:     ld='x86_64-linux-gnu-gcc'\, ldflags =' -fstack-protector-strong -L/usr/local/lib'     libpth=/usr/local/lib /usr/lib/gcc/x86_64-linux-gnu/6/include-fixed /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib     libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt     perllibs=-ldl -lm -lpthread -lc -lcrypt     libc=libc-2.24.so\, so=so\, useshrplib=true\, libperl=libperl.so.5.24     gnulibc_version='2.24'   Dynamic Linking​:     dlsrc=dl_dlopen.xs\, dlext=so\, d_dlsymun=undef\, ccdlflags='-Wl\,-E'     cccdlflags='-fPIC'\, lddlflags='-shared -L/usr/local/lib -fstack-protector-strong'

Locally applied patches​:     DEBPKG​:debian/cpan_definstalldirs - Provide a sensible INSTALLDIRS default for modules installed from CPAN.     DEBPKG​:debian/db_file_ver - https://bugs.debian.org/340047 Remove overly restrictive DB_File version check.     DEBPKG​:debian/doc_info - Replace generic man(1) instructions with Debian-specific information.     DEBPKG​:debian/enc2xs_inc - https://bugs.debian.org/290336 Tweak enc2xs to follow symlinks and ignore missing @​INC directories.     DEBPKG​:debian/errno_ver - https://bugs.debian.org/343351 Remove Errno version check due to upgrade problems with long-running processes.     DEBPKG​:debian/libperl_embed_doc - https://bugs.debian.org/186778 Note that libperl-dev package is required for embedded linking     DEBPKG​:fixes/respect_umask - Respect umask during installation     DEBPKG​:debian/writable_site_dirs - Set umask approproately for site install directories     DEBPKG​:debian/extutils_set_libperl_path - EU​:MM​: set location of libperl.a under /usr/lib     DEBPKG​:debian/no_packlist_perllocal - Don't install .packlist or perllocal.pod for perl or vendor     DEBPKG​:debian/fakeroot - Postpone LD_LIBRARY_PATH evaluation to the binary targets.     DEBPKG​:debian/instmodsh_doc - Debian policy doesn't install .packlist files for core or vendor.     DEBPKG​:debian/ld_run_path - Remove standard libs from LD_RUN_PATH as per Debian policy.     DEBPKG​:debian/libnet_config_path - Set location of libnet.cfg to /etc/perl/Net as /usr may not be writable.     DEBPKG​:debian/mod_paths - Tweak @​INC ordering for Debian     DEBPKG​:debian/prune_libs - https://bugs.debian.org/128355 Prune the list of libraries wanted to what we actually need.     DEBPKG​:fixes/net_smtp_docs - [rt.cpan.org #36038] https://bugs.debian.org/100195 Document the Net​::SMTP 'Port' option     DEBPKG​:debian/perlivp - https://bugs.debian.org/510895 Make perlivp skip include directories in /usr/local     DEBPKG​:debian/deprecate-with-apt - https://bugs.debian.org/747628 Point users to Debian packages of deprecated core modules     DEBPKG​:debian/squelch-locale-warnings - https://bugs.debian.org/508764 Squelch locale warnings in Debian package maintainer scripts     DEBPKG​:debian/skip-upstream-git-tests - Skip tests specific to the upstream Git repository     DEBPKG​:debian/patchlevel - https://bugs.debian.org/567489 List packaged patches for 5.24.1-2ubuntu1 in patchlevel.h     DEBPKG​:debian/skip-kfreebsd-crash - https://bugs.debian.org/628493 [perl #96272] Skip a crashing test case in t/op/threads.t on GNU/kFreeBSD     DEBPKG​:fixes/document_makemaker_ccflags - https://bugs.debian.org/628522 [rt.cpan.org #68613] Document that CCFLAGS should include $Config{ccflags}     DEBPKG​:debian/find_html2text - https://bugs.debian.org/640479 Configure CPAN​::Distribution with correct name of html2text     DEBPKG​:debian/perl5db-x-terminal-emulator.patch - https://bugs.debian.org/668490 Invoke x-terminal-emulator rather than xterm in perl5db.pl     DEBPKG​:debian/cpan-missing-site-dirs - https://bugs.debian.org/688842 Fix CPAN​::FirstTime defaults with nonexisting site dirs if a parent is writable     DEBPKG​:fixes/memoize_storable_nstore - [rt.cpan.org #77790] https://bugs.debian.org/587650 Memoize​::Storable​: respect 'nstore' option not respected     DEBPKG​:debian/regen-skip - Skip a regeneration check in unrelated git repositories     DEBPKG​:debian/makemaker-pasthru - https://bugs.debian.org/758471 Pass LD settings through to subdirectories     DEBPKG​:debian/makemaker-manext - https://bugs.debian.org/247370 Make EU​::MakeMaker honour MANnEXT settings in generated manpage headers     DEBPKG​:debian/devel-ppport-reproducibility - https://bugs.debian.org/801523 Sort the list of XS code files when generating RealPPPort.xs     DEBPKG​:debian/encode-unicode-bom-doc - https://bugs.debian.org/798727 Document Debian backport of Encode​::Unicode fix     DEBPKG​:debian/kfreebsd-softupdates - https://bugs.debian.org/796798 Work around Debian Bug#796798     DEBPKG​:fixes/autodie-scope - https://bugs.debian.org/798096 Fix a scoping issue with "no autodie" and the "system" sub     DEBPKG​:fixes/crosscompile-no-targethost - [23695c0] [perl #127234] Fix the Configure escape with usecrosscompile but no targethost     DEBPKG​:fixes/memoize-pod - [rt.cpan.org #89441] Fix POD errors in Memoize     DEBPKG​:fixes/ok-pod - Added encoding for pod.     DEBPKG​:debian/hurd-softupdates - https://bugs.debian.org/822735 Fix t/op/stat.t failures on hurd     DEBPKG​:fixes/nntp_docs - https://bugs.debian.org/51962 Net​::NNTP​: Correct innd/nnrpd confusion in relation to Reader option     DEBPKG​:fixes/math_complex_doc_great_circle - https://bugs.debian.org/697567 [rt.cpan.org #114104] Math​::Trig​: clarify definition of great_circle_midpoint     DEBPKG​:fixes/math_complex_doc_see_also - https://bugs.debian.org/697568 [rt.cpan.org #114105] Math​::Trig​: add missing SEE ALSO     DEBPKG​:fixes/math_complex_doc_angle_units - https://bugs.debian.org/731505 [rt.cpan.org #114106] Math​::Trig​: document angle units     DEBPKG​:fixes/cpan_web_link - https://bugs.debian.org/367291 CPAN​: Add link to main CPAN web site     DEBPKG​:fixes/time_piece_doc - https://bugs.debian.org/817925 Time​::Piece​: Improve documentation for add_months and add_years     DEBPKG​:fixes/perlbug-refactor - https://bugs.debian.org/822463 [perl #128020] perlbug​: Refactor duplicated file reading code     DEBPKG​:fixes/perlbug-linewrap - https://bugs.debian.org/822463 [perl #128020] perlbug​: wrap overly long lines     DEBPKG​:fixes/hurd_sigaction - https://bugs.debian.org/825016 [d54f4ed] ext/POSIX/t/sigaction.t​: Skip uid and pid tests on GNU/Hurd     DEBPKG​:fixes/hurd_hints - [4694301] https://bugs.debian.org/825020 [perl #128279] Modify hints for Hurd per Debian ticket 825020.     DEBPKG​:fixes/extutils-parsexs-reproducibility - [perl #128517] https://bugs.debian.org/829296 Make the output of ExtUtils​::ParseXS reproducible     DEBPKG​:debian/CVE-2016-1238/sitecustomize-in-etc - Look for sitecustomize.pl in /etc/perl rather than sitelib on Debian systems     DEBPKG​:debian/CVE-2016-1238/test-suite-without-dot - [perl #127810] Patch unit tests to explicitly insert "." into @​INC when needed.     DEBPKG​:debian/CVE-2016-1238/eumm-without-dot - [perl #127810] Add PERL_USE_UNSAFE_INC support to EU​::MM for fortify_inc support.     DEBPKG​:debian/CVE-2016-1238/cpan-without-dot - [perl #127810] Set PERL_USE_UNSAFE_INC for cpan usage     DEBPKG​:debian/document_inc_removal - Document in perlvar that we remove '.' from @​INC by default     DEBPKG​:fixes/extutils_makemaker_reproducible - https​://bugs.debian.org/835815 https://bugs.debian.org/834190 Make perllocal.pod files reproducible     DEBPKG​:debian/CVE-2016-1238/remove-inc-test - Remove test for '.' in @​INC as it might not be     DEBPKG​:debian/customized - Update customized.dat for files patched in Debian     DEBPKG​:fixes/file_path_hurd_errno - File-Path​: Fix test failure in Hurd due to hard-coded ENOENT     DEBPKG​:debian/hppa_op_optimize_workaround - https://bugs.debian.org/838613 Temporarily lower the optimization of op.c on hppa due to gcc-6 problems     DEBPKG​:fixes/test-builder-warning - https://bugs.debian.org/840968 Silence a 'used only once' warning in Test​::Builder     DEBPKG​:fixes/longdblinf-randomness - [dd68853] [perl #130133] https://bugs.debian.org/844752 Configure​: fix garbage filtering with 80-bit long doubles     DEBPKG​:debian/installman-utf8 - https://bugs.debian.org/840211 Generate man pages with UTF-8 characters     DEBPKG​:fixes/list_assign_leak - [1050723] [perl #130766] https://bugs.debian.org/855064 avoid a leak in list assign from/to magic values     DEBPKG​:fixes/perlfunc_inc_doc - [a03e9f8] https://bugs.debian.org/839536 [perl #130832] Documentation fixes for '.' possibly no longer being in @​INC DEBPKG​:fixes/Compress-Raw-Zlib-2.071-Adapt-tests-to-zlib-1.2.11.patch - [PATCH] Adapt tests to zlib 1.2.11     DEBPKG​:fixes/0001-Adapt-tests-to-zlib-1.2.11.patch - [PATCH] Adapt tests to zlib 1.2.11


@​INC for perl 5.24.1​: /home/jima/perl5/lib/perl5/5.24.1/x86_64-linux-gnu-thread-multi     /home/jima/perl5/lib/perl5/5.24.1     /home/jima/perl5/lib/perl5/x86_64-linux-gnu-thread-multi     /home/jima/perl5/lib/perl5     /home/jima/lib/perl     /etc/perl     /usr/local/lib/x86_64-linux-gnu/perl/5.24.1     /usr/local/share/perl/5.24.1     /usr/lib/x86_64-linux-gnu/perl5/5.24     /usr/share/perl5     /usr/lib/x86_64-linux-gnu/perl/5.24     /usr/share/perl/5.24     /usr/local/lib/site_perl     /usr/lib/x86_64-linux-gnu/perl-base


Environment for perl 5.24.1​:     HOME=/home/jima     LANG=en_US.UTF-8     LANGUAGE=en_US     LC_COLLATE=C     LD_LIBRARY_PATH (unset)     LOGDIR (unset) PATH=/home/jima/perl5/bin​:/home/jima/bin​:/home/jima/jima_tools/x86_64/bin​:/home/jima/jima_tools/bin​:/usr/bin​:/bin​:/usr/sbin​:/sbin​:/usr/bin/X11​:/usr/local/bin​:/usr/local/sbin​:/usr/games​:/usr/local/games​:/usr/lib/jvm/java-8-oracle/bin​:/usr/lib/jvm/java-8-oracle/db/bin​:/usr/lib/jvm/java-8-oracle/jre/bin​:.     PERL5LIB=/home/jima/perl5/lib/perl5​:/home/jima/lib/perl     PERL_BADLANG (unset)     PERL_LOCAL_LIB_ROOT=/home/jima/perl5     PERL_MB_OPT=--install_base "/home/jima/perl5"     PERL_MM_OPT=INSTALL_BASE=/home/jima/perl5     SHELL=/bin/bash

p5pRT commented 6 years ago

From zefram@fysh.org

I don't know if this is a regression\, but I noticed it shortly after upgrading to perl 5.24.1

It's a regression in 5.13.7. Bisect fingers commit eae48c8938e50ebb341a72c2886c5ae8587092a5 "refactor and regularise label/statement grammar". That commit was not intended to change this eval behaviour.

Is there anything my code inside the DB function can do to avoid this problem?

No.

-zefram

p5pRT commented 6 years ago

The RT System itself - Status changed from 'new' to 'open'

p5pRT commented 6 years ago

From zefram@fysh.org

This issue is about nextstate ops being elided. A simple failing case is​:

$ perl5.26.0 -lwe '{ package DB; sub f { print eval "\$x" ? "ok" : "fail"; } } if(++(my $x)) { DB​::f(); }' fail

If you look at the optree\, you'll see that the body of the if() had a nextstate op at compile time\, but it got nulled out​:

6 \<;> nextstate(main 5 -e​:1) v​:{ ->7 - \<1> null vK/1 ->d 9 \<|> and(other->a) vK/1 ->d 8 \<1> preinc sK/1 ->9 7 \<0> padsv[$x​:6\,9] sPRM/LVINTRO ->8 - \<@​> scope vK ->- - \<;> ex-nextstate(main 8 -e​:1) v ->a c \<1> entersub vKS ->d - \<1> ex-list K ->c a \<0> pushmark s ->b - \<1> ex-rv2cv sK/1 ->- b \<#> gv[*DB​::f] s ->c

This matters because the eval from DB-land goes looking up the call chain for the last call from a non-DB location\, and uses the lexical state represented by whatever COP was current (i.e.\, most recently executed) at the point of that call. Without the elision\, the current COP at the time of the entersub would be the immediately preceding nextstate op with sequence number 8\, which would mean that $x would be in scope (sequence range (6\, 9]). But with that COP elided\, the current COP is the one preceding the whole if() construct\, with sequence number 5\, leaving $x out of scope.

Note that $x is nevertheless in scope for the subroutine call as judged at compile time of that call. You can pass $x to the sub\, for example\, and that'll be the lexical $x. That works because at compile time lexical scoping is determined on the basis of the COPs as they're generated\, before elision happens. So you can happily pass the lexical $x to a sub that at runtime doesn't see that $x being in scope at the call site.

Adding a no-op statement such as "1;" inside the block\, either before or after the sub call\, can prevent elision of the nextstate op\, making the eval behave correctly.

I said this was a regression in 5.13.7\, but that's not entirely correct. The test case provided by the bug reporter\, with a while() loop\, starts failing in 5.13.7\, and that can be seen by changing "if(++(my $x))" in the above one-liner to "while(++(my $x) > $a++)" or similar. But the if() example fails all the way back to pre-5.6 perls. It's manifesting by different routes\, but there's basically a problem that our compilation code doesn't appreciate the importance of having the correct COP current at the time of a sub call.

If the downside of this problem were only evals from the DB package going wrong\, we might accept that as the price of an optimisation\, especially since it only applies when not in debug mode. (With -d on the command line\, the nextstate ops become dbstate ops\, and are never elided\, avoiding the problem.) But there are other things controlled by COPs. Runtime warnings seem to be OK​: changing lexical warning state prevents COP elision\, afaics. But since 5.9.4 COPs have stored a shadow copy of the compile-time %^H\, accessible by called subroutines through (caller(0))[10]. It's semantically important for that to be correct\, but it's not​:

$ perl5.26.0 -lwe 'sub f { print +((caller(0))[10] || {})->{foo} || "fail"; } if(++(my $x)) { BEGIN { $^H{foo} = "ok"; } f(); }' fail $ perl5.10.0 -lwe 'sub f { print +((caller(0))[10] || {})->{foo} || "fail"; } if(++(my $x)) { BEGIN { $^H{foo} = "ok"; } f(); }' fail

Weirdly\, although the while() version of the eval problem didn't occur prior to 5.13.7\, the while() version of the %^H problem *does* occur prior to 5.13.7. There are multiple nextstate ops in play here\, and the one that the called sub sees pre-5.13.7 has the lexical variable but not the lexical hint.

So generally there's going to have to be less nulling of nextstate ops\, at least around entersub ops.

-zefram