Perl / perl5

🐪 The Perl programming language
https://dev.perl.org/perl5/
Other
1.91k stars 542 forks source link

Bleadperl breaks OALDERS/LWP-Protocol-https-6.07.tar.gz on Windows #16418

Closed p5pRT closed 5 years ago

p5pRT commented 6 years ago

Migrated from rt.perl.org#132863 (status was 'resolved')

Searchable as RT132863$

p5pRT commented 6 years ago

From @chorny

Created by @chorny

https_proxy.t hangs on 5.27.8\, did not hang on 5.27.6\, 5.27.1\, 5.26.0. All tests of https_proxy.t pass\, it just does not exit.

I'm using Windows XP SP3 Russian 32-bit.

Also reported here​: https://github.com/libwww-perl/LWP-Protocol-https/issues/52

Perl Info ``` Flags: category=core severity=high Site configuration information for perl 5.27.8: Configured by strawberry-perl at Tue Feb 13 01:56:34 2018. Summary of my perl5 (revision 5 version 27 subversion 8) configuration: Platform: osname=MSWin32 osvers=5.1.2600 archname=MSWin32-x86-multi-thread-64int uname='Win32 strawberry-perl 5.27.8.1-beta1 #1 Tue Feb 13 01:53:53 2018 i386' config_args='undef' hint=recommended useposix=true d_sigaction=undef useithreads=define usemultiplicity=define use64bitint=define use64bitall=undef uselongdouble=undef usemymalloc=n default_inc_excludes_dot=define bincompat5005=undef Compiler: cc='gcc' ccflags =' -s -O2 -DWIN32 -D__USE_MINGW_ANSI_STDIO -DPERL_TEXTMODE_SCRIPTS -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -DUSE_PERLIO -fwrapv -fno-strict-aliasing -mms-bitfields' optimize='-s -O2' cppflags='-DWIN32' ccversion='' gccversion='7.1.0' gccosandvers='' intsize=4 longsize=4 ptrsize=4 doublesize=8 byteorder=12345678 doublekind=3 d_longlong=define longlongsize=8 d_longdbl=define longdblsize=12 longdblkind=3 ivtype='long long' ivsize=8 nvtype='double' nvsize=8 Off_t='long long' lseeksize=8 alignbytes=8 prototype=define Linker and Libraries: ld='g++.exe' ldflags ='-s -L"H:\perl\278\perl\lib\CORE" -L"H:\perl\278\c\lib"' libpth=H:\perl\278\c\lib H:\perl\278\c\i686-w64-mingw32\lib H:\perl\278\c\lib\gcc\i686-w64-mingw32\7.1.0 libs= -lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool -lcomdlg32 -ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_32 -lmpr -lwinmm -lversion -lodbc32 -lodbccp32 -lcomctl32 perllibs= -lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool -lcomdlg32 -ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_32 -lmpr -lwinmm -lversion -lodbc32 -lodbccp32 -lcomctl32 libc= so=dll useshrplib=true libperl=libperl527.a gnulibc_version='' Dynamic Linking: dlsrc=dl_win32.xs dlext=xs.dll d_dlsymun=undef ccdlflags=' ' cccdlflags=' ' lddlflags='-mdll -s -L"H:\perl\278\perl\lib\CORE" -L"H:\perl\278\c\lib"' @INC for perl 5.27.8: H:/perl/278/perl/site/lib H:/perl/278/perl/vendor/lib H:/perl/278/perl/lib Environment for perl 5.27.8: HOME (unset) LANG (unset) LANGUAGE (unset) LD_LIBRARY_PATH (unset) LOGDIR (unset) PATH=H:\perl\278\perl\site\bin;H:\perl\278\perl\bin;H:\perl\278\c\bin;C:\Program Files\Far\;C:\Prg\PHP\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program Files\ATI Technologies\ATI.ACE\Core-Static;C:\strawberry514\c\bin;C:\strawberry514\perl\site\bin;C:\strawberry514\perl\bin;C:\Program Files\TortoiseHg\;C:\Program Files\MySQL\MySQL Server 5.5\bin;c:\util\;C:\Prg\Subversion\bin;C:\Program Files\TortoiseSVN\bin;D:\vagrant\vagrant\bin;C:\Prg\TortoiseGit\bin;C:\Prg\Git\cmd;C:\Program Files\Skype\Phone\; PERL_BADLANG (unset) SHELL (unset) -- Alexandr Ciornii, http://chorny.net ```
p5pRT commented 6 years ago

From @jkeenan

On Tue\, 13 Feb 2018 11​:43​:20 GMT\, chorny wrote​:

This is a bug report for perl from alexchorny@​gmail.com\, generated with the help of perlbug 1.41 running under perl 5.27.8.

----------------------------------------------------------------- [Please describe your issue here]

https_proxy.t hangs on 5.27.8\, did not hang on 5.27.6\, 5.27.1\, 5.26.0. All tests of https_proxy.t pass\, it just does not exit.

I'm using Windows XP SP3 Russian 32-bit.

Also reported here​: https://github.com/libwww-perl/LWP-Protocol- https/issues/52

I see that the test file which is hanging has code in it like this​:

##### defined( my $pid = fork()) or die "fork failed​: $!";

# child process runs _server and exits if ( ! $pid ) {   @​childs = ();   exit( _server()); } #####

... and as well​:

##### $ grep -n goto t/https_proxy.t 205​: my $cl = $server->accept or goto ACCEPT; 215​: goto ACCEPT; 230​: goto ACCEPT; # wait for next connection 285​: goto SSL_UPGRADE; 306​: goto REQUEST if $keep_alive; #####

'goto' is cited in commit messages for the following commits in the last 3 months​:

##### 436908e565f0e613465123e7cb08fa54487c3b8f b537774295099f6b543a9e2b7375f72593328389 e7afb05e35570e271ae017d47b64dd5aad3e2009 a01f4640266aacbed7ecc9df01890abb555c69b2 e57923a2c4fa758db6d61eb407b8ffb8ac38f5b2 da4e040f42421764ef069371d77c008e6b801f45 e57923a2c4fa758db6d61eb407b8ffb8ac38f5b2 6d90e98384148a470db6f66439a13e5955418298 4d7e83bba2968618bd9026ce17ae87f5529e5f38 4bfb5532d393d56b18d13bc19f70f6f7a64ae781 #####

Are you in a position to build perl at each of these commits and see which is the first in which that test hangs?

List​: Do people on other versions of Windows also get this hang?

Thank you very much.

-- James E Keenan (jkeenan@​cpan.org)

p5pRT commented 6 years ago

The RT System itself - Status changed from 'new' to 'open'

p5pRT commented 6 years ago

From @chorny

2018-02-14 16​:56 GMT+02​:00 James E Keenan via RT \perlbug\-followup@​perl\.org​:

On Tue\, 13 Feb 2018 11​:43​:20 GMT\, chorny wrote​:

https_proxy.t hangs on 5.27.8\, did not hang on 5.27.6\, 5.27.1\, 5.26.0. All tests of https_proxy.t pass\, it just does not exit.

I'm using Windows XP SP3 Russian 32-bit. Are you in a position to build perl at each of these commits and see which is the first in which that test hangs?

I've done manual bisect. I've used another module that hangs - IO​::Socket​::SSL. t/mitm.t hangs and t/session_ticket.t fails. Problem appear starting with commit 8e920bd341e241f50a74dbf8aa343319f204e200 - "set PERL_EXIT_DESTRUCT_END in all embeddings". Commit 503bc07b4b9e34ed04a725b2bc8faec1ae0f3be2 does not have such problems. Other affected distributions are Test-WWW-Mechanize-Catalyst and Test-HTTP-Server-Simple and possibly others.

When installing IO​::Socket​::SSL\, you should install MIKEM/Net-SSLeay-1.82.tar.gz first and skip it's test t/local/33_x509_create_cert.t.

-- Alexandr Ciornii\, http​://chorny.net

p5pRT commented 6 years ago

From @chorny

On Tue\, 13 Feb 2018 03​:43​:20 -0800\, chorny wrote​:

https_proxy.t hangs on 5.27.8\, did not hang on 5.27.6\, 5.27.1\, 5.26.0. All tests of https_proxy.t pass\, it just does not exit.

Similar error is in KAORU/HTTP-Async-0.33.tar.gz (https://rt.cpan.org/Ticket/Display.html?id=125169)

-- Alexandr Ciornii\, http​://chorny.net

p5pRT commented 6 years ago

From [Unknown Contact. See original ticket]

On Tue\, 13 Feb 2018 03​:43​:20 -0800\, chorny wrote​:

https_proxy.t hangs on 5.27.8\, did not hang on 5.27.6\, 5.27.1\, 5.26.0. All tests of https_proxy.t pass\, it just does not exit.

Similar error is in KAORU/HTTP-Async-0.33.tar.gz (https://rt.cpan.org/Ticket/Display.html?id=125169)

-- Alexandr Ciornii\, http​://chorny.net

p5pRT commented 6 years ago

From @jkeenan

On Sat\, 21 Apr 2018 22​:51​:22 GMT\, chorny wrote​:

On Tue\, 13 Feb 2018 03​:43​:20 -0800\, chorny wrote​:

https_proxy.t hangs on 5.27.8\, did not hang on 5.27.6\, 5.27.1\, 5.26.0. All tests of https_proxy.t pass\, it just does not exit.

Similar error is in KAORU/HTTP-Async-0.33.tar.gz (https://rt.cpan.org/Ticket/Display.html?id=125169)

It appears that we neglected to add this RT to the list of perl-5.28.0 blockers (RT 127689). I have done so now.

These failures appear to be Windows-specific. Just now I used cpanm to successfully install every module listed in this RT on both Linux and FreeBSD-11.1.

Thank you very much. -- James E Keenan (jkeenan@​cpan.org)

p5pRT commented 6 years ago

From @iabyn

On Thu\, Mar 08\, 2018 at 01​:06​:24AM +0200\, Alexandr Ciornii wrote​:

I've done manual bisect. I've used another module that hangs - IO​::Socket​::SSL. t/mitm.t hangs and t/session_ticket.t fails. Problem appear starting with commit 8e920bd341e241f50a74dbf8aa343319f204e200 - "set PERL_EXIT_DESTRUCT_END in all embeddings".

Zefram\, that commit was one of yours. I don't quite understand the implications of it\, but note that the test file which newly hangs on Windows contains both a fork() and

  END { kill 9\,@​childs if @​childs };

so perhaps an END block wasn't being called before\, or used to be called only once\, and is now being called multiple times or something?

Do you have any insight?

-- Dave's first rule of Opera​: If something needs saying\, say it​: don't warble it.

p5pRT commented 6 years ago

From @iabyn

On Mon\, Apr 23\, 2018 at 01​:39​:27PM +0100\, Dave Mitchell wrote​:

On Thu\, Mar 08\, 2018 at 01​:06​:24AM +0200\, Alexandr Ciornii wrote​:

I've done manual bisect. I've used another module that hangs - IO​::Socket​::SSL. t/mitm.t hangs and t/session_ticket.t fails. Problem appear starting with commit 8e920bd341e241f50a74dbf8aa343319f204e200 - "set PERL_EXIT_DESTRUCT_END in all embeddings".

Zefram\, that commit was one of yours. I don't quite understand the implications of it\, but note that the test file which newly hangs on Windows contains both a fork() and

END \{ kill 9\,@​childs if @​childs \};

so perhaps an END block wasn't being called before\, or used to be called only once\, and is now being called multiple times or something?

I've now pushed the branch smoke-me/davem/win32_exit\, which may fix this. Note that I've made the change in win32/perlhost.h blind\, as I haven't tried compiling or testing it on win32.

Could someone try it on win32 and see that (a) it doesn't break win32; (b) it fixes the distributions mentioned in this ticket​:

  LWP-Protocol-https-6.0   KAORU/HTTP-Async-0.33   IO​::Socket​::SSL

And ideally confirm my diagnosis that on fork() on win32\, END() blocks were being called twice in the child (and presumably once in the parent)\, but only once after my patch. If someone could write a test for this\, that would be even better\,

Here's the commit message​:

  win323 fork()​: honour PERL_EXIT_DESTRUCT_END  
  RT #132863  
  The PERL_EXIT_DESTRUCT_END flag in PL_exit_flags is designed to defer   the calling of END blocks in perl_run() to being called from   perl_destruct() instead. On UNIX-like builds\, perlmain.c sets this flag.   So main() looks like\, in outline​:  
  PL_exit_flags |= PERL_EXIT_DESTRUCT_END;   if (!perl_parse(my_perl\, xs_init\, argc\, argv\, (char **)NULL))   perl_run(my_perl);   exitstatus = perl_destruct(my_perl);  
  which means that it doesn't matter whether perl_parse() finishes   normally or prematurely (e.g. via BEGIN { exit(1) } or BEGIN { die });   in all cases\, due to PERL_EXIT_DESTRUCT_END being set\, the END blocks   will always be called (from perl_destruct()).  
  Commit v5.27.7-9-g8e920bd341 added PERL_EXIT_DESTRUCT_END to the   equivalent of main() on other platforms such as win32; this   means that the new tests in t/op/blocks.t pass on win32 too\, and all   platforms have the same behaviour for e.g.  
  END { print "end\n"; } BEGIN { exit 1; } # prints "end"  
  However\, that commit was causing some tests in some CPAN distributions to   hang. These were using the win32 fork() emulation.  
  PerlProcFork() on win32 clones an interpreter\, starts a new thread\,   and makes the new thread call win32_start_child()\, which   does the rough equivalent of C\<perl_run(); perl_destruct();>\,   except that it rolls it's own perl_run() equivalent which   *doesn't* honour the PERL_EXIT_DESTRUCT_END flag. So by setting   PERL_EXIT_DESTRUCT_END\, END blocks in fork()ed processes were getting   executed twice​: once by win32_start_child() emulating perl_run()\,   and once by perl_destruct() called from win32_start_child().  
  This commit makes win32_start_child() honour PERL_EXIT_DESTRUCT_END.

-- It's not that I'm afraid to die\, I just don't want to be there when it happens.   -- Woody Allen

p5pRT commented 6 years ago

From @steve-m-hay

On 30 April 2018 at 22​:15\, Dave Mitchell \davem@&#8203;iabyn\.com wrote​:

On Mon\, Apr 23\, 2018 at 01​:39​:27PM +0100\, Dave Mitchell wrote​:

On Thu\, Mar 08\, 2018 at 01​:06​:24AM +0200\, Alexandr Ciornii wrote​:

I've done manual bisect. I've used another module that hangs - IO​::Socket​::SSL. t/mitm.t hangs and t/session_ticket.t fails. Problem appear starting with commit 8e920bd341e241f50a74dbf8aa343319f204e200 - "set PERL_EXIT_DESTRUCT_END in all embeddings".

Zefram\, that commit was one of yours. I don't quite understand the implications of it\, but note that the test file which newly hangs on Windows contains both a fork() and

END \{ kill 9\,@&#8203;childs if @&#8203;childs \};

so perhaps an END block wasn't being called before\, or used to be called only once\, and is now being called multiple times or something?

I've now pushed the branch smoke-me/davem/win32_exit\, which may fix this. Note that I've made the change in win32/perlhost.h blind\, as I haven't tried compiling or testing it on win32.

Could someone try it on win32 and see that (a) it doesn't break win32; (b) it fixes the distributions mentioned in this ticket​:

    LWP\-Protocol\-https\-6\.0
    KAORU/HTTP\-Async\-0\.33
    IO&#8203;::Socket&#8203;::SSL

Your branch builds and passes all tests for me on Windows 7 with 32-bit VS2015.

I've tried out IO​::Socket​::SSL and find that in blead t/mitm.t hangs before doing anything. With your branch it now passes all 8 tests but then hangs instead of exiting. So it's an improvement of sorts\, but something is still amiss somewhere. However\, there's no improvement with t/session_ticket.t. With either build I get the following failure​:

D​:\Temp\IO-Socket-SSL-2.056-0>perl -Mblib t\session_ticket.t 1..6 # listen at 127.0.0.1​:51973 # listen at 127.0.0.1​:51974 access to server[0] creating new ticket key1 server[0] reused=0 # connect to 0​: success reuse=0 ok 1 - no initial session -> no reuse access to server[0] using current ticket secret server[0] reused=1 # connect to 0​: success reuse=1 ok 2 - reuse with the next session and secret[0] access to server[1] using current ticket secret server[1] reused=1 rotate secrets # connect to 1​: success reuse=1 ok 3 - reuse even though server changed\, since they share ticket secret access to server[1] using non-current ticket secret creating new ticket key2 server[1] reused=0 rotate secrets # connect to 1​: success reuse=0 ok 4 - reports non-reuse since server1 changed secret to secret[1] access to server[0] using non-current ticket secret creating new ticket key1 server[0] reused=0 # connect to 0​: success reuse=0 ok 5 - reports non-reuse on server0 since got ticket with secret[1] in last step access to server[0] using current ticket secret server[0] reused=1 # connect to 0​: success reuse=1 ok 6 - reuse again since got ticket with secret[0] in last step not ok 1 - select failed or timed out​: Bad file descriptor at t\session_ticket.t line 113. # # Failed test 'select failed or timed out​: Bad file descriptor at t\session_ticket.t line 113. # ' # at ./t/testlib.pl line 39.

I haven't tried an earlier (pre-8e920bd341e241f50a74dbf8aa343319f204e200) perl to confirm that this used to work. I will try to do that in the next day or two.

p5pRT commented 6 years ago

From @steve-m-hay

On 3 May 2018 at 18​:20\, Steve Hay \steve\.m\.hay@&#8203;googlemail\.com wrote​:

On 30 April 2018 at 22​:15\, Dave Mitchell \davem@&#8203;iabyn\.com wrote​:

On Mon\, Apr 23\, 2018 at 01​:39​:27PM +0100\, Dave Mitchell wrote​:

On Thu\, Mar 08\, 2018 at 01​:06​:24AM +0200\, Alexandr Ciornii wrote​:

I've done manual bisect. I've used another module that hangs - IO​::Socket​::SSL. t/mitm.t hangs and t/session_ticket.t fails. Problem appear starting with commit 8e920bd341e241f50a74dbf8aa343319f204e200 - "set PERL_EXIT_DESTRUCT_END in all embeddings".

Zefram\, that commit was one of yours. I don't quite understand the implications of it\, but note that the test file which newly hangs on Windows contains both a fork() and

END \{ kill 9\,@&#8203;childs if @&#8203;childs \};

so perhaps an END block wasn't being called before\, or used to be called only once\, and is now being called multiple times or something?

I've now pushed the branch smoke-me/davem/win32_exit\, which may fix this. Note that I've made the change in win32/perlhost.h blind\, as I haven't tried compiling or testing it on win32.

Could someone try it on win32 and see that (a) it doesn't break win32; (b) it fixes the distributions mentioned in this ticket​:

    LWP\-Protocol\-https\-6\.0
    KAORU/HTTP\-Async\-0\.33
    IO&#8203;::Socket&#8203;::SSL

Your branch builds and passes all tests for me on Windows 7 with 32-bit VS2015.

I've tried out IO​::Socket​::SSL and find that in blead t/mitm.t hangs before doing anything. With your branch it now passes all 8 tests but then hangs instead of exiting. So it's an improvement of sorts\, but something is still amiss somewhere. However\, there's no improvement with t/session_ticket.t. With either build I get the following failure​:

D​:\Temp\IO-Socket-SSL-2.056-0>perl -Mblib t\session_ticket.t 1..6 # listen at 127.0.0.1​:51973 # listen at 127.0.0.1​:51974 access to server[0] creating new ticket key1 server[0] reused=0 # connect to 0​: success reuse=0 ok 1 - no initial session -> no reuse access to server[0] using current ticket secret server[0] reused=1 # connect to 0​: success reuse=1 ok 2 - reuse with the next session and secret[0] access to server[1] using current ticket secret server[1] reused=1 rotate secrets # connect to 1​: success reuse=1 ok 3 - reuse even though server changed\, since they share ticket secret access to server[1] using non-current ticket secret creating new ticket key2 server[1] reused=0 rotate secrets # connect to 1​: success reuse=0 ok 4 - reports non-reuse since server1 changed secret to secret[1] access to server[0] using non-current ticket secret creating new ticket key1 server[0] reused=0 # connect to 0​: success reuse=0 ok 5 - reports non-reuse on server0 since got ticket with secret[1] in last step access to server[0] using current ticket secret server[0] reused=1 # connect to 0​: success reuse=1 ok 6 - reuse again since got ticket with secret[0] in last step not ok 1 - select failed or timed out​: Bad file descriptor at t\session_ticket.t line 113. # # Failed test 'select failed or timed out​: Bad file descriptor at t\session_ticket.t line 113. # ' # at ./t/testlib.pl line 39.

I haven't tried an earlier (pre-8e920bd341e241f50a74dbf8aa343319f204e200) perl to confirm that this used to work. I will try to do that in the next day or two.

I've now tried a build of the change prior to 8e920bd341e241f50a74dbf8aa343319f204e200 (namely\, 503bc07b4b9e34ed04a725b2bc8faec1ae0f3be2) and unfortunately both t/mitm.t and t/session_ticket.t pass all tests (and don't hang). That's on the same machine with the same compiler and build configuration.

So the problem here isn't yet fixed.

p5pRT commented 6 years ago

From @iabyn

On Fri\, May 04\, 2018 at 05​:58​:50PM +0100\, Steve Hay via perl5-porters wrote​:

So the problem here isn't yet fixed.

What does the following code output under Windows on 1) something pre v5.27.7-9-g8e920bd341 2) blead 3) my smoke-me/davem/win32_exit branch?

  $| = 1;   my $pid = fork();   die "fork​: $!\n" unless defined $pid;   my $who = $pid ? "parent" : "child";   END { print "END​: $who\n" }

-- Overhead\, without any fuss\, the stars were going out.   -- Arthur C Clarke

p5pRT commented 6 years ago

From @steve-m-hay

On 4 May 2018 at 20​:39\, Dave Mitchell \davem@&#8203;iabyn\.com wrote​:

On Fri\, May 04\, 2018 at 05​:58​:50PM +0100\, Steve Hay via perl5-porters wrote​:

So the problem here isn't yet fixed.

What does the following code output under Windows on 1) something pre v5.27.7-9-g8e920bd341 2) blead 3) my smoke-me/davem/win32_exit branch?

$| = 1;
my $pid = fork\(\);
die "fork&#8203;: $\!\\n" unless defined $pid;
my $who = $pid ? "parent" : "child";
END \{ print "END&#8203;: $who\\n" \}

They all output​:

END​: parent

p5pRT commented 6 years ago

From @iabyn

On Fri\, May 04\, 2018 at 09​:36​:17PM +0100\, Steve Hay via perl5-porters wrote​:

On 4 May 2018 at 20​:39\, Dave Mitchell \davem@&#8203;iabyn\.com wrote​:

On Fri\, May 04\, 2018 at 05​:58​:50PM +0100\, Steve Hay via perl5-porters wrote​:

So the problem here isn't yet fixed.

What does the following code output under Windows on 1) something pre v5.27.7-9-g8e920bd341 2) blead 3) my smoke-me/davem/win32_exit branch?

$| = 1;
my $pid = fork\(\);
die "fork&#8203;: $\!\\n" unless defined $pid;
my $who = $pid ? "parent" : "child";
END \{ print "END&#8203;: $who\\n" \}

They all output​:

END​: parent

Hmmm.

Could you show the output from the attached script for each of the perls described above\, thanks?

-- I before E. Except when it isn't.

p5pRT commented 6 years ago

From @iabyn

#!/usr/bin/perl

use warnings; use strict;

my ($pid\, @​pid); my $type = 'parent';

END {   print STDERR "END​: $type​: killing (@​pid)\n";   kill 9\,@​pid;   print STDERR "END​: $type​: done kill (@​pid)\n"; }

$pid = fork; die $! unless defined $pid; if (!$pid) {   $type = 'child';   while (1) {   print STDERR "child​: sleep(1)\n";   sleep(1);   }   print STDERR "child​: exit(1)\n";   exit(1); }

push @​pid\, $pid; print STDERR "parent​: sleep(4)\n"; sleep(4); print STDERR "parent​: finishing\n";

p5pRT commented 6 years ago

From @steve-m-hay

On 8 May 2018 at 14​:27\, Dave Mitchell \davem@&#8203;iabyn\.com wrote​:

On Fri\, May 04\, 2018 at 09​:36​:17PM +0100\, Steve Hay via perl5-porters wrote​:

On 4 May 2018 at 20​:39\, Dave Mitchell \davem@&#8203;iabyn\.com wrote​:

On Fri\, May 04\, 2018 at 05​:58​:50PM +0100\, Steve Hay via perl5-porters wrote​:

So the problem here isn't yet fixed.

What does the following code output under Windows on 1) something pre v5.27.7-9-g8e920bd341 2) blead 3) my smoke-me/davem/win32_exit branch?

$| = 1;
my $pid = fork\(\);
die "fork&#8203;: $\!\\n" unless defined $pid;
my $who = $pid ? "parent" : "child";
END \{ print "END&#8203;: $who\\n" \}

They all output​:

END​: parent

Hmmm.

Could you show the output from the attached script for each of the perls described above\, thanks?

Pre-8e920bd341 outputs this​:

parent​: sleep(4) child​: sleep(1) child​: sleep(1) child​: sleep(1) child​: sleep(1) parent​: finishing END​: parent​: killing (-12584) END​: parent​: done kill (-12584)

Blead and your branch both output this​:

parent​: sleep(4) child​: sleep(1) child​: sleep(1) child​: sleep(1) child​: sleep(1) parent​: finishing child​: sleep(1) child​: sleep(1) child​: sleep(1) child​: sleep(1) child​: sleep(1) child​: sleep(1) child​: sleep(1) child​: sleep(1) child​: sleep(1) child​: sleep(1) child​: sleep(1) child​: sleep(1) child​: sleep(1) child​: sleep(1) child​: sleep(1) child​: sleep(1) child​: sleep(1) [...] Ctrl+C to kill it Terminating on signal SIGINT(2)

p5pRT commented 6 years ago

From @iabyn

On Tue\, May 08\, 2018 at 05​:57​:10PM +0100\, Steve Hay via perl5-porters wrote​:

Pre-8e920bd341 outputs this​: ... END​: parent​: killing (-12584)

Blead and your branch both output this​: [...] Ctrl+C to kill it

Thanks for that\, I finally know what's going on.

TL;DR​: a can of worms has been opened\, and for 5.28 we should close the can by reverting for now the 8e920bd341 PERL_EXIT_DESTRUCT_END change (plus the tests)\, then re-address this issue post 5.28.

Long form​:

As I mentioned earlier in this thread\, the main() function on UNIXYy builds looks like

  PL_exit_flags |= PERL_EXIT_DESTRUCT_END;   if (!perl_parse(my_perl...)   perl_run(my_perl);   exitstatus = perl_destruct(my_perl);

Setting the PERL_EXIT_DESTRUCT_END flag moves the calling of END blocks from being handled at the end of perl_run() to being handled at the beginning of perl_destruct(). This means that a premature exit from perl_parse()\, such as BEGIN { exit(1) }\, doesn't skip calling END blocks.

Zefram's 8e920bd341 commit merely set the PERL_EXIT_DESTRUCT_END flag on other platforms such as win32\, so that they too would call END blocks even after BEGIN { exit(1) }.

However\, calling END blocks isn't *quite* the first thing in perl_destruct(); before that it does

  /* wait for all pseudo-forked children to finish */   PERL_WAIT_FOR_CHILDREN;

so by setting PERL_EXIT_DESTRUCT_END\, we've moved the calling of END blocks to *after* the reaping of pseudo-forked children.

It so happens that of the failing distributions and tests that I looked at\, they shared a common idiom​: fork() a child to start a test server which can be used to run tests against\, then at the end\, kill off any such children with

  END { kill 9\, @​child_pids }

Now on win32\, the END block won't get called until after the children have finished\, but the children won't finish until the have been killed by the parent calling the END block. Hence deadlock.

The fix may well be to move PERL_WAIT_FOR_CHILDREN to after the call_list(PL_endav) in perl_destruct()\, but I think messing about like that in late code freeze is asking for trouble\,

There are also wider issues. For example\, one of the short bits of code Steve ran for me earlier showed that the win32 fork() emulation wasn't calling END() blocks in the child.

We need to look at all the cases and decide whether and when END blocks should be called\, and if so fix them on platforms where they don't match expectations \, e.g.

  BEGIN { die }   BEGIN { exit 0 }   BEGIN { exit 1 }   die   exit 0   exit 1   fork; then in child​: eval 'BEGIN { die }'   fork; then in child​: eval 'BEGIN { exit 0 }'   fork; then in child​: eval 'BEGIN { exit 1 }'   fork; then in child​: die   fork; then in child​: exit 0   fork; then in child​: exit 1   fork; then from parent\, kill child   fork; then from parent\, kill 9\, child

-- Hofstadter's Law​: It always takes longer than you expect\, even when you take into account Hofstadter's Law.

p5pRT commented 6 years ago

From @iabyn

On Wed\, May 09\, 2018 at 09​:48​:27AM +0100\, Dave Mitchell wrote​:

TL;DR​: a can of worms has been opened\, and for 5.28 we should close the can by reverting for now the 8e920bd341 PERL_EXIT_DESTRUCT_END change (plus the tests)\, then re-address this issue post 5.28.

One final favour\, Steve​:

Could you run blead with 8e920bd341e241f50a74dbf8aa343319f204e200 reverted\, and and show me which tests t/op/blocks.t (or elsewhere) fail? Its' hard to determine which tests to skip without actually running it on a win32 platform.

Thanks.

-- This email is confidential\, and now that you have read it you are legally obliged to shoot yourself. Or shoot a lawyer\, if you prefer. If you have received this email in error\, place it in its original wrapping and return for a full refund. By opening this email\, you accept that Elvis lives.

p5pRT commented 6 years ago

From @steve-m-hay

On 9 May 2018 at 11​:04\, Dave Mitchell \davem@&#8203;iabyn\.com wrote​:

On Wed\, May 09\, 2018 at 09​:48​:27AM +0100\, Dave Mitchell wrote​:

TL;DR​: a can of worms has been opened\, and for 5.28 we should close the can by reverting for now the 8e920bd341 PERL_EXIT_DESTRUCT_END change (plus the tests)\, then re-address this issue post 5.28.

One final favour\, Steve​:

Could you run blead with 8e920bd341e241f50a74dbf8aa343319f204e200 reverted\, and and show me which tests t/op/blocks.t (or elsewhere) fail? Its' hard to determine which tests to skip without actually running it on a win32 platform.

Thanks.

Current blead with 8e920bd341e241f50a74dbf8aa343319f204e200 reverted passes all tests except t/op/blocks.t tests 10\, 11\, 13\, 14\, 16 and 17. Verbose output from this test is attached.

p5pRT commented 6 years ago

From @steve-m-hay

# Failed test 10 - BEGIN{exit 1} should exit at ./test.pl line 1062 # got "begin\nunitcheck\ncheck" # expected "begin\nunitcheck\ncheck\nend" # PROG​: # BEGIN { $| = 1; } BEGIN { print "begin\n"; } UNITCHECK { print "unitcheck\n"; } CHECK { print "check\n"; } INIT { print "init\n"; } END { print "end\n"; } print "main\n"; BEGIN { exit 1; } # STATUS​: 256 # Failed test 11 - BEGIN{die} should exit at ./test.pl line 1074 # got 'begin\nDied at - line 1.\nBEGIN failed--compilation aborted at - line 1.\nunitcheck\ncheck' # expected /(?^​:\Abegin\nDied[^\n]*\.\nBEGIN failed[^\n]*\.\nunitcheck\ncheck\nend\z)/ # PROG​: # BEGIN { $| = 1; } BEGIN { print "begin\n"; } UNITCHECK { print "unitcheck\n"; } CHECK { print "check\n"; } INIT { print "init\n"; } END { print "end\n"; } print "main\n"; BEGIN { die; } # STATUS​: 65280 # Failed test 13 - UNITCHECK{exit 1} should exit at ./test.pl line 1062 # got "begin\nunitcheck\ncheck" # expected "begin\nunitcheck\ncheck\nend" # PROG​: # BEGIN { $| = 1; } BEGIN { print "begin\n"; } UNITCHECK { print "unitcheck\n"; } CHECK { print "check\n"; } INIT { print "init\n"; } END { print "end\n"; } print "main\n"; UNITCHECK { exit 1; } # STATUS​: 256 # Failed test 14 - UNITCHECK{die} should exit at ./test.pl line 1074 # got 'begin\nDied at - line 1.\nUNITCHECK failed--call queue aborted.\nunitcheck\ncheck' # expected /(?^​:\Abegin\nDied[^\n]*\.\nUNITCHECK failed[^\n]*\.\nunitcheck\ncheck\nend\z)/ # PROG​: # BEGIN { $| = 1; } BEGIN { print "begin\n"; } UNITCHECK { print "unitcheck\n"; } CHECK { print "check\n"; } INIT { print "init\n"; } END { print "end\n"; } print "main\n"; UNITCHECK { die; } # STATUS​: 65280 # Failed test 16 - CHECK{exit 1} should exit at ./test.pl line 1062 # got "begin\nunitcheck\ncheck" # expected "begin\nunitcheck\ncheck\nend" # PROG​: # BEGIN { $| = 1; } BEGIN { print "begin\n"; } UNITCHECK { print "unitcheck\n"; } CHECK { print "check\n"; } INIT { print "init\n"; } END { print "end\n"; } print "main\n"; CHECK { exit 1; } # STATUS​: 256 # Failed test 17 - CHECK{die} should exit at ./test.pl line 1074 # got 'begin\nunitcheck\nDied at - line 1.\nCHECK failed--call queue aborted.\ncheck' # expected /(?^​:\Abegin\nunitcheck\nDied[^\n]*\.\nCHECK failed[^\n]*\.\ncheck\nend\z)/ # PROG​: # BEGIN { $| = 1; } BEGIN { print "begin\n"; } UNITCHECK { print "unitcheck\n"; } CHECK { print "check\n"; } INIT { print "init\n"; } END { print "end\n"; } print "main\n"; CHECK { die; } # STATUS​: 65280 op/blocks.t .. 1..22 ok 1 - Order of execution of special blocks ok 2 - blocks interact with packages/scopes ok 3 - can name blocks as sub FOO ok 4 - eval-UNITCHECK-eval (bug 70614) ok 5 - constant named after a special block ok 6 - context ok 7 - null PL_curcop in newGP ok 8 - blocks execute in right order ok 9 - BEGIN{exit 0} doesn't exit yet not ok 10 - BEGIN{exit 1} should exit not ok 11 - BEGIN{die} should exit ok 12 - UNITCHECK{exit 0} doesn't exit yet not ok 13 - UNITCHECK{exit 1} should exit not ok 14 - UNITCHECK{die} should exit ok 15 - CHECK{exit 0} doesn't exit yet not ok 16 - CHECK{exit 1} should exit not ok 17 - CHECK{die} should exit ok 18 - INIT{exit 0} should exit ok 19 - INIT{exit 1} should exit ok 20 - INIT{die} should exit not ok 21 - RT \#2917​: No constraint on how late INIT blocks can run # TODO RT #2917​: INIT{} in eval is wrongly considered too late # Failed test 21 - RT \#2917​: No constraint on how late INIT blocks can run at ./test.pl line 1062 # got "" # expected "in init" # PROG​: # eval "INIT { print qq(in init); };"; # STATUS​: 0 ok 22 - RT \#113934​: goto out of BEGIN causes assertion failure Failed 6/22 subtests

Test Summary Report


op/blocks.t (Wstat​: 0 Tests​: 22 Failed​: 6)   Failed tests​: 10-11\, 13-14\, 16-17 Files=1\, Tests=22\, 7 wallclock secs ( 0.01 usr + 0.00 sys = 0.01 CPU) Result​: FAIL

p5pRT commented 6 years ago

From @iabyn

On Wed\, May 09\, 2018 at 01​:15​:47PM +0100\, Steve Hay via perl5-porters wrote​:

Current blead with 8e920bd341e241f50a74dbf8aa343319f204e200 reverted passes all tests except t/op/blocks.t tests 10\, 11\, 13\, 14\, 16 and 17.

Thanks. I've just pushed smoke-me/davem/win32_exit_revert which contains the reversion of the PERL_EXIT_DESTRUCT_END plus skipping the relevant tests under win32 etc.

-- "Procrastination grows to fill the available time"   -- Mitchell's corollary to Parkinson's Law

p5pRT commented 6 years ago

From @xsawyerx

On 05/09/2018 11​:48 AM\, Dave Mitchell wrote​:

On Tue\, May 08\, 2018 at 05​:57​:10PM +0100\, Steve Hay via perl5-porters wrote​:

Pre-8e920bd341 outputs this​: ... END​: parent​: killing (-12584)

Blead and your branch both output this​: [...] Ctrl+C to kill it

Thanks for that\, I finally know what's going on.

TL;DR​: a can of worms has been opened\, and for 5.28 we should close the can by reverting for now the 8e920bd341 PERL_EXIT_DESTRUCT_END change (plus the tests)\, then re-address this issue post 5.28.

Thank you\, Dave.

We should apply the fix that does this.

[...]

p5pRT commented 6 years ago

From @steve-m-hay

On 9 May 2018 at 14​:20\, Dave Mitchell \davem@&#8203;iabyn\.com wrote​:

On Wed\, May 09\, 2018 at 01​:15​:47PM +0100\, Steve Hay via perl5-porters wrote​:

Current blead with 8e920bd341e241f50a74dbf8aa343319f204e200 reverted passes all tests except t/op/blocks.t tests 10\, 11\, 13\, 14\, 16 and 17.

Thanks. I've just pushed smoke-me/davem/win32_exit_revert which contains the reversion of the PERL_EXIT_DESTRUCT_END plus skipping the relevant tests under win32 etc.

This passes (or skips...) all tests for me.

p5pRT commented 6 years ago

From @iabyn

On Thu\, May 10\, 2018 at 06​:03​:40PM +0100\, Steve Hay via perl5-porters wrote​:

On 9 May 2018 at 14​:20\, Dave Mitchell \davem@&#8203;iabyn\.com wrote​:

On Wed\, May 09\, 2018 at 01​:15​:47PM +0100\, Steve Hay via perl5-porters wrote​:

Current blead with 8e920bd341e241f50a74dbf8aa343319f204e200 reverted passes all tests except t/op/blocks.t tests 10\, 11\, 13\, 14\, 16 and 17.

Thanks. I've just pushed smoke-me/davem/win32_exit_revert which contains the reversion of the PERL_EXIT_DESTRUCT_END plus skipping the relevant tests under win32 etc.

This passes (or skips...) all tests for me.

Now merged as v5.27.11-38-g64a9c78095. I'll remove this ticket from the blockers list.

-- Red sky at night - gerroff my land! Red sky at morning - gerroff my land!   -- old farmers' sayings #14

p5pRT commented 5 years ago

From @jkeenan

On Fri\, 11 May 2018 07​:46​:14 GMT\, davem wrote​:

On Thu\, May 10\, 2018 at 06​:03​:40PM +0100\, Steve Hay via perl5-porters wrote​:

On 9 May 2018 at 14​:20\, Dave Mitchell \davem@&#8203;iabyn\.com wrote​:

On Wed\, May 09\, 2018 at 01​:15​:47PM +0100\, Steve Hay via perl5- porters wrote​:

Current blead with 8e920bd341e241f50a74dbf8aa343319f204e200 reverted passes all tests except t/op/blocks.t tests 10\, 11\, 13\, 14\, 16 and 17.

Thanks. I've just pushed smoke-me/davem/win32_exit_revert which contains the reversion of the PERL_EXIT_DESTRUCT_END plus skipping the relevant tests under win32 etc.

This passes (or skips...) all tests for me.

Now merged as v5.27.11-38-g64a9c78095. I'll remove this ticket from the blockers list.

OP reports that problem was fixed in perl-5.28. See​: https://github.com/libwww-perl/LWP-Protocol-https/issues/52#event-2548830179

Resolving ticket.

-- James E Keenan (jkeenan@​cpan.org)

p5pRT commented 5 years ago

@jkeenan - Status changed from 'open' to 'resolved'