perl hangs when two files are redirected into a pipe

[p5pRT]

Migrated from rt.perl.org#1900 (status was 'resolved')

Searchable as RT1900$

[p5pRT]

From summer@OS2.ami.com.au

I took some code 'man perlfunc' and modified it to read thus​: #!/usr/bin/perl open(OLDOUT\, ">&STDOUT"); open(OLDERR\, ">&STDERR");

open(STDOUT\, "| cat >foo.out") || die "Can't redirect stdout"; # JCS open(STDERR\, ">&STDOUT") || die "Can't dup stdout";

select(STDERR); $| = 1; # make unbuffered select(STDOUT); $| = 1; # make unbuffered

print STDOUT "stdout 1\n"; # this works for print STDERR "stderr 1\n"; # subprocesses too

close(STDOUT); close(STDERR);

open(STDOUT\, ">&OLDOUT"); open(STDERR\, ">&OLDERR");

print STDOUT "stdout 2\n"; print STDERR "stderr 2\n";

The only line I changed is labeled JCS.

This hangs (waits indefinitely)\, apparently at one or other of the closes (lines 14/15). I presume cat's not getting EOF.

I'm running perl on RHL 5.x​: I'm not use I beleive this info;-) [summer@​emu summer]$ perl -V Summary of my perl5 (5.0 patchlevel 5 subversion 2) configuration​:   Platform​:   osname=linux\, osvers=2.0.35\, archname=i386-linux-thread   uname='linux caliban.xs4all.nl 2.0.35 #2 sat jul 18 01​:37​:18 cest 1998 i686 unknown '   hint=recommended\, useposix=true\, d_sigaction=define   usethreads=define useperlio=undef d_sfio=undef   Compiler​:   cc='cc'\, optimize='-O2'\, gccversion=2.7.2.3   cppflags='-D_REENTRANT -Dbool=char -DHAS_BOOL -I/usr/local/include'   ccflags ='-D_REENTRANT -Dbool=char -DHAS_BOOL -I/usr/local/include'   stdchar='char'\, d_stdstdio=define\, usevfork=false   intsize=4\, longsize=4\, ptrsize=4\, doublesize=8   d_longlong=define\, longlongsize=8\, d_longdbl=define\, longdblsize=12   alignbytes=4\, usemymalloc=n\, prototype=define   Linker and Libraries​:   ld='cc'\, ldflags =' -L/usr/local/lib'   libpth=/usr/local/lib /lib /usr/lib   libs=-lnsl -lndbm -lgdbm -ldb -ldl -lm -lpthread -lc -lposix -lcrypt   libc=\, so=so\, useshrplib=false\, libperl=libperl.a   Dynamic Linking​:   dlsrc=dl_dlopen.xs\, dlext=so\, d_dlsymun=undef\, ccdlflags='-rdynamic'   cccdlflags='-fpic'\, lddlflags='-shared -L/usr/local/lib'

Characteristics of this binary (from libperl)​:   Built under linux   Compiled at Aug 9 1998 10​:17​:03   @​INC​:   /usr/lib/perl5/5.00502/i386-linux-thread   /usr/lib/perl5/5.00502   /usr/lib/perl5/site_perl/5.005/i386-linux-thread   /usr/lib/perl5/site_perl/5.005   . [summer@​emu summer]$ This seems more probable​: [summer@​emu summer]$ uname -a Linux emu.os2.ami.com.au 2.2.12 #2 Sat Oct 2 21​:20​:39 WST 1999 i586 unknown [summer@​emu summer]$

It also happens on RHL 6.1 [summer@​emu summer]$ perl -V Summary of my perl5 (5.0 patchlevel 5 subversion 2) configuration​:   Platform​:   osname=linux\, osvers=2.0.35\, archname=i386-linux-thread   uname='linux caliban.xs4all.nl 2.0.35 #2 sat jul 18 01​:37​:18 cest 1998 i686 unknown '   hint=recommended\, useposix=true\, d_sigaction=define   usethreads=define useperlio=undef d_sfio=undef   Compiler​:   cc='cc'\, optimize='-O2'\, gccversion=2.7.2.3   cppflags='-D_REENTRANT -Dbool=char -DHAS_BOOL -I/usr/local/include'   ccflags ='-D_REENTRANT -Dbool=char -DHAS_BOOL -I/usr/local/include'   stdchar='char'\, d_stdstdio=define\, usevfork=false   intsize=4\, longsize=4\, ptrsize=4\, doublesize=8   d_longlong=define\, longlongsize=8\, d_longdbl=define\, longdblsize=12   alignbytes=4\, usemymalloc=n\, prototype=define   Linker and Libraries​:   ld='cc'\, ldflags =' -L/usr/local/lib'   libpth=/usr/local/lib /lib /usr/lib   libs=-lnsl -lndbm -lgdbm -ldb -ldl -lm -lpthread -lc -lposix -lcrypt   libc=\, so=so\, useshrplib=false\, libperl=libperl.a   Dynamic Linking​:   dlsrc=dl_dlopen.xs\, dlext=so\, d_dlsymun=undef\, ccdlflags='-rdynamic'   cccdlflags='-fpic'\, lddlflags='-shared -L/usr/local/lib'

Characteristics of this binary (from libperl)​:   Built under linux   Compiled at Aug 9 1998 10​:17​:03   @​INC​:   /usr/lib/perl5/5.00502/i386-linux-thread   /usr/lib/perl5/5.00502   /usr/lib/perl5/site_perl/5.005/i386-linux-thread   /usr/lib/perl5/site_perl/5.005   . [summer@​emu summer]$

-- Cheers John Summerfield http​://os2.ami.com.au/os2/ for OS/2 support. Configuration\, networking\, combined IBM ftpsites index.

[p5pRT]

From @tamias

On Wed\, Dec 15\, 1999 at 09​:09​:45PM +0800\, John Summerfield wrote​:

open(STDOUT\, "| cat >foo.out") || die "Can't redirect stdout"; # JCS open(STDERR\, ">&STDOUT") || die "Can't dup stdout";

close(STDOUT); close(STDERR);

This hangs (waits indefinitely)\, apparently at one or other of the closes (lines 14/15). I presume cat's not getting EOF.

I see the same behavior here in 5.005_03. Stepping through with the debugger\, Perl hangs on close(STDOUT). Switching the order of the close() statements\, so that STDERR is closed before STDOUT\, avoids the problem.

Ronald

[p5pRT]

From [Unknown Contact. See original ticket]

Ronald J Kimball \rjk@​linguist\.dartmouth\.edu wrote

I see the same behavior here in 5.005_03. Stepping through with the debugger\, Perl hangs on close(STDOUT). Switching the order of the close() statements\, so that STDERR is closed before STDOUT\, avoids the problem.

Yes - that's because Perl doesn't "know" that STDERR is thr result of a pipe open. So it doesn't wait in close(STDERR);.

The prpoblem is a more general one\, and will hit if the file descriptor is copied in any way. For example\, this code digs you into a similar hole (and I assume perl_clone would be an equally efficient spade)​:

% perl -w open FH\, "| cat >temp.foo" or die "open​: $!\n"; my $pid = fork or sleep 1000000\, exit; print "Forked $pid\n"; close FH; print "Closed\n"; __END__ Forked 5516

  ... then silence.

And of course\, there's other twists if you close the filehandle in the child from the fork\, as it'll try to wait() for a PID which isn't it's child.

Seems to me like a fundamental restriction in the way piped opens work\, which should be documented as "Don't do that\, then."

Mike Guy

[p5pRT]

From [Unknown Contact. See original ticket]

Ronald J Kimball \rjk@​linguist\.dartmouth\.edu wrote

I see the same behavior here in 5.005_03. Stepping through with the debugger\, Perl hangs on close(STDOUT). Switching the order of the close() statements\, so that STDERR is closed before STDOUT\, avoids the problem.

Yes - that's because Perl doesn't "know" that STDERR is thr result of a pipe open. So it doesn't wait in close(STDERR);.

The prpoblem is a more general one\, and will hit if the file descriptor is copied in any way. For example\, this code digs you into a similar hole (and I assume perl_clone would be an equally efficient spade)​:

% perl -w open FH\, "| cat >temp.foo" or die "open​: $!\n"; my $pid = fork or sleep 1000000\, exit; print "Forked $pid\n"; close FH; print "Closed\n"; __END__ Forked 5516

... then silence.

For more than a million seconds? Eleven and a half days? I change it to a more reasonable interval and it ran just fine. [summer@​emu summer]$ testperl Forked 9067 Closed [summer@​emu summer]$

And of course\, there's other twists if you close the filehandle in the child from the fork\, as it'll try to wait() for a PID which isn't it's child.

Seems to me like a fundamental restriction in the way piped opens work\, which should be documented as "Don't do that\, then."

That would break a lot of documentation (Camel book for one)\, and it seems to me\, is very unperl.

Lots of people are going to rely on existing printed material for years to come\, and there's a good chance this one will slip past the writers of new documentation as well.

If there's way\, pls find it.

-- Cheers John Summerfield http​://os2.ami.com.au/os2/ for OS/2 support. Configuration\, networking\, combined IBM ftpsites index.

[p5pRT]

From @gsar

On Thu\, 16 Dec 1999 18​:25​:15 GMT\, "M.J.T. Guy" wrote​:

Ronald J Kimball \rjk@​linguist\.dartmouth\.edu wrote

I see the same behavior here in 5.005_03. Stepping through with the debugger\, Perl hangs on close(STDOUT). Switching the order of the close() statements\, so that STDERR is closed before STDOUT\, avoids the problem.

Yes - that's because Perl doesn't "know" that STDERR is thr result of a pipe open. So it doesn't wait in close(STDERR);.

I don't think waiting in close(STDERR) is the problem. It's the close(STOUT) that hangs when it waits. The subprocess won't get EOF until all the dup-ed handles are closed too\, so the parent waits indefinitely for the child to quit\, and the child waits indefinitely for the parent to close all the handles\, which it never will\, because it is waiting for the child to quit. Deadlock ensues.

The prpoblem is a more general one\, and will hit if the file descriptor is copied in any way. For example\, this code digs you into a similar hole (and I assume perl_clone would be an equally efficient spade)​:

% perl -w open FH\, "| cat >temp.foo" or die "open​: $!\n"; my $pid = fork or sleep 1000000\, exit; print "Forked $pid\n"; close FH; print "Closed\n"; __END__ Forked 5516

... then silence.

As John noticed correctly\, that doesn't hang; it merely sleeps a great long while\, because you asked for it. Here's a better example​:

  open(A\, "| cat >foo.out") or die;   open(B\, ">&A") or die;   print A "foo\n";

  close(A); # hangs...   close(B); # ...unless you flip this line with the one above

The problem really originates in the old fclose/pclose duplicity. This has the same issue​:

  #include \<stdio.h>   int   main(int argc\, char **argv)   {   FILE *f = popen("cat > foo.out"\, "w");   if (f) {   FILE *d = fdopen(dup(fileno(f))\, "w");   fprintf(f\, "foo\n");   pclose(f); /* hangs... */   fclose(d); /* ...unless you flip this line with the one above */   }   }

Chalk up another reason to rewrite stdio.

Sarathy gsar@​ActiveState.com