exodist / Child

(perl) Object oriented simple interface to fork()
9 stars 2 forks source link

Child IPC leaks pipes? #10

Closed SpikesDivZero closed 9 years ago

SpikesDivZero commented 10 years ago

With a max files ulimit of 256, after roughly 110 children are executed, pipes start failing to be created:

Starting job 129
Got data back from child: my pid is 26033
print() on unopened filehandle $aout at /usr/share/perl5/Child/Link/IPC.pm line 61.
Got data back from child: my pid is 26034
Starting job 130
readline() on unopened filehandle $ain at /usr/share/perl5/Child/Link/IPC.pm line 50.
Use of uninitialized value $data in concatenation (.) or string at runner.pl line 27.
Got data back from child: 

And an strace will start showing output like:

pipe(0x3b56c69dc30)                     = -1 EMFILE (Too many open files)
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x2d1ac5d79d0) = 25675

To use the below proof of concept, you'll need to launch runner.sh on any Linux system. POC likely also works on OS X, but not tested there.

runner.sh:

#!/bin/bash
ulimit -n 256
exec perl runner.pl

runner.pl:

#!/usr/bin/env perl
use strict;
use warnings;
use Parallel::Runner;
use IO::Handle;

STDOUT->autoflush(1);
STDERR->autoflush(1);

my $Runner = Parallel::Runner->new(
    50,
    data_callback => \&runner_data_callback,
);
for my $i (1..1000) {
    print "Starting job $i\n";
    $Runner->run(sub { return "my pid is $$" });
}
$Runner->finish;

sub runner_data_callback {
    my ($data) = @_;
    print "Got data back from child: $data\n";
}
SpikesDivZero commented 10 years ago

A bit more information. Reproduced in a virgin perlbrew environment

perlbrew install stable
perlbrew install-cpanm
cpanm --local-lib=~/perl5 local::lib && eval $(perl -I ~/perl5/lib/perl5/ -Mlocal::lib)
cpanm Parallel::Runner Child Devel::Cycle

Perl-core version 5.20.0 Parallel::Runner version 0.013 Child version 0.010

Devel::Cycle doesn't seem to find the cycle in either the main $Runner or the $Child returned from run().

SpikesDivZero commented 9 years ago

Confirmed that it looks like 0.011 fixes this bug. Thanks, @exodist !