Child processes disappear

[adkinsbl]

I'm experiencing an issue in Windows when using the package. The problem doesn't seem to be consistent, the child processes disappear intermittently and randomly. Also, the problem doesn't seem to occur with a smaller set ~60 compared to ~4k.

$pm->run_on_finish(sub { my ($pid,$exit_code,$ident,$exit_signal,$coredump,$ds) = @; $screen_count++; if ($exit_code == 1) { push @matches, $ds; } }); DATA_LOOP: foreach my $ctx_file (@{$ctx_files}) { $pm->start($ctx_file) and next DATA_LOOP; my $ctx = &slurp_ctx($ctx_file); my $str = quotemeta("string_to_find"); if ($$ctx =~ m/$str/i) { $pm->finish(1,[$ctx,$ctx_file]); } else { $pm->finish(0); } } $pm->wait_all_children;

The error looks like:

child process '-9748' disappeared. A call to waitpid outside of Parallel::ForkManager might have reaped it.

[yanick]

The error is more of a warning. It means that 9748 was a child process spawned by P::FM, but when it tried to reap it, it was done. Check your codebase for any other call to waitpid. Also, if you can it might be interesting to try your code on Unix/Linux to see if the same problem appears there.

[adkinsbl]

Unfortunately I can't run Unix on my work machine. I see no other calls to waitpid outside of the P::FM or any suspect modules, File::Find and Path::Tiny. My concern is that there may be a match that doesn't get reported back because it never gets to run_on_finish,

[adkinsbl]

Would a clip of the whole code be useful?

[yanick]

If you can reduce the program to a small, minimalistic example that reproduce the problem, it'd be fantastic.

[adkinsbl]

script.txt I updated this, removed the library and pared it down for what's actually used

[adkinsbl]

I've also noticed that I tend to get the warning when the number of files >> MAX_PROCESSES. When I've processed ~100 files, I haven't gotten a warning. The warning always happens when it's > 1000 files

[yanick]

Sorry, when I said "a small, minimalistic example that reproduce the problem", I meant one that I can run. Your example has, for example, I don't have the Cimplicity module locally.

[adkinsbl]

I updated the previous post to remove the module and show what I was using.

[stu0x43]

I suspect this is a bug in the windows perl implementation of forks (version <=2.4) - The perl docs do say perlfork in windows has issues. I've found that you cant rely on the parallel forkmanager to run its run_on_finish routines within a windows environment. As long as you dont care about the results from your children and can put up with the failed to reap warnings the processes do seem to complete gracefully. My guess is the windows O/S is tidying the threads (which is how perlfork is implemented under windows) before the Forkmanager has a chance to deal with them.

[marioroy]

Today, I have seen the "child process disappeared" warning four times on the Windows Platform with the following example. The test environment is Window 7 via Parallels Desktop on macOS, Strawberry Perl 5.26.1, and Parallel::ForkManager 1.20. To reproduce the warning, one may need to run the example 3 times consecutively in a row.

use strict;
use warnings;

use Parallel::ForkManager;
use Time::HiRes 'time';

my $start = time;

my $pm = Parallel::ForkManager->new(20);
$pm->set_waitpid_blocking_sleep(0);

$pm->run_on_finish( sub {
    my ($pid, $exit_code, $ident, $exit_signal, $core_dumped, $resp) = @_;
  # print "child $pid completed: $ident => ", $resp->[0], "\n";
});

DATA_LOOP:
foreach my $data (1..4000) {
    # forks and returns the pid for the child
    my $pid = $pm->start($data) and next DATA_LOOP;
    my $ret = [ $data * 2 ];

    $pm->finish(0, $ret);
}

$pm->wait_all_children;

printf STDERR "duration: %0.03f seconds\n", time - $start;