Can't close output pipe fh (|-) after mce_loop_f block

[akotlar]

Test on MCE-1.805 , test program attached. This program fails on both Mac OS X 10.10.5 and Amazon's latest AMI. Pseudo code below, see the attached program for a more comprehensive example.

Goal is to close a file handle, which was opened using "|-", aka pipe to some command. Can only close the out file handle before the mce_loop_f block.

This is a major issue for me, because I need to stream to a separate process the results generated in the mce_loop_f, and need to be able to send an "EOF", via close.

test_mce_loop.pl.zip

sub main {

  open(my $outFh, "|-", "cat -");

  # Can also use "-|"
  open(my $inFh, "<", ARGV[0]);

  print $outFh "Hello world";
  # Closing $outFh before the block would work fine
  mce_loop_f {
   # Do stuff
  } $inFh;

  say "before close";

  close $outFh;

  # Never reached
  say "after close";
}

[test_mce_loop.pl.zip](https://github.com/marioroy/mce-perl/files/505349/test_mce_loop.pl.zip)

[marioroy]

Alex, thank you for the test case. Workers from MCE Models (e.g. Loop, Flow, Step) persist after running. As a workaround, shutting down MCE resolves the issue.

print $statsFh "HELLOO\n";

mce_loop_f {
  ...
} $fh;

MCE::Loop::finish;    #  shutdown MCE workers

say "closing statsfh";
close $statsFh;
say "closed";

[marioroy]

For extra performance, pass $ARGV[0] to mce_loop_f for direct reads by MCE workers. This involves lesser IPC when the file path is known.

mce_loop_f {
  ...
} $ARGV[0];

[marioroy]

This issue is not a bug with MCE. The same thing is seen with fork. Here, waitpid is called after close to simulate with a simple fork. The MCE::Loop::finish line is necessary for your test case above.

use 5.10.0;
use strict;
use warnings;

$| = 1;

open(my $statsFh, "|-", "cat -");

print $statsFh "HELLO\n";

my $pid = fork;

if (!defined $pid) {
   die "cannot fork: $!\n";
}
elsif ($pid == 0) {
   print "from child before sleep\n";
   sleep 3;
   print "from child after sleep\n";
   exit;
}

say "closing statsfh";
close $statsFh;
say "closed";

waitpid $pid, 0;

The main script blocks on close $statsFh until the forked process exits.

closing statsfh
from child before sleep
HELLO
from child after sleep
closed

[akotlar]

Mario, thanks so much. Yes, MCE::Loop::finish absolutely fixes this, except in the case that MCE::Shared is also used. I have attached a test case, and pseudo code below.

Interestingly, MCE::Shared->stop() before the close call fixes that. Do you have insight into why this is?

 use MCE;
 use MCE::Loop;
 use MCE::Shared;

 my $chunkSize;
 # We need to know the chunk size, and only way to do that 
  # Is to get it from within one worker, unless we use MCE::Core interface
  my $m1 = MCE::Mutex->new;
  tie $chunkSize, 'MCE::Shared', 0;

sub main {

  open(my $outFh, "|-", "cat -");

  # Can also use "-|"
  open(my $inFh, "<", ARGV[0]);

  print $outFh "Hello world";
  # Closing $outFh before the block would work fine
  mce_loop_f {
   # Do stuff
  } $inFh;

  say "before close";

  close $outFh;

  # Never reached, UNLESS MCE::Shared->stop(); called before
  say "after close";
}

test_mce_loop_with_shared.zip

[marioroy]

Running MCE starts the shared-manager process (fork on Unix) if not already started. As with MCE, stopping is required before closing the piped handle. Another way is starting the shared-manager process early in the script. Note that the IO::FDPass module is recommended if later constructing MCE::Shared::{ Condvar, Handle, Queue } objects.

use MCE::Loop;
use MCE::Shared;

MCE::Shared->start;

open(my $fh, "-|", "cat $ARGV[0]");
...

MCE::Loop->finish;
close $statsFh;

[akotlar]

@marioroy With regard to passing the file path directly: I assume this only works for non-compressed files?

Thanks very much for your help, and for writing this package. It's been a tremendous help in my current work.

[marioroy]

Yes. You're welcome @akotlar :)