Warnings: Parallel calculations with PDL::Opt::NonLinear

[YuryPakhomov]

Hello all!

I use PDL::Opt::NonLinear module in my code. Works fine. The code can be paralleled with Parallel::ForkManager for calculation of some function. Works fine. When I try to combine them, many warnings appear:

    (in cleanup) INVALID PDL MAGICNO, got hex=0xf02100 (37593680)
 at /usr/local/share/perl5/Parallel/ForkManager/Child.pm line 26.
    eval {...} called at /usr/local/share/perl5/Parallel/ForkManager/Child.pm line 26
Warning: special data without datasv is not freed currently!! at test.pl line 0.
    eval {...} called at test.pl line 0
Warning: special data without datasv is not freed currently!! at test.pl line 0.
    eval {...} called at test.pl line 0
Warning: special data without datasv is not freed currently!! at test.pl line 0.
    eval {...} called at test.pl line 0
Warning: special data without datasv is not freed currently!! at test.pl line 0.
    eval {...} called at test.pl line 0
Warning: special data without datasv is not freed currently!! at test.pl line 0.
    eval {...} called at test.pl line 0

The simple code to reproduce this output is:

#! /usr/bin/perl
use PDL;
use PDL::NiceSlice;
use PDL::Opt::NonLinear;
use Parallel::ForkManager;

my $pm = Parallel::ForkManager->new(4);

my $x=pdl(1);
my $fx=pdl(0);
my $gx=zeroes(nelem($x));

# This single call works fine
#fg_func($fx,$gx,$x); exit;

# Optimization algorithm
                my $bounds =  zeroes(nelem($x),2);
                $bounds(0,0).=  pdl(0.0);
                $bounds(0,1).= pdl(10.0);

                my $tbounds = zeroes(nelem($x));
                $tbounds.=2;

                my $gtol = pdl(0.9);
                my $pgtol = pdl(0.1);
                my $factr = pdl(1e7);
                my $m = pdl(10);
                my $print = pdl(-1);
                my $maxit = pdl(long,15);
                my $info = pdl(long,1);
                my $iv = zeroes(long,44);
                my $v = zeroes(29);

                lbfgsb($fx, $gx, $x, $m, $bounds, $tbounds, $maxit, $factr, $pgtol, $gtol,
                $print, $info,$iv, $v,\&fg_func);

sub fg_func{
 my ($f, $g, $x) = @_;
 $f .= 0;
 $g .= pdl(0);

# Parallel calculations block
# If to comment this block then program works fine
 DATA_LOOP:
 for my $i (0 .. nelem($x)-1){
  my $ppid = $pm->start and next DATA_LOOP;
  $pm->finish($i); # Terminates the child process
 }
 $pm->wait_all_children;

 return 0;
}

Child.pm line 26 contains CORE::exit($x || 0) of subroutine finish:

sub finish {
  my ($s, $x, $r)=@_;

  $s->store($r);

    CORE::exit($x || 0);
}

The messages "INVALID PDL MAGICNO" and "Warning: special data without datasv is not freed currently!" are generated by Basic/Core/pdlapi.c

[yanick]

From the NVALID PDL MAGICNO in the error message, that seems to be coming from PDL code. And PDL seems to be its own (big) thing, so I don't think there is much we can do on our side. Sorry. :-(

[mohawk2]

@yanick As you can see on the PDL issue (https://github.com/PDLPorters/PDL-Opt-NonLinear/issues/6), this was a case of PDL working as designed, but not able to handle a process exiting because that DESTROYs objects that were set up as lightweight and supposed to be then dealt with on return from the Perl function. My recommendation was to switch from using ->finish from this module to POSIX::_exit.

Is there however some scope for this module to have something like a finish_nodestroy or similar to handle this sort of situation?

[yanick]

I think that you're in luck. Create a Child role like:

package Parallel::ForkManager::NoExitChild;

use strict;
use warnings;

use Carp;

use Moo::Role;

sub is_child  { 1 }
sub is_parent { 0 }

sub start {
    croak "Cannot start another process while you are in the child process";
}

sub finish {
  my ($s, $x, $r)=@_;

  $s->store($r);
}

1;

And use it as the child role in your ForkManager:

my $manager = Parallel::ForkManager->new( max_proc => 4, child_role => 'Parallel::ForkManager::NoExitChild' )

Although be careful! The children will not terminate as they hit finish, so you'll have to make sure they don't do anything they shouldn't do in the main process loop.