How to handle MCE::Hobo running concurrently with MCE::Core loop?

[akotlar]

I am having some trouble having 2 separate loops; and the 2 conflict. The code I'm referencing is in this PR: https://github.com/bystrogenomics/bystro/pull/545/files#diff-c50affcfb987a9ee780316c4727890bcb8ff8bab08a9214227cab93729399283R573

• We set up an MCE::Hobo process, that periodically issues a heartbeat for a job reserved in a queue
• After that, my main function (annotate) is called, where we initiate an MCE::Loop, https://github.com/bystrogenomics/bystro/blob/master/perl/lib/Seq.pm#L162
• At the end of the main function, we finish the loop, MCE::Finish

TL;DR:

# Seq.pm
use MCE::Loop
use Seq

sub annotate {
   MCE::Loop::init(...)

   # do stuff

   MCE::Loop::finish()
   return $data;
}

#Server
use MCE::Shared;
use MCE::Hobo;

use Seq;

main {
my $done = MCE::Shared->scalar()
my $heartbeatClient = MCE::Hobo->create(
        sub { 
        if($done->get()) {
           return
        }
      }
)

my $res = Seq::annotate()

say "I'm done!";

$done->set(1)
$heartbeatClient->join()

Depending on import order, we will never get to "I'm done" (only if MCE::Hobo is imported after Seq). However, even then, if the server picks up a 2nd job, it will never get past attempting to create the MCE::Loop

[marioroy]

I will take a look later today and report back.

[akotlar]

Thank you!

[akotlar]

""" Hmm, you provided invalid Perl code i.e. missing semi-colons. The main { ... without the closing brace. I tend to ignore stuff like this. """

I shared the actual code with you, and enough to go on to reproduce. I can try to give you a full working example later, but you have enough to go on to figure out the issue I suspect.

Thanks, Alex

[akotlar]

This is not resolved.

[marioroy]

Your request is unusual with Perl code written like that. Please, never do that to me or anybody for that matter. :) Well, you know. Life is short and I too have limited time. The following is something I tried. Be sure to initialize the shared variable.

Seq.pm

use v5.010;
use strict;
use warnings;

package Seq;

use MCE::Loop;
use MCE::Candy;

sub annotate {
    my @res;
    MCE::Loop::init(
        max_workers => 8,
        chunk_size  => 1,
        posix_exit  => 1,
        gather      => MCE::Candy::out_iter_array(\@res),
    );

    my @input = ('a'..'z');

    mce_loop {
        my ($mce, $chunk_ref, $chunk_id) = @_;
        my $item = $chunk_ref->[0];
        MCE->say("$$: $item");
        MCE->gather($chunk_id, uc($item));
    } \@input;

    MCE::Loop::finish();
    return \@res;
}

1;

server.pl

use v5.010;
use strict;
use warnings;

use MCE::Shared;
use MCE::Hobo;
use Seq;

sub main {
    my $done = MCE::Shared->scalar(0);
    my $heartbeatClient = MCE::Hobo->create( sub {
        while (1) {
            return if $done->get();
            sleep 1;
        }
    });

    my $res = Seq::annotate();

    say "I'm done!";
    $done->set(1);
    $heartbeatClient->join();

    say "@{$res}";
}

main();

Output:

3919: b
3922: a
3916: c
3918: e
3917: d
3920: g
3921: h
3915: f
3919: i
3916: k
3922: j
3918: m
3917: l
3921: n
3922: q
3920: o
3915: p
3916: r
3917: t
3921: v
3919: s
3922: y
3918: u
3920: x
3916: z
3915: w
I'm done!
A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

[marioroy]

Reading through your project and Python code, have you considered Celery - a Distributed Task Queue?