Best way to debug workers in MCE::Map

[srchulo]

Hi, I'm wondering if there is a good way to debug what is going on in workers, or their ongoing progress. I have an issue where I'm running a job and some of the workers never finish. Here is my setup:

MCE::Map->init(
    max_workers => 4,
    user_begin => sub {
        $log->info("Worker ## " . MCE->wid .  " started");
    },
    user_end => sub {
        $log->info("Worker ## " . MCE->wid . " completed");
    },
);

mce_map {
    # do stuff
} @items;

And I will get this output:

2020-10-30T06:19:54.646-05:00 | [2020-10-30 11:19:54.64649] [23] [info] Worker ## 2 started
-- | --
  | 2020-10-30T06:19:54.646-05:00 | [2020-10-30 11:19:54.64649] [22] [info] Worker ## 1 started
  | 2020-10-30T06:19:54.648-05:00 | [2020-10-30 11:19:54.64770] [24] [info] Worker ## 3 started
  | 2020-10-30T06:19:54.661-05:00 | [2020-10-30 11:19:54.66083] [25] [info] Worker ## 4 started
  | 2020-10-30T09:30:14.884-05:00 | [2020-10-30 14:30:14.86524] [23] [info] Worker ## 2 completed
  | 2020-10-30T09:30:46.107-05:00 | [2020-10-30 14:30:46.10763] [22] [info] Worker ## 1 completed

And workers 3/4 will never finish. This is using mce_map. Is there a more granular way than user_begin and user_end to see what is occuring within workers? Or even a way to print out an aggregate progress update, like 300/1000 processed?

Thank you!

[marioroy]

Hi, Adam

I cannot tell why workers are not completing in your case or whether workers crashed :( Or if the log buffer is not being flushed.

Is there a more granular way...? Yes, one may specify chunk_size => 1 to disable chunking.

MCE::Map->init(
    max_workers => 4,
    chunk_size => 1,
    user_begin => sub {
        $log->info("Worker ## " . MCE->wid .  " started");
    },
    user_end => sub {
        $log->info("Worker ## " . MCE->wid . " completed");
    },
);

mce_map {
    $log->info("Worker ## " . MCE->wid . " processing chunk ## " . MCE->chunk_id);
    # do stuff
} @items;

For progress updates, one may specify the progress option.

Hoping that all is well.

Regards!

[srchulo]

@marioroy Thank you for the help! That helped me track down the issue :)