marioroy / mce-perl

Many-Core Engine for Perl
Other
45 stars 5 forks source link

MCE::new: (/tmp/<file_name>.2432.0) is not a directory or does not exist #7

Closed akotlar closed 7 years ago

akotlar commented 7 years ago

This error was just generated. The line in the debug message corresponds to the end of the file loop


mce_loop_f {

 ... 

} $fh;

Where $fh corresponds to a pipe.
marioroy commented 7 years ago

Hi Alex,

I tried, but cannot reproduce the error. Is there another script involved? The error indicates the MCE temp dir does not exist. Something removed the temp dir.

What is the version of MCE?

Regards,

Mario

marioroy commented 7 years ago

Another reason is when the process is killed, which triggers MCE cleanup. Something killed the process or removed the temp directory.

akotlar commented 7 years ago

Is it possible that during cleanup it removes temp directory not created in the parent process?

I had 4 long running programs, each of which had loaded the MCE library. 2 were killed a relaunched. 2 were not. I am wondering if killing those processes cleared the running processes' MCE temp dirs.

marioroy commented 7 years ago

The temp directory is created by MCE::Signal. The temp dir is cleaned up prior to script exiting or receiving a signal, which typically ends the scripts. I like to know how the four MCE sessions were spawned so that I can reproduce this.

marioroy commented 7 years ago

@akotlar, thank you. I will address this for the upcoming MCE 1.828 release.

akotlar commented 7 years ago

My pleasure, and thank you for addressing this.

Regarding how they were spawned: There is a long running process like so

use Package_That_Uses_MCE;
while(my $data = $self->long_blocking_operation()) {
  if ($data) {
    my $runner = Package_That_Uses_MCE->new();

    $runner->go();
 }
}

4 programs listening for these blocking operations are launched. Each one handles slightly different tasks, and all use MCE.

Please let me know if you need any additional information.

marioroy commented 7 years ago

Thanks @akotlar.

Knowing whether you're running on Unix or Windows is helpful. Also the version of MCE. I like to resolve this before releasing MCE 1.828.

So far, am having no luck in making subsequent MCE sessions fail after killing a running MCE session. Are all 4 programs launched from the same process? The reason I'm asking is that killing any child process ends up stopping all workers including the main process. Basically, how were the two MCE sessions killed while not impacting the other two?

Does ->new or ->go create a child or thread and run MCE inside it? Unfortunately, I lack the minimum info to reproduce the scenario regarding subsequent MCE sessions failing after killing a session.

Thank you for any info you can share on how MCE workers are spawned from inside the package: e.g. from inside fork or thread, or background tasks via system, or open.

Regards.

marioroy commented 7 years ago

MCE 1.828 was released today. The tmp_dir and sess_dir are made on demand in MCE::Signal and MCE respectively for backwards compatibility. This should address the issue.

akotlar commented 7 years ago

Thank you!

marioroy commented 7 years ago

You're welcome. Also, MCE::Shared 1.825 was released and forgot to mention that.