Thread safety issue

[Zer0-Tolerance]

Hi, I'm getting a lot of crashes when I use P5 code with promise, it seems that when you reach a certain amount of threads a kind of race condition is triggered. The following code reproduces the problem easily:

use Inline::Perl5;
my $perl5 = Inline::Perl5.new;
$perl5.run(q:to/PERL5/);
use 5.010;
use Crypt::Digest::MD5 qw(md5_hex);
sub p5md5 {
    say md5_hex('data string');
}
PERL5
my @promise;
@promise.push(start {
  $perl5.call('p5md5');
}) for 1..@*ARGS[0];
await @promise;

• raku p5.raku 1 => runs fine
• raku p5.raku 10 => runs fine
• raku p5.raku 100 => crashes

I'm using the following Perl5 :

Summary of my perl5 (revision 5 version 24 subversion 4) 
Compile-time options: HAS_TIMES MULTIPLICITY PERLIO_LAYERS
                        PERL_COPY_ON_WRITE PERL_DONT_CREATE_GVSV
                        PERL_HASH_FUNC_ONE_AT_A_TIME_HARD
                        PERL_IMPLICIT_CONTEXT PERL_MALLOC_WRAP
                        PERL_PRESERVE_IVUV PERL_USE_SAFE_PUTENV
                        USE_64_BIT_ALL USE_64_BIT_INT USE_ITHREADS
                        USE_LARGE_FILES USE_LOCALE USE_LOCALE_COLLATE
                        USE_LOCALE_CTYPE USE_LOCALE_NUMERIC USE_LOCALE_TIME
                        USE_PERLIO USE_PERL_ATOF USE_REENTRANT_API

Any idea how to fix this ?

[Zer0-Tolerance]

any update ?

[Zer0-Tolerance]

Any plan to fix it ?

[raiph]

Hi @Zer0-Tolerance

I'm not nine (IP5 author). But I saw that you'd not gotten a response and decided to see if I can help. Some thoughts:

1. "Any plan to fix it?" The following is a plan I suggest for you to consider following to try fix it.

2. You haven't said so, but I'm guessing you've used IP5 for non threading scenarios with good results. In case not, I'll note that the typical thing to do when using Perl modules is to write use Such::And::Such::Module:from<Perl5>; in Raku code and then use the module's features as if it were a Raku module (rather than writing a Perl use statement as you've done, and then write Perl code using the Perl module). But, like I said, I'm guessing/presuming you just chose something simple to emulate writing custom Perl code because you encountered a problem in that particular scenario, where you are combining both threading and writing custom Perl code, rather than just directly using a Perl module in a threading scenario, after having stuff working OK in non-threading scenarios. Right?

3. As far as I can tell your code is only creating one IP5 interpreter. I haven't tried to understand or use IP5 in a threading context, but that seems suspect. The relevant verbiage in the README seems to be "If you want to use more than one IP5 interpreter safely, for instance from within Raku threads...", and that suggests to me that you need an Inline::Perl5.new for each thread.

4. I see you've got a use 5.010; in your Perl code. I get that the perl is itself (presumably) 5.24..., and that the current threading model has been working since around 5.8 or so, but I wonder why you've not written use 5.24;?

5. You've not provided system info. I'd say that, at minimum, you should provide (links to gists of) the full output of raku -v for Rakudo, and of perl -V for perl, and of make test and make install for IP5.

6. You haven't confirmed that you fully followed the instructions provided in [the README's BUILDING section](in https://github.com/niner/Inline-Perl5#building). One of the reasons full output is likely necessary for the IP5 make test and make install, and of perl -v, is that it will confirm (or not) whether the perl that Rakudo/IP5 are using appears to have been correctly compiled.

7. Presuming you add info so that there's enough that someone other than me might be able to usefully respond to your comment, it will likely need to be nine if things are really to move forward, and he focuses on the system he uses, which means you would have to do what it takes if your system is sufficiently different from his. You hadn't even said what OS you're using so it's impossible for me to guess at how that might pan out, but suffice to say, if you are respectful of his time, I'm sure he will be of yours, and I'd imagine you'll get at least some follow up this year.

8. If you have hopes to reduce the turn around time this year I suggest three things. First confirm, perhaps by asking someone in the perl world who should know, that it's possible, at least in principle, given the perl interpreter's current threading behavior, to be running in 100 green threads. Ideally reproduce exactly that by running that scenario using any other system (not involving Raku/Rakudo). If you can provide sufficient info that someone can reproduce that, then that'll be a huge help.

9. If you get to the point where there's system info, and something close to proof that 100 green threads running Perl should work (either a perl expert saying so, or a reproduction of it working with some non-Raku system), but there's no progress in this thread, then it will make sense to post it as a StackOverflow question. (But not before; it needs to be possible to both reproduce your results and know it should work, or the response you get on SO will be no different than it is here.) By all means include a link to this issue in the SO if the necessary details are in this issue.

10. Good luck! :)

[niner]

@Zer0-Tolerance sorry for the radio silence. I have let myself get sucked into $day-job to an unhealthy degree and only now realized that that was not a good decision.

Your problem is that Perl 5 is just not thread-safe. Indeed that was one of the reasons for starting the whole Perl 6 and now Raku effort. We cannot magically fix that. However, there are some things that indeed are possible. Perl 5 does support running multiple independent interpreters in the same process and Inline::Perl5 makes use of that in its somewhat experimental and apparently totally undocumented threading-support.

This works as desired:

use Inline::Perl5 "thread-safe";
use Crypt::Digest::MD5:from<Perl5>;
my @promises = (1..@*ARGS[0]).map: {
    start {
        say Crypt::Digest::MD5.md5_hex;
    }
};
await @promises

What happens here is that in "thread-safe" mode Inline::Perl5 will create on Perl interpreter on each thread that you try to run Perl code on. These interpreters operate independently of each other and must not share any data. So all Perl objects must only be accessed within the same thread that they were created on. This sounds quite restrictive but it's still good enough for many situations like for example using Perl modules in worker threads (think typical web application) and indeed that's what I wrote this support for: to make Perl available to Cro applications. If you do want to share data between multiple threads, this data must be converted to appropriate Raku types first.

One more caveat is that this support currently only exists for modules imported with :from<Perl5> like in my example and only for methods on these modules and their objects. That's why I call md5_hex as if it was a method instead of importing it as a function.