actor model?

[yatesco]

Hi there, I can absolutely see the benefit of this crate for short lived jobs, and it looks really great for that. However, can I use this for long lived and stateful jobs, as the basis of an actor model?

For example, I have a bunch (~10) of different Processes which might "wake up" at certain times and push work onto a central queue. These processes are:

• stateful
• CPU intensive when they have work to do, which might itself be parallellellellisable
• live for the life of the application

As an example, one of these Processes might sweep through the system every couple of hours, noticing and flagging any stale data for review.

(A simple std approach would have a ThreadPool with each Process being a thread using channels to chat to each other and the central queue.)

I can see how each Process could use rayon, and I can see how I could use rayon in an event loop calling the individual Processes which then die until the next loop, but for long lived stateful jobs? I'm struggling to see if rayon is a good fit there?

Am I over thinking this (just for outer Processes)? Should I just use rayon and loop/yield within the processes? Should I just use a normal thread pool and move onto more interesting things?

Thanks!

P.S. Google gave me no hits for rayon and actor, which was a big sign I might be missing some foot gun here.

[cuviper]

Should I just use rayon and loop/yield within the processes?

I think the footgun there is that rayon yielding is nested, blocking that code on the stack by whatever gets called next. This is not like async/await that captures all the state to move off the stack. So, you would not want to be in a situation where one long-running loop yields to another, because only the inner one will get to make progress.

You could just create a dedicated thread for each actor, so they can loop with independent progress, and then have them farm their CPU intensive work to the shared rayon pool as needed. You could also write the actors in a true async/await framework, which may come with its own threadpool, or again lean on rayon for the CPU work. For example, here's a blog post on that topic, Actors with Tokio.

[yatesco]

Thanks @cuviper - that’s really helpful. I decided against tokio because I wanted something new :-) and there’s no IO, only CPU work going on.