andre-merzky
commented
3 years ago Is the Worker forked from the Master
No, as in the general case workers run on different nodes that masters.
In any case, coding against ZMQ queue items or events will likely be preferable in many cases, so this has been added to the RP raptor feature requests.
Accepted as a feature request. We are currently able to provide ZMQ Queue or PubSub channels. This needs to be exposed as config option on the application side. ETA: Mai 2021
eirrgang
All Worker Tasks received as Requests from the Master/scheduler are run in Interpreters forked from the main Worker interpreter. (Is the Worker forked from the Master?) This means that Tasks cannot generally pass objects by reference between each other or with the framework.
(There may be exceptions in which the task may be able to receive objects by reference, depending on whether the interpreter is simply forked through the multiprocessing module, versus launching via POpen or mpiexec.)
Moreover, we cannot yet pin tasks to Worker processes. (See issue #105) The Linux filesystem cache is very fast and effective these days, but we do not currently have a way to explicitly manipulate data locality.
Currently, we can communicate objects
For a third generic alternative, we can grant Workers and their tasks access to a Zero-MQ channel. ZMQ is multi-language and, I believe, has high-performance C++ header library functions to efficiently map message buffers to/from compiled data structures, the way gRPC does. (should check on this). In any case, coding against ZMQ queue items or events will likely be preferable in many cases, so this has been added to the RP raptor feature requests.