search

tomerfiliba-org / rpyc

RPyC (Remote Python Call) - A transparent and symmetric RPC library for python
http://rpyc.readthedocs.org
Other
1.57k stars 243 forks source link

Improve message routing and make ZeroDeploy "killer" #373

Open traverseda opened 4 years ago

traverseda commented 4 years ago

https://mitogen.networkgenomics.com/

Mitogen is "a Python library for writing distributed self-replicating programs". It has a lot of very nice primitives for managing distributed programs, but it's definitely not an RPC system.

Right now rpyc is in this weird space where it's not really good for anything, there's no killer feature. Perhaps easily running your code across a cluster of ssh machines could be that killer feature? Mitogen plus a well-documented multiprocessing-like tool could work very nicely.

traverseda commented 4 years ago

I think the first step would be being able to instantiate an rpyc connection across an arbitrary mitogen-channel. I've tried to do similar things across stdin/stdout pairs and have not been successful, I'm a bit unclear on the "proper" way to create a new rpyc connection.

Python's multiprocessing relies forks copy-on-write semantics, which makes it challenging to provide an equivalent API. We don't really want every object in the global namespace to be pointing at a netref, some libraries obviously need to be imported locally... Even if we're using mitogen's handy import-forwarder to not need them installed locally.

comrumino commented 4 years ago

Right now rpyc is in this weird space where it's not really good for anything, there's no killer feature.

Suppose we measured "good" by depedents for a given RPC repository. angr and pupy are dependents of RPyC, so it's good for something :1st_place_medal:

Jokes aside, there are things that could be done better by RPyC and there is no obvious way forward for adding mitogen as a backed. To achieve what I believe you are looking to do here, I would start with zerodeploy. Maybe we should consider improving RPyC's ability to route messages by host-id or node-id? Improving the usability of existing utils and expanding their features would make more sense to me.

comrumino commented 4 years ago

Supporting the feature set for both Linux and Windows would be less of a uphill battle by working with what we got.

traverseda commented 4 years ago

Somewhat related to #340 , if we can get the transport working with stdin/stdout, getting it working with mitogen channels should be relatively easy.

cebaa commented 2 years ago

if we can get the transport working with stdin/stdout

On top of making it easier to implement mitogen & other integration, if I'm not mistaken the current zerodeploy client / server model doesn't have authentication, so other users on the same machine can access it freely. Using stdin / stdout would make it better from security standpoint as well - and likely make it simpler as well by removing the socket code.