EelcoHoogendoorn
commented
9 years ago To expand a little on my observations:
zmq defines a bit of a transport protocol; its not just a raw buffer, but it already defines how messages are sent; as opposed to a gevent.StreamServer, for instance. Also, its 'sockets' arnt really sockets, but more like pipes. That it, it requires some thinking to cleanly separate the various components of the system into truly orthogonal, generalizable pieces. My gevent.StreamServer and pickle based communication work pretty much analogous to the zmq/JSON based system, but aside from the dispatcher, the components are not interchangeable. But rather than an ad-hoc way to make those pieces fit, it might be worthwhile to think about the more fundamental design refinements needed to arrive at such interchangeability of components.
lnoor
Not so much an issue as a general point of discussion:
I have been thinking about how to do inter-process communication in python elegantly for a while, and the (tiny)RPC model looks really elegant compared to the typical ad-hoc queue mess.
Relative to what tinyrpc offers, I have written a pickle-based communication layer (to more easily facilitate numpy arrays or other python-specific datastructures; we don't need the language-neutrality of JSON for IPC), and a gevent.stream.StreamServer based transport layer, as zmq is overkill for this I would argue; plus StreamServer routes messages in a one-socket-per-greenlet style natively, which has to be emulated in zmq.
So far I havnt quite been able to hammer this approach into the tinyRPC design; specifically, it appears the RPCServer base class is somewhat tied to the zmq transport layer, with its context,message tuples. As such, integrating such a different transport layer in a natural way would require some changes to the existing architecture it seems.
What I have put together works really well for me, and I cant imagine doing inter-process communication the old way anymore. As far as I can tell, no such out of the box solutions exist as of yet, so itd be a nice addition to tinyrpc I think. So my questions are: