Auto-proxy for nonserializable objects

[lneuhaus]

In our use-case, we employ Pyro4 to try to expose "as much as possible" of a remote object to a client. By that we mean the following

1. All methods, class and instance attributes, descriptors etc. of the remote objects should be available to the client - we want the client to experience the proxy object just as if it was a local object.
2. When a remote call evaluates to a "custom object", i.e. one whose class is defined in a certain list of python modules (our user-code), return a proxy to that object rather than trying to serialize the object (serialization usually fails anyways on our custom objects).

We achieve this so far by iterating through the entire attribute hierarchy of the "root" object and registering with the Pyro daemon any of its attributes that are "custom objects" before entering the server loop. Pyro's auto-proxy feature is of great help here. To also expose instance attributes, we added some extra logic to a wrapper around Pyro's Proxy class and extra methods remote_get(name_of_attribute) and remote_set(name, value) to get/set instance attributes from the server.

We hit an issue when an instance attribute is changed from one of trivial type (e.g. None) to a "custom object" at runtime, e.g. triggered by another remote function call. In this case, the new "custom object" isn't automatically registered with the Pyro daemon, so it's also not exposed. When a client requests it, pyro then tries to serialize it and fails.

A solution to this would be a "auto-proxy-when-not-serializable" feature: the server side should be able to register the new object with its daemon automatically when it discovers that it cannot serialize it. That would get the answer to the question "should this live on the server or client" right in more than 90% of the cases for us.

It seems to me that we're moving away from the intended use-case of Pyro. Still, I feel like what we want to achieve shouldn't be such a rare request in environments where security is not a concern. Let me know if you think I should rather try to solve this using a different / custom library. Another option would be for me to implement this addition in my own fork, but I first wanted to ask whether this is interesting for the main repo. Please let me know if anything is unclear.

[irmen]

Sorry: Pyro4 is feature-frozen, so any behavioral changes will not be added anymore. What about Pyro5 then? Its focus is to provide a secure and reasonably "nice" remoting library -- but you're right that what you're describing is moving away from that core principle. Pyro5 will not get a mechanism that will auto-expose unknown/new objects.

While your intended approach may work if security is not a concern, this is not the path Pyro is taking. Also I am unaware of any other libraries that do provide a complete distributed object model like you describe, for Python. I feel the best approach is to not depend on this and rather architect your system around well defined constrained interfaces.

Btw Pyro exposes instance attributes just fine as long as they are @properties. I'm not sure as to why you would require wrapper code for that.

[lneuhaus]

Thanks a lot for your quick answer!

Sorry: Pyro4 is feature-frozen, so any behavioral changes will not be added anymore. What about Pyro5 then? Its focus is to provide a secure and reasonably "nice" remoting library -- but you're right that what you're describing is moving away from that core principle. Pyro5 will not get a mechanism that will auto-expose unknown/new objects. While your intended approach may work if security is not a concern, this is not the path Pyro is taking.

Makes sense to not support Pyro4 any more. And Pyro5 is unfortunately not an option for us (we need pickle to serialize reasonably fast), and as you say, Pyro5 seems to be moving rather into the "secure RPC" direction than the one I'd need for my current project.

Also I am unaware of any other libraries that do provide a complete distributed object model like you describe, for Python. I feel the best approach is to not depend on this and rather architect your system around well defined constrained interfaces.

Thanks for the advice, as you say there must be a reason for not having such libraries around. I understand that well-defined interfaces are nice to have, but for development purposes I find it practical to have access to the full object from a client, so ideally I'd want both: something very open at first that you can restrict as you figure out what calls are actually needed. While slightly different, I suppose you ran such an experiment with Pyro Flame, which I see is discontinued in Pyro5 - probably because it encouraged wrong style? For the short term I might just go with a Pyro4 fork as this seems to be the path of least effort for me.

Btw Pyro exposes instance attributes just fine as long as they are @properties. I'm not sure as to why you would require wrapper code for that.

The issue is that the attributes in already existing classes are not always @properties (classes are allowed to be written in a Pyro-agnostic way in our framework, so it would be too much to ask to replace all instance attributes assigned in the __init__ method by @properties).

And don't get me wrong, Pyro4 has served us very well until now! Thanks for the library.

[irmen]

Thanks for your understanding and explanations!

Forking Pyro4 should be safe -- no big changes will come around anymore. What you need is probably too intrusive to monkey-patch on the library as is