An RPC wrapper library

[KOLANICH]

Project description

A tool in python which wraps another library.

Use cases:

• in distant future: interoperability between libraries written in different languages and using different interpreters;
• network transparency;
• stability - a crash of one interpreter won't cause the crash of another one
• using GPL, LGPL and other libs with viral licenses in the apps with another licenses. May be illegal: FSF GPL FAQ opinion states

It depends on how the main program invokes its plug-ins. If the main program uses fork and exec to invoke plug-ins, and they establish intimate communication by sharing complex data structures, or shipping complex data structures back and forth, that can make them one single combined program.

Using shared memory to communicate with complex data structures is pretty much equivalent to dynamic linking.

If the main program and the plugins are a single combined program then this means you must license the plug-in under the GPL or a GPL-compatible free software license and distribute it with source code in a GPL-compliant way.

though this opinion may be biased and I'm not a lawer.

Look & Feel

import tool_name
#the tool registers custom import hook

tool_name.remote.add("some_lib_which_can_cause_crash") # marking that lib as to be remotely used
import some_lib_which_can_cause_crash # import hooks work

some_lib_which_can_cause_crash.doSomething() # crashing the lib process won't crash the process using the lib

The tool contains of 2 parts sharing the same code: 1 a cli tool as a backend 2 a library as a frontdend

The library should add an import hook and have a set of modules imported through that lib. When a the hook is activated, it checks if the name asked to be imported is in the set, and if it is, it starts the backend and communicates to it via a shared memory page (in the case of local one) or via network. All the communication must be authenticated and optionally encrypted. The backend imports the library and allows the frontend which have started it to remotely execute code by sending special commands in a custom binary protocol. This protocol must have some comands to get values of variables.

The frontend creates a lazy object returning objects of special kind. When a user used any library functions which produce a value other than null (Note in the case of python) a pair of objects is created: the returned value on a backend and a proxy object on a frontend. When a method is called on a proxy object, the data is serialized and passed to backend. When a proxy object is used in a function on the frontend, the value is passed to frontend.

Relevant Technology

• python lazy_object_proxy
• python pickle (I hate it! But sometimes it may be useful, for example for passing variables back and forth)
• https://github.com/ray-project/ray
• google protobuf
• kaitai struct
• jupyter notebook

[ksamuel]

Wouldn't crossbar.io and the WAMP protocol already fit the bill ?

[KOLANICH]

@ksamuel, is the handwritten schema and manual registration required? I mean that the usage of the lib should be transparent as if it is used in the same process.

[ksamuel]

You don't need to write schema. You do need registration, but it's one line. And honestly, it's preferable: you really don't want all your functions to be exposed to the network.

The call is not transparent as:

• it requires await. Again, it's a good thing. You want this. Network calls are asynchronous by nature and blocking on this would limit your app to a few calls.

• calling is call('name_of_function') and not name_of_function(). You can, however, create a 3 lines wrapper if that bother you.

However, it's pretty great:

• no need to define the schema.
• Serialization is automatic and transparent.
• errors on one side are reported as native exception on the other side.
• JS code can call Python code. Python code can call PHP. PHP can call Java. Java can call C#.
• It uses WebSocket. So it works on any network that allows HTTP. Which means most networks firewall won't bother you and most switches and NAT are going to let you through.
• Did I say it uses WebSocket ? Because that also means you can use in the browser.
• It has a great security model.

[KOLANICH]

Thank you. This would simplify the task. For the local use (as a way to avoid viral licenses restrictions with the look & feel the same as if a real lib was used) I'd prefer it be as transparent as possible, so it is possible to tinker a lib around it hooking everything needed to make it transparent.