lf-lang / lingua-franca

Intuitive concurrent programming in any language
https://www.lf-lang.org
Other
240 stars 63 forks source link

Custom Serialization in Python Target #2375

Closed Depetrol closed 3 months ago

Depetrol commented 4 months ago

This PR enables a new feature for using arbitrary python package names that follows a specific API as the connection serializer in Python target federated execution.

Syntax:

<source_port_reference> -> <destination_port_reference> after <time_value> serializer "<serializer_name>"

This feature is backwards compatible. If <serializer_name> is any of [NATIVE, ROS2, PROTO], it will use these serialization methods instead of finding a python package.

The custom Python package would have to export a Serializer class that has the following API:

class Serializer():
    def serialize(self, obj)->bytes:
        return <bytes_object>
    def deserialize(self, message:bytes):
        return <deserialized_object>

The Serializer class will be initialized at the first send and first receive on both ends of the connection. It is persisted in each networksender and networkreceiver as a state variable. Python objects are serialized with serialize(), sent over the socket and deserialized with deserialize().

This feature allows for user-defined custom serialization for any strategy. For example, the user could define a serialization method that enables fast inter-process communication with shared memory. Potentially ROS2 and PROTO serialization could be implemented with custom python packages too. It also supports different serialization methods for different connections.

Depetrol commented 3 months ago

CI is failing because of #2376

cmnrd commented 3 months ago

I still cannot say that I like installing something into the user's system as a side effect of the preamble in a test file. I also think that requiring the serializer to be a standalone Python package that is installed separately is not a practical design from a usability perspective. However, I have made those points before and will not continue arguing for them. I cannot give this my approval, but if someone else does, I will not block it.

Depetrol commented 3 months ago

I agree that installing something into the user's system as a side effect of the preamble in a test file is not a perfect solution, but it doesn't require developers to install it manually in order to run the test, and makes the test self-contained.

However I would argue that installing a standalone Python package is more user friendly compared with using relative imports. There are several reasons:

lhstrh commented 3 months ago

@edwardalee sorry I re-requested a review but I hadn't noticed that you just provided feedback already