Interest in a Python module?

[david-drinn]

Python, when available, is my go-to language for quick, reliable, portable, maintainable script work. Is there interest from anyone else for a Python module of rpc_core?

I would suggest using the format strings laid out by the built-in struct module as a starting point for defining RPC packets in an rpc_core Python module. Figuring out how to expand that to the supported C++ std:: objects would be very interesting. An alternative route would be to do what protobuf and flatbuffers do, and use a special compiler for rpc_core schema files that are compiled and in turn used by the rpc_core Python module to describe the objects.

The biggest problem for implementation I see, though, is that the standard Python is CPython, not C++. I don't know how we could get over that hurdle gracefully.

Perhaps pybind11 or Boost.Python would be the way to go.

[shuai132]

It will not be very hard to make a pure-python module. The only difficult work is serialization design.

Detials

When we use standard serialization like json or flatbuffers, rpc_core already support in this repo (c++). And, by the way, json can be transform to anonymous class easily on dynamic language like python and javascript. e.g. JSON.parse for JavaScript, json.loads() for Python.

But if anyone want use build-in serialization of rpc-core in other language, it's impossible use JSON.parse-like utils. Because the serialization-data is not self-descriptive.

Go back to main topic: How to implement a python rpc-core module.

Main rpc logic is no need to say, let's focus to serialization, there are two ways:

1. use standard serialization e.g. json

2. fit the build-in serialization for python

for now, build-in serialization of rpc-core is only for c++, but we can support other language include python:

We should define a python class: for example:

C++:

struct X {
  std::string name;
  uint32_t age;
  std::vector<int> data;
}

Python:

@dataclass
class X:
    name: str
    age: int
    data: List[int]

we should read the code: include/rpc_core/serialize and use reflection function of python to obtain attributes. and then, serialize or deserialize attributes.

To be short, It can be done and should be works well. By the way, this repo already has rust version, and it only support json serialization.

[shuai132]

I don't have much time to do this, and rust version is just for learn rust for me(it available, can work with c++). Maintain more language is a very hard work, the c++ version is first-class for now.

If you/anyone want make a python module, or other language implementation. Welcom to PR, it will be very nice.

[david-drinn]

The problem I see is that uint32_t age, uint16_t age, uint8_t age, int32_t, int16_t, etc. are different types in C++, while age: int in Python fits virtually all such cases.

An important use case for me, is communicating with a small microcontroller running the C++ rpc_core using a higher-powered device such as your laptop, running Python rpc_core. How does the Python developer using the rpc_core module, make it clear that the C++ RPC side is expecting only an integer that fits in an unsigned 16-bit type?

This is why I mentioned the case of Python's built-in struct module and its format strings.

So for example, maybe something like this in the Python rpc_core:

import struct
from dataclasses import dataclass

class U16Int(int):
    FORMAT = struct.Struct("H")

@dataclass
class X:
    name: str
    age: U16Int = 0
    data: List[int]

And then use the Python Struct() class method pack() to make sure that the provided age value in the Python code actually fits into an unsigned 16-bit integer.

Or does the built-in de-serialization handle this already? Or is this just up to the end user of rpc_core to make sure they match sufficiently on both sides of the RPC?

[shuai132]

Yes, It's up to the end user of rpc_core to make sure they match sufficiently on both sides.

Yes, It's necessary to define Uint16 and Uint8, but no need to define Uint32/Uint64, because it's auto size type. The reasion is to save a bit of memory for uint8 and uint16 in c++. auto size type will record some type size info. So, one should read the serialize-code to impl python module.

RPC_CORE_DETAIL_DEFINE_RAW_TYPE(bool, 1);
RPC_CORE_DETAIL_DEFINE_RAW_TYPE(char, 1);
RPC_CORE_DETAIL_DEFINE_RAW_TYPE(signed char, 1);
RPC_CORE_DETAIL_DEFINE_RAW_TYPE(unsigned char, 1);
RPC_CORE_DETAIL_DEFINE_RAW_TYPE(short, 2);
RPC_CORE_DETAIL_DEFINE_RAW_TYPE(unsigned short, 2);
RPC_CORE_DETAIL_DEFINE_RAW_TYPE_AUTO_SIZE(int, detail::auto_intmax);
RPC_CORE_DETAIL_DEFINE_RAW_TYPE_AUTO_SIZE(unsigned int, detail::auto_uintmax);
RPC_CORE_DETAIL_DEFINE_RAW_TYPE_AUTO_SIZE(long, detail::auto_intmax);
RPC_CORE_DETAIL_DEFINE_RAW_TYPE_AUTO_SIZE(unsigned long, detail::auto_uintmax);
RPC_CORE_DETAIL_DEFINE_RAW_TYPE_AUTO_SIZE(long long, detail::auto_intmax);
RPC_CORE_DETAIL_DEFINE_RAW_TYPE_AUTO_SIZE(unsigned long long, detail::auto_uintmax);

RPC_CORE_DETAIL_DEFINE_RAW_TYPE(float, 4);
RPC_CORE_DETAIL_DEFINE_RAW_TYPE(double, 8);
RPC_CORE_DETAIL_DEFINE_RAW_TYPE(long double, 16);

Finally, I suggest use json serialization in defferent language, it's simple and readable, for now it already support by define RPC_CORE_SERIALIZE_USE_NLOHMANN_JSON. My rust version use json to work with c++, and tested.