[wpiutil] need to support dynamic serialization object for robotpy

[virtuald]

What I'd like to be able to do is make a PyStruct and use that to satisfy the wpi::Struct concept anywhere that there's a function or class that needs a wpi::StructSerializable, and the PyStruct would delegate the actual serialization/deserialization to the underlying data structure; either the normal struct stuff if it's a C++ object that implements that, or some python interface/metaclass/something that implements the right functions.

It was mentioned that currently the struct stuff is just a thin wrapper around Raw bytes in the places that it's currently used, but I would expect this stuff to be used more in the future?

[KangarooKoala]

Sorry, I can't figure out what you're envisioning. Making a type T satisfy the wpi::StructSerializable concept just requires defining wpi::Struct<T> with the appropriate members (type string, size in bytes, schema, and static pack and unpack methods), so I'm not sure what the PyStruct would do. Maybe there could be some helper methods for implementing wpi::Struct<T> for Python objects, but given that those would be specifically for Python I don't think those would belong in the main WPILib library.

[virtuald]

Currently you cannot implement wpi::Struct without everything being known at compile time, which means I cannot use it from Python. I would like a dynamic version to exist so I can use it from Python.

There are lots of things in WPILib that exist because RobotPy needs it, so this wouldn't be a particularly unique case.

[KangarooKoala]

Ah, I see- You're talking about a single type that could be an object whose type is unknown at compile time. That'd make sense to add, since it isn't necessarily tied to specifically Python. (Earlier I was concerned about adding stuff that specifically interacted with the Python/C API, but that was because I didn't understand what was being asked.)

[virtuald]

5992 didn't fix this completely. In particular, the thing I need is to be able to implement a PyStruct and have it dynamically switch the underlying struct serialization at runtime. Because Struct uses static member functions, there's no way to provide any context to inform the serializer/deserializer about the dynamic object that it's supposed to handle.

For example, consider S::GetSize() in a dynamic context -- what exactly is it getting the size of? It either has to be known at compile time, or S can only be used for one type (which means all the types have to be known at compile time).

[virtuald]

We can use a parameter pack to provide an optional parameter to the various struct methods. It's possible that the ntcore/datalog implementations can do some extra work to hide this from the users, but I think the API will likely be mostly the same.

#include <memory>
#include <span>
#include <stdio.h>
#include <vector>

template <typename T, typename... I> struct Struct {};

template <typename T, typename... I>
concept StructSerializable =
    requires(std::span<const uint8_t> in, std::span<uint8_t> out, T&& value, I&&... info) {
      { Struct<typename std::remove_cvref_t<T>, I...>::GetSize(info...) } -> std::convertible_to<size_t>;
    };

template <StructSerializable S, typename ...I> void Serialize(I... info) {
    size_t sz = Struct<S, I...>::GetSize(info...);
    printf("Size is %d\n", sz);
}

struct Pose2d {};

template <>
struct Struct<Pose2d> {
    static constexpr size_t GetSize() {
        return 5;
    }
};

struct Info {
    size_t val;
};

// dynamic case
template <>
struct Struct<Pose2d, Info> {
    static constexpr size_t GetSize(Info info) {
        return info.val;
    }
};

static_assert(StructSerializable<Pose2d>);
static_assert(StructSerializable<Pose2d, Info>);

int main() {

    Serialize<Pose2d>();

    Info info{42};
    Serialize<Pose2d, Info>(info);
    return 0;
}

Output:

$ g++ -std=c++20 t.cpp -o t && ./t
Size is 5
Size is 42