search

JuliaIO / ProtoBuf.jl

Julia protobuf implementation
Other
205 stars 55 forks source link

Would StructTypes.jl integration be an idea for the long run (?) #166

Closed rapus95 closed 2 years ago

rapus95 commented 3 years ago

Given that ProtoBuf.jl is mostly about de/serialization and StructTypes.jl is a kinda idiomatic approach to generic de/serialization I was wondering whether it'd make sense to streamline the API/interface. The idea would be that ProtoBuf is the StructTypes.jl-agnostic codegen and StructTypes makes the generic interface for de/serialization into binary (as for ProtoBuf) or textual (as for JSON) etc but all through the same interface. In some sense the Tables.jl for Structs. Right now as far as I see it, both packages hold code for de/serialization and thus, switching to use StructTypes as the API or serialization might be a chance for reducing duplicate code, integrating into a larger ecosystem and pushing boundaries on the ecosystem/maturing it.

Count this as a VERY "defensive" suggestion, an idea, I'd like to get your thoughts on.

Drvi commented 2 years ago

I think in the long run, we want to use some sort of shared abstraction for julia codegen. But I have a hard time envisioning an API general enough to support things like Clang.jl, ProtoBuf.jl, JSON3.jl plus formats like JSONSchema or ASN1.

rapus95 commented 2 years ago

Have you had a look at https://github.com/JuliaData/StructTypes.jl ? Do you have some fast objections or points which make it feel like that package couldn't solve all those requirements?

Drvi commented 2 years ago

I skimmed the package, but it is still not clear to me how the it should be used, tbh. Are you suggesting that instead of the encode / decode methods we generate with protojl, we'd instead generate StructTypes.jl metadata methods and make the encode / decode method in this package generic?

@quinnj do you have any thoughts on this?

rapus95 commented 2 years ago

protobuf needs to de/serialize at some point. Now instead of offering its own api for that and generating custom code my suggestion was to hook into the api of StructTypes and build a serializer based on the methods of that package that conforms to the protobuf specs. defining the structtypes api for my own type means that I somewhat describe what is data. And you can use that description to build a serialized output of it. no need for meta programming at all. best probably would be to go both ways and offer support for all types that support de/serialization by using that api and keep the current approach for everything else. Advantage of this approach is that at some point in the future you can just replace JSON3.jl with ProtoBuf.jl and everything still works because both uses StructTypes.jl as de/serialization interface. StructTypes can be used as a de/serialization framework. The user provides a frontend (Type<->StructTypes.jl) and the protocol author provides a backend (StructTypes.jl <-> De/Serialization) by extracting Data from Julia objects using the StructTypes.jl API.

I hope this clarifies it a bit. Keep asking if not! 😄

quinnj commented 2 years ago

In my opinion, this could live as a separate package that did a ProtoBuf.jl + StructTypes.jl integration, but I also think it will be non-trivial, and probably have lots of corner case weirdness. There are probably simple cases that would work for the most part, and maybe the aim would just to keep things restricted to simple cases, but in general, I tried this kind of approach with the original FlatBuffers.jl package and it lead to a lot of trouble later on. So I say if someone is up for the challenge, then go for it! Happy to help answer questions/help review along the way.