Serialize and deserialize protocol buffers

[kyleconroy]

As outlined in https://github.com/tinygo-org/tinygo/issues/447, TinyGo should support common serialization formats. While most people may think of JSON, I personally want TinyGo to support protocol buffers. This issue is meant to track my progress on adding Protocol Buffer support to TinyGo.

Design

The default Protocol Buffer package (https://pkg.go.dev/google.golang.org/protobuf) makes heavy use of reflection. TinyGo only supports a small amount of the reflect package. Heavy use of reflection leads to slow code, so the folks at PlanetScale wrote vtprotobuf, which generates explicit serialization / deserialization code. This code works today with TinyGo. However, it requires the existing, slow code to compile.

So the plan is simple: Support compiling the code generated by protoc-gen-go and running and of the code contained in init() functions. Use the code generated by vtprotobuf to serialize and deserialize the data.

Roadmap

• [x] Compile code generated by https://github.com/planetscale/vtprotobuf
• [x] Compile code generated by protoc-gen-go
  • [x] https://github.com/tinygo-org/tinygo/pull/2624
  • [x] https://github.com/tinygo-org/tinygo/issues/2806
• [ ] Import code generated by protoc-gen-go
  • [x] https://github.com/tinygo-org/tinygo/issues/2652
  • [x] https://github.com/tinygo-org/tinygo/issues/2668

[paralin]

@kyleconroy This is a great idea, I'm using vtprotobuf a lot currently and it works great. Having some kind of auto-transpiler that automatically converts the reflection-based protobuf-go to use vtprotobuf would be great.

Quick question: as of March 2023, it seems like most of the reflection features are supported - is there anything missing for the existing reflection version to work currently?

[dgryski]

And even more reflect is working, including the last bug on you roadmap.

[gitslav]

@kyleconroy @dgryski looking at the roadmap is it time for the confetti emoji?

[rockwotj]

So it seems that the default protobuf compiler is not supported still even with the improvements to reflection. I have yet to test vtprotobuf

% ./wasmtime-v14.0.4-x86_64-linux/wasmtime run protobuf-example.wasm
panic: unimplemented: (reflect.Type).MethodByName()
Error: failed to run main module `protobuf-example.wasm`

Caused by:
    0: failed to invoke command default
    1: error while executing at wasm backtrace:
           0: 0x5a15 - <unknown>!runtime._panic
           1: 0x5939 - <unknown>!(reflect.rawType).MethodByName
           2: 0x6484 - <unknown>!(*reflect.rawType).MethodByName
           3: 0x103253 - <unknown>!(*google.golang.org/protobuf/internal/impl.MessageInfo).makeStructInfo
           4: 0xb95e6 - <unknown>!(*google.golang.org/protobuf/internal/impl.MessageInfo).init
           5: 0xb8e02 - <unknown>!google.golang.org/protobuf/proto.protoMethods
           6: 0x34c67 - <unknown>!(google.golang.org/protobuf/proto.MarshalOptions).marshal
           7: 0x2d45a - <unknown>!runtime.run$1
           8: 0x2cf76 - <unknown>!runtime.run$1$gowrapper
           9:  0xdc1 - <unknown>!tinygo_launch
          10: 0x2ce48 - <unknown>!_start
       note: using the `WASMTIME_BACKTRACE_DETAILS=1` environment variable may show more debugging information
    2: wasm trap: wasm `unreachable` instruction executed

[rockwotj]

FWIW I have successfully used https://github.com/planetscale/vtprotobuf to marshal/unmarshal protobuf in tinygo -target=wasi

(this can be closed then?)

[wrnlb666]

How are you able to use vtprotobuf with tinygo? Isn't the generated code from vtprotobuf depends on the result of protobuf-go? I am still getting the same error with my _vtproto.pb.go file.

[rockwotj]

How are you able to use vtprotobuf with tinygo? Isn't the generated code from vtprotobuf depends on the result of protobuf-go?

The reflection methods are stubbed out, so it still compiles and the vtprotobuf version doesn't invoke the reflection methods, so it works fine for me.

I am still getting the same error with my _vtproto.pb.go file.

Post the error and make sure you're on the latest tinygo version

[wrnlb666]

I am on the latest tinygo v0.31.1, and I am also using it with wasmtime. The error I got is exactly the same with the one you posted before without using vtprotobuf. Did you change any of the code generated by protoc --go_out ?

[rockwotj]

no modifications needed. Can you post a reproduction?

[wrnlb666]

I am so sorry to bother u, it was me making a dumb mistake causing the error, I fixed the error and it works totally fine. Thank you so much.

[paralin]

vtprotobuf generates static code for Marshal and Unmarshal. We just need a version of protoc-gen-go that does not import reflect nor any of the "heavy" protobuf packages that add a lot of complexity to the resulting binary.

I have forked to protoc-gen-go-lite here and have already dropped everything but the protoc plugin, and am working on adjusting the compiler to generate bare structs without referencing reflect: https://github.com/aperturerobotics/protobuf-go-lite/issues/1

Please join the effort there as this is intended to make it easier to build lightweight tinygo protobuf projects.

[paralin]

I have completed protobuf-go-lite which is a reflection-free fork of protobuf-go merged with vtprotobuf and protoc-gen-go-json.

Library: https://github.com/aperturerobotics/protobuf-go-lite Example: https://github.com/aperturerobotics/protobuf-project/tree/protobuf-lite

RPCs are available as well with starpc: https://github.com/aperturerobotics/starpc

protobuf-go-lite now supports reflection-free protobuf and protojson encoding!