protoc-gen-go: fully generate marshaling and unmarshaling code

[awalterschulze]

The Conversation started in https://github.com/golang/protobuf/issues/276

Is there a possibility of golang/protobuf starting to generate some marshaling and unmarshaling code? I would recommend this over any use of unsafe, since then protocol buffers doesn't just play nice with appengine, but also gopherjs. Also generated code is much faster, which is one of the reasons people flock to gogoprotobuf. It would also be great if the generated code didn't depend on the proto package. But this is just my personal wish, one less dependency for users of a package using protobufs.

• Walter Schulze

We're looking into ways to speed up marshal and unmarshal. Entirely generated code seems unlikely to be the right end point (too much code, not enough performance win), but we're evaluating multiple points in the solution space. Code not depending on the proto package at all is an interesting idea, although that implies even more generated code. I don't think that was on our radar.

• @rsc

[awalterschulze]

I really think the amount of speed gained is significant enough to warrant the extra code generation for marshal and unmarshal.

Here is an independant benchmark https://github.com/alecthomas/go_serialization_benchmarks

BenchmarkGoprotobufMarshal-8                2000000        746 ns/op      312 B/op        4 allocs/op
BenchmarkGoprotobufUnmarshal-8              2000000        978 ns/op      432 B/op        9 allocs/op
BenchmarkGogoprotobufMarshal-8             10000000        211 ns/op       64 B/op        1 allocs/op
BenchmarkGogoprotobufUnmarshal-8            5000000        289 ns/op       96 B/op        3 allocs/op

Also once you have those methods, the extra generated code required to not use the proto package is very little.

[rsc]

fyi /cc @randall77 @cherrymui @dsnet

[awalterschulze]

This surely cannot be news?

But if it is, this is gogoprotobuf, it does code generation (and other gadgets) for protobuffers for go, since 2013. https://github.com/gogo/protobuf

It has users, including Kubernetes https://github.com/gogo/protobuf#users

[awalterschulze]

One of the newer things you can do with gogoprotobuf. Using this command you can still import github.com/golang/protobuf/proto, but get the generated marshaler, size and unmarshaler. So only your code generation steps depends on gogoprotobuf.

go install github.com/gogo/protobuf/protoc-gen-gofast
protoc --gofast_out=. my.proto

Maybe this can give you an idea of what generating code in golang/protobuf would be like.

[johanbrandhorst]

Bump, this discussion should be revisited, as it still has no satisfying conclusion. Recent discussion implies this still hasn't been closed as a possibility. What are the current thoughts on this?

[dsnet]

Wanted update this thread. @neild and I have talked on/off about this in the past. One of things we'd like to explore first before addressing this is to figure out whether to provide a streaming or scatter/gather API (see #609). One significant detriment with generated code is that the implementation easily grows stale unless users regenerate the code (which rarely occurs as has been observed).

Though it's not a hard ordering requirement, we would presumably want to know if we're going to go anywhere with streaming or scatter/gather APIs before doing generated serialization as the former would likely affect the internal APIs and implementation of the latter.

[srikrsna]

planetscale/vtprotobuf Can be used to generate these methods. I think it's best if it is optional and don't mind it living in a separate repo (just like grpc). Because a majority will not significantly benefit from the extra speed and most likely will grow to need this rather than need it from the on set.

One important requirement for tools like vtprotobuf to work more seamlessly is to support custom marshal/unmarshal interfaces similar to encoding/json. I am not familiar with how proto.Marshal works but in my understanding it is as simple as checking for the interface on messages and calling the methods if it satisfies.

Eg:

type Marshaler interface {
    MarshalPROTO() ([]byte, error)
}

func Marshal(v Message) ([]byte, error) {
     if m, ok := v.(Marshaler); ok {
           return m.MarshalPROTO()
     }
     // Current implementation
}

Similarly for unmarshal.

[seeruk]

I've just finished optimising an application as much as I feasibly can for the time being, and I'm at a point now where just over half of the CPU time when under a load is dedicated solely to unmarshalling ProtoBuf messages. Historically, we did use gogoproto, but of course it's now deprecated, so we now use the most up-to-date official method.

I'm really keen to see this at least become an option. The difference, as has been illustrated in the first comments here, are huge. In my current use-case I'd see a pretty significant speed boost!

[quaintdev]

I am not sure why they are hesitating to implement this. This clearly has benefits and there is unnecessary fragmentation in Go protobuf ecosystem because of this.