Generic marshaller for protobuf

[ebilling]

My particular problem is that I want to marshal a protobuf that can contain OneOf fields into and out of YAML.

This is a problem for a few reasons:

1. One of fields are problematic in that they are not marshaled cleanly for almost any parser except the protojson parser which is part of this package.
2. The protoc-gen-go package only gives marshaling instructions for JSON and protobuf, so the YAML parser uses capitalized names for fields which is not ideal.
3. Some people have solved this by marshaling into and out of JSON, and then marshaling into and out of a map[string]interface{} object, and then into whatever other format they want. The performance of this is less than ideal.

Potential Solutions:

1. Create a new YAML marshaller for protobufs that is essentially an YAML specific implementation of the protojson marshaller.
2. Create a marshaller that goes directly from the protobuf object in go to the map[string]interface{} and back. This approach might be generic enough that any protobuf can be marshaled into and out of just about any format that could support the generic map object.

Alternative Solutions:

I've tried to introduce changes for go-yaml that allow the use of JSON format directives when YAML directives are not available, but it was rejected. Also, it would not deal with the oneOf issue.

My other option is to write my own marshaller. Because YAML is so common in the K8s community, I would expect that this is common enough of a problem that a solution in the protobuf package would be useful to the wider community.

I also suggested a few years ago we extend the protoc-gen-go compiler to allow for format specific extensions.

[puellanivis]

The golang protobuf project had added json struct tag directives because it was thought that it would be so easy to do, that it was a positive feature to bring to the project. And then an official JSON mapping standard was added into the proto3 standard, and as a result, the simple JSON implementation became out of spec, and was part of why we had to develop the protojson package.

As a result, this package, being the official implementation realized that adding features unilaterally is a bad idea, because only the Go implementation would provide that feature, which is antithetical to the multi-language design goal of protobufs.

Summary: we would not consider a YAML marshal/unmarshaller at this time. We would want to see a formalized standard in the larger project for how to convert from-to YAML like we have for JSON. Though, there’s nothing really stopping anyone from using the protoreflect package to write their own YAML implementation—knowing that if a standard were ever released, they might find themselves out-of-spec.

The structpb package already provides AsMap and there is a reasonable amount of work that could be done to convert any arbitrary protobuf into a generic map… again should be possible through the protoreflect package.

[ebilling]

Thanks for the rapid response.