Add support for "option" field in protobuf message

[idokleinman]

For a protobuf message of type

message AuthRequest {
  option (type_id) = 1001;
  // Commissioning credential
  string password = 1;
}

Resulting Swift code does not contain the option (type_id) = 1001; and the number cannot be accessed. Are there plans to add support for class descriptions (aka protobuf option)? Is it already supported and I'm missing a plugin switch?

[thomasvl]

Options are metadata that is only captured in the descriptor describing the messages. The Swift proto runtime currently doesn't need any of this information because doesn't use it and the data can be a non trivial amount of data added to binaries that most things don't need.

The first step would likely be to bundle the generated code for descriptor.proto with the library (Issue #727), and then figure out how to capture the data into the runtime and then expose it from messages/enums/etc.

[richardasmall]

Has there been any more thought put into this? Just discovered this myself. Our team uses custom options extensively in our protos (for better or worse).

[thomasvl]

Not really. We did start bundling descriptor.proto a while ago to remove that issue.

As mentioned, since the data can be non trivial in size, figuring out a way to do this that doesn't bloat things is important. The other issues asking for the protobuf reflection type apis (act on message fields generically), also need some of the same information/apis, so there's a good chance that would all be done together.

[tbkka]

Other protobuf implementations handle this by dumping the serialized descriptors into the generated code in their entirety. When you access your custom option, they deserialize the descriptor and look up the option data there.

If this is important to you, you can do this manually: Use protoc to dump the descriptors, bundle them into your code, and then deserialize them and inspect them to retrieve the custom options. It's a little tedious, but feasible.

We've been slow to implement this by default for several reasons:

1. Swift does not currently have a really good way to embed binary data in the generated code.
2. The deserialized descriptors can use a lot of memory. Many people object to this.
3. Getting useful data out of the descriptors can be pretty tedious. Other implementations have pretty involved collections of helper code to present a nicer interface to this. We've built a little of that support for the code generator but it's not yet complete enough to support these kinds of uses.

[tbkka]

Here are some hints for folks who want to use options:

1. You can generate a FileDescriptorSet using protoc:

   protoc --descriptor_set_out=.  some.proto other.proto

2. That FileDescriptorSet is a protobuf-serialized copy of your proto definitions. You can deserialize it with the descriptor type that's included in the SwiftProtobuf library:

   import SwiftProtobuf
   let fileDescSet = Google_Protobuf_FileDescriptorSet(serializedData: data)

3. From this, you can walk the proto definition and read off any options you're interested in. The details of the file descriptor set are in descriptor.proto here: https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/descriptor.proto

In particular:

• A "FileDescriptorSet" holds an array of "FileDescriptor" objects, one for each file in the set.
• A "FileDescriptor" object has an array of "Descriptor" objects defining each message.
• A "Descriptor" has an array of "FieldDescriptor" objects defining the fields of that message.
• Each "Descriptor" and "FieldDescriptor" includes an "options" array with all options defined on that message or field.

The above information should be enough to make use of option data. The only missing piece is somehow embedding the information into your project. The C++ implementation does this by embedding the entire FileDescriptorSet into the generated output, and implicitly deserializing the entire thing in order to look up option information. This has obvious problems with size (code size for the embedded data and memory size for the decoded descriptor information) which is a major reason we've been hesitant to implement this so far.

If enough people experiment with this, maybe we can collectively acquire enough experience with different approaches to come up with a good solution. (For example, I've recently started wondering if there could be a separate generator that just generated this information as extensions to the Swift message types; you could then run that separate generator in addition to protoc-gen-swift to get this additional support. A similar approach might be used to split out the JSON/text support logic that some people would like to omit.)