Well-Known Plugins/Plugin Groups

[bufdev]

In Protoeasy, I took the tact that Protoeasy should know the entire world, and it will only handle known plugins. This worked well for basic usage, but proved too limiting, as I eventually had to add the --extra-plugin flag, and keeping up to date with all handled plugins was not scalable in any sense of the word. In prototyping Prototool, I took the compete opposite tact, that I like to think of as the "dependency injection approach", where Prototool is not aware of any plugins, instead leaving you to specify what to compile and where to output it. Through discussions with the Protobuf team at Google, it's become apparent that a middle-ground approach is desired, where we do handle "well-known plugins" in a consistent manner, but leave room to "break glass" if necessary.

The current setup requires something like this:

gen:
  go_options:
    import_path: github.com/uber/prototool/example/idl/uber
    extra_modifiers:
      google/api/annotations.proto: google.golang.org/genproto/googleapis/api/annotations
      google/api/http.proto: google.golang.org/genproto/googleapis/api/annotations
  plugins:
    - name: gogoslick
      type: gogo
      flags: plugins=grpc
      output: ../../gen/proto/go
    - name: yarpc-go
      type: gogo
      output: ../../gen/proto/go
    - name: java
      output: ../../gen/proto/java
    - name: grpc-gateway
      type: go
      output: ../../gen/proto/go

This will output stubs for github.com/gogo/protobuf including grpc-go stubs, github.com/yarpc/yarpc-go, java (the built-in "plugin" for java), and github.com/grpc-ecosystem/grpc-gateway. We denote the type for a Golang plugin to say whether to do Well-Known Type imports from github.com/golang/protobuf or github.com/gogo/protobuf. The import_path will probably be needed in some form regardless, unless we get too cute. extra_modifiers are needed for grpc-gateway, and all the plugins need to know where to output their result.

One of the goals of Prototool is for users to not have to think about these things, and for output to be consistent - a user can start using Prototool, and all output will be in the same location. Roughly, I have a few potential ideas in my head of how to accomplish this.

• Add a base_output option to gen. In the example above, base_output would be ../../gen/proto. This says the directory to put all output to. The default value might be gen/proto.

• Add the concept of Well-Known Plugins. This would be the set of the most commonly used plugins , and we would define default values for the type, flags, output values. We might also overload the concept of name to have some Well-Known Plugins do gRPC generation. As an instructive example, the following would be the equivalent (except for the java override):

gen:
  base_output: ../../gen/proto
  plugins:
    - name: gogoslick-grpc
    - name: yarpc-go
    - name: java
       output: java/example
    - name: grpc-gateway

Where we say output is go for gogoslick-grpc, yarpc-go, grpc-gateway, and the output is java/example for java (the default would have been java).

The Well-Known Plugins off the top of my head would be all the builtins (cpp, csharp, java, etc), all the golang/protobuf and gogo/protobuf plugins, grpc variants of all of these that we can, along with grpc-gateway and yarpc-go (yarpc-go is needed for Uber).

• Another idea is to add the concept of Plugin Groups. This effectively extends the idea of Well-Known plugins to say you have a known group that you always generate. This works great if you define your Protobuf definitions in a different repository than your server implementation, something I want to move towards (and was attempting to prototype back in the day with https://github.com/protocafe). The Plugin Group named "uber" for example might be gogoslick-grpc, yarpc-go, java, cpp, python along with some gRPC variants. The Plugin Group named "default" or "google" might be much more extensive. Think the following:

gen:
  plugin_group:
    name: uber
    exclude:
      - cpp
      - python

[amckinney]

This is awesome, and can actually be introduced in a backwards-compatible way since the configuration keys you're suggesting are either equivalent to the current state, or additional.

I would even rename base_output to simply output, which could otherwise be overridden by each of the configured plugins.

Also, we could rely on a custom UnmarshalYAML implementation so that users can configure their well-known plugins without the name key.

So, your full example would become:

gen:
  output: ../../gen/proto
  plugins:
    - gogoslick-grpc
    - yarpc-go
    - name: java
      output: java/example
    - grpc-gateway

This way, the well-known plugins are clearly visible, needing nothing more than a simple identifier. But users are still free to override some of their behavior, as in the java example.

[bufdev]

I don't think this is needed for v1.0 @amckinney @AlexAPB

[bufdev]

Closing this - working towards Inbox Zero, I don't think we're going to do this, and I've actually soured on the idea as it's a Pandora's Box in terms of support.