compiler/protoc: a Go wrapper around protoc

[myitcv]

Is your feature request related to a problem? Please describe.

Yes. I'm required to manually discover and then install the correct version of the standard C++ compiler (per https://github.com/golang/protobuf#installation). I say "correct" because I want to ensure that the contents of https://github.com/golang/protobuf/blob/master/protoc-gen-go/descriptor/descriptor.pb.go (for example) correspond to the correct compiler version, i.e. regenerating github.com/golang/protobuf/protoc-gen-go/descriptor should be a no-op.

Describe the solution you'd like

Ideally, if I use v1.2.0, say, of github.com/golang/protobuf (via Go modules) then there should be a Go-based wrapper around protoc (e.g. github.com/golang/protobuf/protoc) that downloads and runs the correct version (v3.6.1) as required.

Describe alternatives you've considered

Manually discovering the correct version to install from: https://github.com/golang/protobuf/blob/c823c79ea1570fb5ff454033735a8e68575d1d0f/.travis.yml#L12

installing a non-Go dependency, writing scripts, etc.

Additional context

I haven't fully kept up to speed with your plans for v2, so apologies if this is covered elsewhere.

This would also make the protoc compiler a fully-fledged Go dependency, and would therefore make it more conducive to using via go generate or gg

[myitcv]

To add one other "wish" to the list: that the wrapper understands -I arguments as package paths. This could obviously be via a different flag, e.g. -Ipkg example.com/mypkg. Values would be translated via go list -f={{.Dir}} and passed as a -I value to the underlying protoc.

[bufdev]

I do basically this in Prototool FYI https://github.com/uber/prototool/blob/dev/internal/protoc/downloader.go

A lot of Prototool is effectively a protoc wrapper.

Having done this, I think I would not do it again. It's fine for local development, but in Docker, Bazel, etc, you want to explicitly declare your dependencies and download them ahead of time, so you end up overriding the functionality. Additionally, for protoc to really work as it's intended, you need to also provide a copy of the WKT (and automatically add them to includes for protoc, which surprisingly is done for you if they are at ../include to the path to protoc).

Way nicer for the community would be an eventual Golang protoc port, but there's a lot of stuff that would need to happen first.

[myitcv]

Thanks for the thoughts @peter-edge, and for the link! I was fairly certain this was not a "new" idea 😄

Having done this, I think I would not do it again

Can you expand on this a bit? Beyond the dependency management point?

It's fine for local development, but in Docker, Bazel, etc, you want to explicitly declare your dependencies and download them ahead of time,

I have a proof of concept (doesn't support all flags etc) at https://godoc.org/myitcv.io/cmd/protoc (code https://github.com/myitcv/x/tree/master/cmd/protoc). I'm using this wrapper in //go:generate directives (in conjunction with https://godoc.org/myitcv.io/cmd/gg to ensure the code generation happens in the right order).

Because I'm using Go modules, dependency management is taken care of by virtue of me being able to require a specific version of protoc (my wrapper). For now it's part of my wider mono-repo, but that's a detail, particularly if you consider that such a wrapper could/should live in this repo.

you need to also provide a copy of the WKT

Sorry, I'm probably missing something obvious here; but what is WKT?

Way nicer for the community would be an eventual Golang protoc port, but there's a lot of stuff that would need to happen first.

But that's my point; a Go wrapper is almost as good, and we can do that now.

FWIW I'm having some success with this approach for now, but I'll be sure to report back with any findings.

[bufdev]

The dependency management point is the main one - most build systems/build setups will have protoc installed on a specific version as it is, although this is useful for those unmanaged (smaller codebases, OSS, etc) situations.

WKT are the Well-Known Types, ie google/protobuf/timestamp.proto, google/protobuf/duration.proto, etc...If you have installed protoc normally, they are automatically available, but if you download it somewhere, you can't rely on this, so you need to do protoc -I path/to/wkt.

Note you can do all of the above with prototool heh:

protoc:
  version: 3.6.1
generate:
  go_options:
    import_path: github.com/foo/bar/path/to/this/file
  plugins:
    - name: go
      type: go
      flags: plugins=grpc
      output: gen

The prototool binary also takes care of setting all the Mfile=package values, including for the WKT for either golang/protobuf or gogo/protobuf.

Then docker run -v $(pwd):/work uber/prototool prototool generate will do all of this, including a managed golang/protobuf version. See https://github.com/uber/prototool#prototool-generate and https://github.com/uber/prototool/blob/dev/docs/docker.md

[lucasls]

I'm not sure if it applies to this situation, I tried to follow the thread but I'm too out of context (not being a Go developer myself) to understand the problem. But well, I've developed here at Free Now a very simple shell script based wrapper for protoc.

https://github.com/freenowtech/protoc-wrapper

It can be easily downloaded by pasting a one liner to the terminal. It'll ask for the desired version and store it to a companion file.

It's inspired by Gradle Wrapper and works very similarly. Basically, you call ./protocw instead of protoc and it will download the right binary for the specified version and current operating system, if needed, and forward all commands to the binary.

It's not a full featured solution like Prototool, by design, only meant to be a wrapper. Let me know if it applies to your problem and feel free to provide feedback and ask for improvements.

[tooolbox]

Since I found this thread looking for a Go wrapper for protoc, I'm making a note that this is what I eventually went with: https://github.com/sixt/protoc

#!/bin/sh
export PATH=$PATH:$(gobin -m -p github.com/golang/protobuf/protoc-gen-go)
gobin -m -run github.com/sixt/protoc/v3 <arguments, etc.>
# (Roughly)