Mixing public API definitions into private module with Go module references to the external Go module

I have two different buf modules (one private and one public). The private module acts as a proxy to the public module . The private module defines the same RPC API surface as the public one. I'm using managed mode, and I want the Go protobuf generated sources (including the grpc-gateway and other plugins) to reference the Go module for the publicly generated sources instead of using the go_package_prefix default for the public (external) Go module import. I'm having issues getting buf to do this for me.

I've read the following resources to try and get more knowledge:

https://docs.buf.build/generate/managed-mode#go
override keyword

This setting is used for workspace environments, where you have a module that imports from another module in the same workspace, and you need to generate the Go code for each module in different directories. This is particularly relevant for repositories that decouple their private API definitions from their public API definitions (as is the case for buf).

This sounds like what I'm trying to do as well, but I'm struggling to get it to work.

Sample code:

// module: buf.build/myorg/publicapi
// file: echo/v1/echo_service.proto

syntax = "proto3";

package echo.v1;

service EchoService {
    rpc Echo(EchoRequest) returns (EchoResponse)
}

message EchoRequest {...}
messge EchoResponse {...}

// module: buf.build/myorg/privateapi
// file: privateecho/v1/privateecho_service.proto
syntax = "proto3";

package echo.v1;

import "echo/v1/echo_service.proto" // comes from myorg/publicapi

service PrivateEchoService {
    rpc Echo(echo.v1.EchoRequest) returns (echo.v1.EchoResponse)
}

I have a buf.yaml and buf.gen.yaml for the private module that looks like this:

# buf.yaml
version: v1
deps:
  - buf.build/myorg/publicapi

# buf.gen.yaml
version: v1
managed:
  enabled: true
  go_package_prefix:
    default: github.com/myorg/privateapi/proto
  override:
    GO_PACKAGE:
      echo/v1/echo_service.proto: "go.buf.build/myorg/go/myorg/publicapi/echo/v1"
plugins:
  - remote: buf.build/protocolbuffers/plugins/go:v1.27.1-1
    out: proto/
    opt:
      - paths=source_relative
  - remote: buf.build/grpc/plugins/go:v1.1.0-1
    out: proto/
    opt:
      - paths=source_relative
  - remote: buf.build/grpc-ecosystem/plugins/grpc-gateway:v2.8.0-1
    out: proto/
    opt:
      - paths=source_relative
      - logtostderr=true
  - remote: buf.build/grpc-ecosystem/plugins/openapiv2:v2.10.0-1
    out: docs/openapiv2
    opt:
      - openapi_naming_strategy=simple

If I buf generate with the configurations above, then I get code generated from the grpc-gateway plugin that looks like this:

// Code generated by protoc-gen-grpc-gateway. DO NOT EDIT.
// source: privateecho/v1/privateecho_service.proto

/*
Package service is a reverse proxy.

It translates gRPC into RESTful JSON APIs.
*/
package privateechov1

import (
    "context"
    "io"
    "net/http"

    "github.com/grpc-ecosystem/grpc-gateway/v2/runtime"
    "github.com/grpc-ecosystem/grpc-gateway/v2/utilities"
    "go.buf.build/myorg/go/myorg/publicapi/echo/v1" // missing the import prefix echov1
    "google.golang.org/grpc"
    "google.golang.org/grpc/codes"
    "google.golang.org/grpc/grpclog"
    "google.golang.org/grpc/metadata"
    "google.golang.org/grpc/status"
    "google.golang.org/protobuf/proto"
)

...

... // omitting other details here
    var protoReq v1.EchoRequest // uses `v1` import prefix but it should be `echov1`, so this throws a compiler error.

What's the proper way to mix private and public modules like this in a way that works with this usage pattern? I want my private generated Go sources to references the publicly hosted Go module (from the Buf Registry).

bufbuild / buf

Mixing public API definitions into private module with Go module references to the external Go module #1119