[feat][protoc-gen-openapi] Support Multiple Output Files

[jeffsawatzky]

Currently protoc-gen-openapi generates one openapi.yaml doc from a list of proto files. My use case will require multiple openapi.yaml files, I think ideally 1 per proto file (or perhaps one per package?)

My use case is as follows. Assume I have two microservices, service_a and service_b. Currently they are both at version v1. Therefore I would have something that looks like:

protos\
   service_a\
      v1\
         a.proto
   service_b\
      v1\
         b.proto

Right now when I generate the OpenAPI doc it adds all of the protos to the single openapi.yaml doc which is fine for now.

Now lets assume I need to make a breaking change to service_a so I create a v2 and now I have something like:

protos\
   service_a\
      v1\
         a.proto
      v2\
         a.proto
   service_b\
      v1\
         b.proto

I want to be able to create a new OpenAPI doc without the service_a.v1 definitions in it.

My current plan would be to have each proto get compiled to it's own unique openapi.yaml file based on the proto folder structure. Then have some post build step that would combine only the openapi.yaml files I want in the file documentation (perhaps using something like yq)

It should be noted that we are using buf to build our protobufs, so we don't have a huge amount of control over how proto files are provided to the generators.

This would be similar to how the protoc-gen-openapiv2 from the grpc-gateway project works by default.

@timburks / @morphar is there a simpler solution that I am missing?

[morphar]

There was another request for this, some time ago. I think that 1 openapi.yaml is the desired default, right @timburks? But for compatibility with buf, it seems that 1 .yaml file per .proto is what is needed.

With regards to your use case: is there any hints in the buf documentation? I would imagine, they have thought this scenario through?

I built up my architecture around how gnostic works. My protos is ordered like this: flowstack/api/accounts/v1/accounts_service.proto flowstack/api/accounts/v1/accounts.proto These files have package flowstack.api.accounts.v1

flowstack/api/accounts/v2/accounts_service.proto flowstack/api/accounts/v2/accounts.proto These files have package flowstack.api.accounts.v2

So I duplicate the files from v1 to v2, before I make changes.

This seems to be pretty much the same as yours, right?

In a Makefile, I create 2 openapi.yaml files (openapi_v1.yaml and openapi_v2.yaml). The important parts:

V1_PROTOS=${shell find ${PROTOS_PATH}flowstack/ -path "*/v1/*.proto"}
V2_PROTOS=${shell find ${PROTOS_PATH}flowstack/ -path "*/v2/*.proto"}

generate-openapi:
    @${PROTOC_CMD} \
        --openapi_opt=title="FlowStack API",version=1.9.1 \
        --openapi_out=${OPENAPI_OUT} \
        ${V1_PROTOS}
    @mv ${OPENAPI_OUT}openapi.yaml ${OPENAPI_OUT}flowstack_v1.yaml

    @${PROTOC_CMD} \
        --openapi_opt=title="FlowStack API",version=2.0.1 \
        --openapi_out=${OPENAPI_OUT} \
        ${V2_PROTOS}
    @mv ${OPENAPI_OUT}openapi.yaml ${OPENAPI_OUT}flowstack_v2.yaml

Maybe that can translate to your scenario?

An alternative could be to create a list in the Makefile and iterate over it, generating 1 .yaml for each .proto file.

[jeffsawatzky]

@morphar I see the duplicate #268. I will close this and add comments there.

But to answer your questions, when we create a new version of a microservice we do start by copying all of v1 to v2 and then make our modifications. However, we want to be able to version the back-end services independently, so bumping service_a to v2 doesn't require bumping service_b to v2 as well. Each service is owned by a different team here, and we will have many of them, so having to keep them all in sync will be annoying for the teams. Every time you create a new version you need to create new service and client stubs, implement the new service while also retaining the previous until you have updated all the clients, etc. We would like to have to do that ONLY when needed.

We could amend your script do work in our case, but the other problem is that by using buf we don't get control over how protoc is called, and if we use the remote plugins (as we are) protoc isn't even run locally.

I have more comments which I will add to #268

[timburks]

I'll follow at #268, thanks