Add some type of environment variable support to buf.gen.yaml

[bleleve]

Hello everyone,

Unless I'm mistaken, there is no placeholder to identify the proto file (and its path) being processed during generation.

I use buf to generate the grpc-gateway part. For greater clarity I do not integrate the HTTP rules in the proto but in a separate YAML file (example.proto and _examplehttp.yaml).

It would be interesting to have access to placeholders in the buf.gen.yaml allowing to update an option depending on the proto basename and path:

- name: grpc-gateway
  out: gen/go
  opt:
    - paths=source_relative
    - logtostderr=true
    - allow_repeated_fields_in_body=true
    - grpc_api_configuration={{BUFGEN_PROTO_PATH}}/{{BUFGEN_PROTO_BASENAME}}_http.yaml

Take care of yourself.

[flyte]

I've been looking for something like this too. @bleleve Did you work out a way around this in the absence of variables?

[johanbrandhorst]

Note that you can specify your generation configuration as a yaml or JSON literal:

 $ buf generate --template "{\"version\": \"v1\",\"plugins\": [{\"name\": \"grpc-gateway\",\"out\": \"gen/go\",\"opt\": [\"paths=source_relative\",\"logtostderr=true\",\"allow_repeated_fields=true\",\"grpc_api_configuration=${BUFGEN_PROTO_PATH}/${BUFGEN_PROTO_BASENAME}_http.yaml\"]}]}"

This might do what you want?

[flyte]

Thanks. That's an interesting idea but I'm not sure it really helps me. I've gone back to just calling protoc from a Makefile on this project since it's quite a simple one anyway.

[bleleve]

Hi @flyte and sorry for the delay, I did not see your question! 8 months to respond... that's my record...

While waiting to be able to do otherwise, I use buf to generate everything except the gateway part. In my Makefile buf is followed by a direct call to protoc.

generate: generate-buf generate-gateway

generate-buf:
    @echo "Generating stubs and swagger with Buf"
    @PATH=${PATH} buf generate

# Manually generate gRPC gateway - Waiting for https://github.com/bufbuild/buf/issues/463
generate-gateway:
    @for DIR in $(shell find ${WORKDIR}/proto -name '*.proto' -print0 | xargs -0 -n1 dirname | xargs realpath | sort | uniq) ; do \
        for FILE in $${DIR}/*.proto ; do \
            CFG=$${DIR}/$$(basename -s .proto $${FILE})_config_http.yaml ; \
            if [ -f $${CFG} ]; then \
                PATH=${PATH} protoc -I${WORKDIR}/proto --grpc-gateway_out=paths=source_relative,logtostderr=true,grpc_api_configuration=$${CFG},generate_unbound_methods=true:${WORKDIR}/gen/go $${FILE}; \
            fi \
        done \
    done

[bleleve]

Note that you can specify your generation configuration as a yaml or JSON literal:

 $ buf generate --template "{\"version\": \"v1\",\"plugins\": [{\"name\": \"grpc-gateway\",\"out\": \"gen/go\",\"opt\": [\"paths=source_relative\",\"logtostderr=true\",\"allow_repeated_fields=true\",\"grpc_api_configuration=${BUFGEN_PROTO_PATH}/${BUFGEN_PROTO_BASENAME}_http.yaml\"]}]}"

This might do what you want?

Hi,

I tried this approach on a new project (with remote plugins) but it is not working, buf gives me an error saying that it can't access the YAML file.

If anyone has an idea I'm a interested ;)

buf generate --template "{\"version\":\"v1\",\"managed\":{\"enabled\":true,\"go_package_prefix\":{\"default\":\"github.com/my-account/my-repo/v1/gen/go\"}},\"plugins\":[{\"plugin\":\"buf.build/grpc-ecosystem/gateway\",\"out\":\"gen/go\",\"opt\":[\"paths=source_relative,logtostderr=true,grpc_api_configura
tion=/path/to/the/file/xxx_config_http.yaml\"]}]}"

Error :

Failure: plugin "buf.build/grpc-ecosystem/gateway:v2.15.2" failed: failed to read gRPC API Configuration description from "/path/to/the/file/xxx_config_http.yaml": open /path/to/the/file/xxx_config_http.yaml: no such file or directory

Thank you !

[mfridman]

Unfortunately, plugins that rely on local filesystem access do not work as remote plugins - we require everything to support the contract specified here:

https://github.com/protocolbuffers/protobuf/blob/main/src/google/protobuf/compiler/plugin.proto

So the only option at the moment is to run plugins that require filesystem access as local plugins.

[bufdev]

As @mfridman mentioned, we don't support plugins that rely on local filesystem access - in fact, these aren't actually spec-compliant protoc plugins to begin with! The plugin interface is to send CodeGeneratorRequests to stdin, and CodeGeneratorResponses to stdout - any access outside of this is breaking the protoc plugin contract itself.

But specifically on environment variable support in buf.gen.yaml - that seems like an interesting idea, but I'd want to hear other use cases as I'm not sold. Feel free to chime in!