HTTP binding specifications without a matching selector

[WizMe-M]

When I call protoc-gen-grpc-api-gateway to generate Go gRPC gateway it responds with:

protoc-gen-grpc-api-gateway: HTTP binding specifications without a matching selector

My buf.gen.yaml:

version: v2
clean: true

inputs:
  - directory: .
    paths:
      - module\api
      - module\domain

managed:
  enabled: true
  override:
    - file_option: go_package
      value: example.com/module

plugins:
  - local: ../../../../third_party/protoc-gen-grpc-api-gateway
    out: gen/go
    opt:
      - generate_unbound_methods=true
      - config_search_path=cfg
      - gateway_config=api_config.yaml

My api_config.yaml:

openapi:
  services:
    - selector: 'module.api.service.v1.FooService'
      method:
        Foo: # some configuration

gateway:
  endpoints:
    - selector: 'module.api.service.v1.FooService.Foo'
      post: '/v1/foo'
      body: '*'

I don't know why gateway plugin fails to map HTTP binding with service endpoint. OpenAPI plugin handles the same config correctly. Am I missing some options for protoc-gen-grpc-api-gateway?

[peymanmortazavi]

@WizMe-M is that the entire error? it's ought to list the methods it was unable to find. What would help me reproduce this is a very basic example with a file structure and also indicate from which directory you run these commands. It's definitely a little funny that the API is able to find them. They use the same underlying code for scanning methods and proto files.

[WizMe-M]

@WizMe-M is that the entire error?

Nope, of course there's a huge list with all of the methods. Yeah, it fails to find every method presented in gateway config:

protoc-gen-grpc-api-gateway: HTTP binding specifications without a matching selector: .module.api.service.v1.FooService.Foo (cfg\full_api_config.yaml), <and so on>

What would help me reproduce this is a very basic example with a file structure and also indicate from which directory you run these commands

I call codegen from the proto root directory. So, the file structure is:

proto
├───buf.gen.yaml
├───cfg/
│   └───api_config.yaml   
└───module/
    ├───api/
    │   ├───rpc/
    │   │   └───v1/
    │   └───service/
    │       └───v1/
    │           └───foo_service.proto
    └───domain/
         └───v1/

[WizMe-M]

@peymanmortazavi sorry to rush you, but I would like this issue to be resolved ASAP. Is there anything what I could do to solve it?

It's definitely a little funny that the API is able to find them. They use the same underlying code for scanning methods and proto files.

I think so, but I'm confused that two buf.gen configs (buf.gen.openapi.yaml and buf.gen.grpc_gateway.yaml) referencing the same cfg/api_config.yaml have different behavior. While openapi generation works fine and is able to resolve all methods with specified selectors correctly, gateway plugin, on the contrary, cannot find the necessary methods.

I'm attaching buf.gen configs (as I said in a comment above, they're located under proto root directory and this is where I from call buf generate --template <either openapi, or gateway config>):

• buf.gen.openapi.yaml:

  
  version: v2
  clean: false

inputs:

• directory: . paths:
  • module\api\service\

managed: enabled: true override:

• file_option: go_package value: example.com/module

plugins:

• local: ../../../../third_party/protoc-gen-openapiv3.exe out: gen/swagger/v3 opt:
  • openapi_config=cfg/api_config.yaml
  • gateway_config=cfg/api_config.yaml
  • disable_service_tags=true
  • output_mode=merge
  • output_filename=full
  • output_format=yaml
  • warn_on_broken_selectors=true

• buf.gen.grpc_gateway.yaml:

  
  version: v2
  clean: true

inputs:

• directory: . paths:
  • module\api
  • module\domain

managed: enabled: true override:

• file_option: go_package value: example.com/module

plugins:

• local: ../../../../third_party/protoc-gen-grpc-api-gateway.exe out: gen/go opt:
  • gateway_config=cfg/api_config.yaml

[WizMe-M]

@peymanmortazavi As I see, this is an issue with global gateway config. I use the same config for many services and they're location in file structure is asymmetric.

If there is an issue with global config loading, I suppose two solutions:

1. overwrite options gateway_config_pattern and config_search_path to load not the corresponding config but global;
2. break the global config into smaller ones;

I do not like second way because I can't predict how does plugin handle conflicting API info in openapi.document.info

[WizMe-M]

I tried to reproduce my file structure in a blank new env and that's what I have found:

1. I copied specified buf.gen.yaml, openapi.yaml and my_service.proto (with request and response messages inside it) to the corresponding dirs. Also I added blank foo_service.proto aside with my_service.proto.
   So, buf.gen located in root dir, openapi in cfg dir and protos are in module/api/service/v1 dir.

And it works for some reason...

2. When I extracted messages for request and response into separate folder (module/api/rpc/v1) generation fails with the same error as earlier:

   Failure: plugin ../protoc-gen-grpc-api-gateway.exe: HTTP binding specifications without a matching selector: .module.api.service.v1.MyService.List (api_openapi.yaml)

Used api_openapi.yaml:

gateway:
  endpoints:
    - selector: 'module.api.service.v1.MyService.List'
      get: '/v1/my/list'

3. What is more interesting this is that if I add selector for the foo_service.proto:

   gateway:
   endpoints:
   - selector: 'module.api.service.v1.MyService.List'
     get: '/v1/my/list'
   - selector: 'module.api.service.v1.FooService.FooBar'
     get: '/v1/foo'

And call buf generate again, it will append second selector to the list of missing, however plugin is able to find it in "normal" conditions.

---

This led me to the idea that the plugin only needs to transfer proto-files with services inside. So I changed paths in inputs to the module/api/service and the issue was solved.

---

I suppose, I should close the issue, but I want to know one thing at last:

Which plugins should I use to create reverse proxy (to translate REST requests to gRPC and transfer them to the gRPC Server)? Does I need the only protoc-gen-grpc-api-gateway or should I include other go, grpc and grpc-gw plugins too?

[peymanmortazavi]

@WizMe-M I'm sorry I still don't quite understand a few thing about your project. I'll review again and see if I can follow but here's a few notes that might be helpful. I think in the future, we should prepare an environment where you can create PRs that can reproduce the issue, that way I can see the file structure and thanks for using and reporting issues here by the way!

Plug-ins

• The protoc-gen-grpc-api-gateway is the one that generates the reverse proxy, the actual Go code that maps the HTTP request to gRPC request.
• The OpenAPI plug-in takes in your gateway config and OpenAPI configs or any annotations defined in your proto files to generate OpenAPI specifications that can describe your HTTP API.
• You still need to generate the gRPC methods and the models so yes, you do need the protoc-gen-go-grpc and protoc-gen-go as well but that should be all that is needed.

Conflicting Configs One of the things I tried to work on is to be able to configure each service where it is defined. If you have a global config and have many services, this global config grows rather large. You can always use the global to define general-purpose values like documentation or auth settings and they'll get merged with the service configs. Here's an example:

proto/
    global_config.yaml # this can be used for adding general values like documentation references or auth.
    account/
         v1/
              service.proto
              service_gateway.yaml # you can add OpenAPI and/or gRPC Gateway configs here.
     user/
          v1/
              user_service.proto
              user_service_gateway.yaml # you can add OpenAPI and/or gRPC Gateway configs here.

One thing that is important is that the plug-in MUST be able to find all the proto files used to be able to scan and index them. If you use some message, the plug-ins must be able to spot them.

IF you use these local configs, setting your config search path becomes very important and needs to be set correctly. In simple cases, blank will do fine but if you ever use buf gen <some-directory> then you should set your config search path.

[peymanmortazavi]

@WizMe-M https://github.com/meshapi/grpc-api-gateway/blob/main/examples/internal/buf.gen.yaml that is an example of where config search path is needed, but again that's only if you're using local configs. It seems like you want to mostly use the global config route.