Symbol not found: grpc.reflection.v1alpha.ServerReflection

[Kernald]

On a server with reflection enabled (works fine with both grpcurl and grpcox), I get this error when running grpcui:

$ grpcui -vvv -bind 0.0.0.0 -port 8082 -plaintext localhost:4567
INFO: 2019/06/14 10:49:37 parsed scheme: ""
INFO: 2019/06/14 10:49:37 scheme "" not registered, fallback to default scheme
INFO: 2019/06/14 10:49:37 ccResolverWrapper: sending update to cc: {[{localhost:4567 0  <nil>}] }
INFO: 2019/06/14 10:49:37 ClientConn switching balancer to "pick_first"
INFO: 2019/06/14 10:49:37 pickfirstBalancer: HandleSubConnStateChange: 0xc00006ee30, CONNECTING
INFO: 2019/06/14 10:49:37 pickfirstBalancer: HandleSubConnStateChange: 0xc00006ee30, READY
Failed to compute set of methods to expose: Symbol not found: grpc.reflection.v1alpha.ServerReflection

[jhump]

I think the issue is that grpcui, unlike grpcurl, tries to pro-actively download descriptors for all services and methods. So grpcurl would likely encounter the same problem if you ever tried to actually use grpcurl to invoke a reflection RPC.

What language is your server written in? Also, can you paste the output of grpcurl ... list and grpcurl ... describe grpc.reflection.v1alpha.ServerReflection?

[Kernald]

The server I'm using is written in C++ (with the official gRPC implementation from here). grpcurl seems to successfully use the reflection:

$ grpcurl -plaintext localhost:4567 list                                             
grpc.reflection.v1alpha.ServerReflection
(My own endpoints here)

$ grpcurl -plaintext localhost:4567 describe grpc.reflection.v1alpha.ServerReflection 
grpc.reflection.v1alpha.ServerReflection is a service:
service ServerReflection {
  rpc ServerReflectionInfo ( stream .grpc.reflection.v1alpha.ServerReflectionRequest ) returns ( stream .grpc.reflection.v1alpha.ServerReflectionResponse );
}

I can also successfully list/describe the methods of my own endpoints.

[jhump]

@Kernald, thanks for the extra info. Hopefully I can dig into it this weekend.

[manuelnp]

Any updates regarding this issue? I just run into the same problem.

[jhump]

@manuelnp, sorry, I haven't gotten a chance to actually debug it.

[manuelnp]

@jhump I found a solution for my case. I'm linking with grcpui statically so I need to link with grpc++_reflection using -whole-archive. Because nowhere in my code is calling to grpc.reflection.v1alpha.ServerReflection, the symbol is discarded at linking time.

-Wl,-whole-archive
-lgrpc++_reflection
-Wl,-no-whole-archive

[jhump]

@manuelnp, oh wow. So it was a linker optimization, throwing away only the descriptor? 🤯

@Kernald, if you have not already found a work-around, can you try out the one @manuelnp just mentioned and let me know if it works for you, too?

[Kernald]

I'll see if I can test this later this week - no promises though, I haven't worked on this in quite a while.

[zmj1368]

I'll see if I can test this later this week - no promises though, I haven't worked on this in quite a while. hi, i have the same info, did you solved this question ?

[xguo-prestolabs]

I have the same problem, server written in c++

[jhump]

@xguo-prestolabs, can you try the compile/link options mentioned above when building your server?

[doraeric]

I have the same problem. It looks fine for grpcurl, but not for grpcui. I am sorry that I don't know how to modify the compile options for server because I am using pre-built packages, and the environment is complicated.

If someone wants to reproduce it, the following script may help. Please run the script in a virtual machine since it pollutes your system. Get a fresh ubuntu 20.04, then run the following in the virtual machine

curl -sL https://raw.githubusercontent.com/doraeric/p4lang-bootstrap/main/install.sh | sudo sh

cd ~
git clone https://github.com/p4lang/tutorials.git
cd ~/tutorials/exercises/basic
make

The grpc server is listening on 0.0.0.0:50051 now, you can confirm with

grpcurl -plaintext $VM_IP:50051 list

But grpcui will fail

$ grpcui -plaintext $VM_IP:50051
Failed to compute set of methods to expose: Symbol not found: gnmi.gNMI

grpcui and grpcurl are both built with go1.17.5

[jhump]

I think this will be fixed in the next release. If you want to try it out before then, build grpcui using the sha from #160 or newer.

I think it's possible that some of the issues were in the server's reflection implementation providing unexpected extra files in the response, and the reflection conflict making incorrect assumptions about the order of those files. Also see https://github.com/jhump/protoreflect/issues/466.

[inliquid]

I have a similar problem (sort of) with production-ready service written in Go which uses gogo lib:

$ grpcui -vv -plaintext localhost:8082
2022/02/23 20:23:50 INFO: [core] original dial target is: "localhost:8082"
2022/02/23 20:23:50 INFO: [core] parsed dial target is: {Scheme:localhost Authority: Endpoint:8082 URL:{Scheme:localhost Opaque:8082 User: Host: Path: RawPath: ForceQuery:false RawQuery: Fragment: RawFragment:}}
2022/02/23 20:23:50 INFO: [core] fallback to scheme "passthrough"
2022/02/23 20:23:50 INFO: [core] parsed dial target is: {Scheme:passthrough Authority: Endpoint:localhost:8082 URL:{Scheme:passthrough Opaque: User: Host: Path:/localhost:8082 RawPath: ForceQuery:false RawQuery: Fragment: RawFragment:}}
2022/02/23 20:23:50 INFO: [core] Channel authority set to "localhost:8082"
2022/02/23 20:23:50 INFO: [core] ccResolverWrapper: sending update to cc: {[{localhost:8082  <nil> <nil> 0 <nil>}] <nil> <nil>}
2022/02/23 20:23:50 INFO: [core] ClientConn switching balancer to "pick_first"
2022/02/23 20:23:50 INFO: [core] Channel switches to new LB policy "pick_first"
2022/02/23 20:23:50 INFO: [core] Subchannel Connectivity change to CONNECTING
2022/02/23 20:23:50 INFO: [core] Subchannel picks a new address "localhost:8082" to connect
2022/02/23 20:23:50 INFO: [core] Channel Connectivity change to CONNECTING
2022/02/23 20:23:50 INFO: [core] Subchannel Connectivity change to READY
2022/02/23 20:23:50 INFO: [core] Channel Connectivity change to READY
Failed to compute set of methods to expose: Symbol not found: product.service.catalog.admin.CatalogAdmin
caused by: File not found: github.com/gogo/protobuf/gogoproto/gogo.proto
2022/02/23 20:23:50 INFO: [core] Channel Connectivity change to SHUTDOWN
2022/02/23 20:23:50 INFO: [core] Subchannel Connectivity change to SHUTDOWN

[jhump]

@inliquid, the latest release included an improvement to the server reflection client to fix cases like this. If you have already tried using that latest version and are still seeing an issue, it is likely a server-side issue.

I spy "gogo" in there, and it is a known source of issues regarding inter-op with other aspects of the protobuf Go ecosystem. Is it possible that you are compiling your stuff with the standard protoc-gen-go, but your proto imports something whose corresponding Go library was compiled using gogoproto? A mismatch like that can definitely cause issues like this. IIUC, with gogoproto generated code, you really need to be "all in" or none at all. The gogo packages have their own registry of files. Server reflection looks at the standard proto runtime set of registered files, but can't find one of the protos because it was registered to a different place.

Anyhow, admittedly that is some speculation. I've largely stayed away from gogo because I've seen weird issues like this pop up before. So it could be something totally different (possibly not even related to gogo).

[ramseyjiang]

Please try grpcui -plaintext 0.0.0.0:4567 directly If find the error below, perhaps you just need to regenerate your *.pb.go, then that error will be fixed,

Failed to compute set of methods to expose: Symbol not found: ***