Support gRPC reflection

[Daniel-B-Smith]

gRPC Reflection is very useful for debugging and with tools like grpc_cli. I would probably consider this a lower priority feature but do think it should be on the roadmap.

[LucioFranco]

Hi! Yes, this would be really nice to support! The biggest issue right now is we are trying to decouple our codegen/encoding (prost/protobuf/flatbuffers) from tonic's core code. This will make it harder to get the proper abstracted file descriptors needed to generate this. That said I would def like to support this but its gonna be a bit lower priority.

[mehcode]

Has any forward progress been made on this? How involved do you think it would be for someone to jump in and try to implement it? Using Tonic right now for a project and it works great :100: but if we could support the reflection API that would be a nice bonus.

[LucioFranco]

@mehcode I think this is a pretty hard one to be honest. The big blockers are first even getting reflection support in prost then finding a way to do this via codegen. The second problem is that we are going to soon support flatbuffers out of the box and that means we will need to be more generic over our encoding type which will mean we will need to either only support reflection for protobuf or find a generic way to support it.

I have not really spent that much time on this so its not been a high priority but I am happy to mentor anyone that is interested in starting the work.

[rokadias]

Looks like I filed a duplicate for wanting this functionality as well. I potentially have a way of accomplishing that. There is the ability to add attributes for types and fields in prost.

https://docs.rs/prost-build/0.4.0/prost_build/struct.Config.html#method.type_attribute

Then an attribute macro could be created to supply the data that the service needs to service it's requests. At a high level, I think that could work, but I admit that I haven't built something like that before so I could be off on some details here.

https://doc.rust-lang.org/reference/procedural-macros.html#derive-macros

I'm a little hesitant to take on this work as I'm not sure if the work with flatbuffers will conflict.

[sleipnir]

Hi guys! Undoubtedly having this functionality would be very important. Generally, most people mention that this would be important only for things like grpcurl, however, there are many important use cases like proxies, dynamic routers, and etc. that could be built on top of this functionality. I for example work on a project that creates a sidecar kubernetes pod and implements the grpc interfaces of the adjacent container. This is only possible because the sidecar is able to reflect the client container grpc contract. Today we do this in Scala + AKKA gRPC but if we could implement it in Rust it would be much more efficient and at a much lower cost in cluster resources.

[LucioFranco]

Yeah, I do think it makes sense to implement, just need someone to champion it :)

[jen20]

I looked at this for a few minutes this evening and think it should be doable. prost_types already contains definitions for FileProtoDescriptor, ProtoDescriptor et al. In the interests of science I took the (gzipped) descriptor generated by protoc-gen-go v1.3.5, and they appear to decode OK.

As yet, I haven't looked at the internal structures used by prost-build yet, but I'd imagine it must be possible to build the descriptors from them, and encode them as gzipped arrays in the generated code for use by an implementation of the reflection service.

I don't know how much time I'll get to work on this in the very near term, but I'll update here as and when I find more.

[jen20]

Something I had not appreciated while writing last comment is that the FileProtoDescriptor set is actually the request format between protoc and it's plugins, and is produced by protoc when no plugin is specified. This is how prost-build uses protoc, here:

https://github.com/danburkert/prost/blob/master/prost-build/src/lib.rs#L524-L542

The temporary file is then parsed as FileDescriptorSet, here:

https://github.com/danburkert/prost/blob/master/prost-build/src/lib.rs#L552-L553

It seems like it should be possible to retain this in order to power reflection. Actually implementing reflection is probably best left to a separate crate since it presents as a gRPC service (tonic-health provides us a model for this).

The most "official-looking" definition of the reflection API suggest that it is protocol-buffers specific, so even if we supported other encodings in a reflection crate at this point it's unlikely any other implementations would be able to make use of it.

@LucioFranco Does this seem like a reasonable approach to you?

[sleipnir]

Great job @jen20

[LucioFranco]

@jen20 seems like a good idea, if we can already invoke protoc via prost-build then calling it again from a separate crate for reflection would make sense. I think we can add a tonic-reflection that is similar to what the health crate does but also includes a build step for generating static versions of the filedescriptorset. What would be really nice is to statically build the descriptor set and embed it in binary form into our source.

[jen20]

Statically building it is a nice option I hadn't really considered. I'll have a play around with that when I next get a break from writing Go 🙄

[LucioFranco]

next get a break from writing Go 🙄

Sorry to hear this, good luck. 😄

[jen20]

I've started to make some progress on implementing the gRPC end of the protocol. Definitely nowhere near done, but grpcurl is now usable:

$ grpcurl -plaintext 127.0.0.1:7000 list
grpc.reflection.v1alpha.ServerReflection
helloworld.Greeter

Once I have things in a shape that is not a complete mess, I'll open up a draft work in progress PR to track this.

[wchargin]

Thanks to @jen20 and everyone involved for pushing this through! I’m quite excited for it. I’ve tested it out by taking a Git dependency on head, and it seems to work as advertised. At long last, I can avoid dances with pointing grpc_cli --protofiles to a bespoke symlink tree!

@LucioFranco: Any plans to release soon? I see discussion of a v0.4.1 release in #563, but presumably this feature would wait until v0.5.0.

[LucioFranco]

@wchargin https://docs.rs/tonic-reflection/0.1.0/tonic_reflection/ was released a while back.

[talksik]

I can't figure out how to use tonic-reflection. Is there no easy guide? The docs (https://docs.rs/tonic-reflection/0.1.0/tonic_reflection/) doesn't explain anything.

Thanks 🙏🏾

[talksik]

For others out there, there is an example here: https://github.com/hyperium/tonic/blob/master/examples/src/reflection/server.rs

You sort of have to piece together what's needed based on this file and also the cargo.toml used.

[Jawell]

Hi! How to compile reflected services in the build time or runtime of client? I can't find any examples. It seems like a useless feature without the ability to create a client that can use reflected data 🤔

[praveenperera]

@LucioFranco is flatbuffers support still on the horizon or is it already possible?

[listepo]

How to use for generated code? without tonic::include_proto! ?