[Discussion] Streaming support in grpcurl

[ritwik12]

I am not able to figure out where this method is coming from. Is it something which we expect every grpc service to have? https://github.com/fullstorydev/grpcurl/blob/d5b8e4d4ce4c13204ae33c5e2857b9b84d1dba4a/invoke.go#L146

I wanted to understand how grpcurl currently handles streaming and if a grpcurl wrapper is trying to achieve the same thing. How can that be achieved?

currently, I am passing grpcurl args to a subprocess in Python and capturing output there. For streaming, this code will keep on waiting for the output and won't print anything.

grpcurl_process = subprocess.Popen(grpcurl_args, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
output, error = grpcurl_process.communicate()

return output, error, grpcurl_process.returncode

I have worked on a solution to this by creating threads to read the output streams in real-time which prints response line by line. My current challenge is to know if a service is streaming or not. How can I achieve that with grpcurl?

I am thinking of using describe to check if a stream is there in RPC definition or not, but before doing that. I wanted to know if all that is needed actually since grpcurl is handling streaming well directly.

[ritwik12]

@jhump Asking you as you will have better context on this.

[jhump]

@ritwik12, grpcurl handles streaming directly. Your client mainly needs to know if the endpoint expects a stream of request messages from the client, so your program knows whether it must send exactly one request or if it can send any number of messages. It doesn't technically need to know whether the response is a stream: it could just consume all response messages and the error at the end. For a normal (aka "unary") method, it will receive either one response or an error. For streams, it will receive any number of messages followed optionally by an error. But a stream that replies with one message and succeeds or that replies with zero messages and then fails is effectively indistinguishable from a unary response.

I am thinking of using describe to check if a stream is there in RPC definition or not

This seems reasonable. Though the output of describe is intended more for human consumption than machine consumption. You might instead just do a "list" of the service in question and also use the -protoset-out flag. The file created should be easier for a program to load.

TBH, since Python is reasonably well supported with Protobuf and gRPC, you might also just use RPC directly, instead of invoking RPCs via shelling out to grpcurl. If you really need to do dynamic RPC, that is also possible with Python, though I don't know of any libraries to make it easy. (The building blocks used to build grpcurl are mostly described here).

[ritwik12]

Thanks @jhump for your detailed response. It really helps.

[ritwik12]

Thanks for the suggestion but list will only list out method names for the service or host and that won't be useful for my usecase.

I am targeting for stream keyword in a method definition to differentiate it from Unary.

[jhump]

Thanks for the suggestion but list will only list out method names for the service or host and that won't be useful for my usecase.

@ritwik12 , combined with the -protoset-out flag I suggested, it also creates a file. You can load that file (it's an encoded google.protobuf.FileDescriptorSet) and then see if the method of interest is streaming. That will likely be more robust than trying to parse the human-readable output of "describe" to look for the stream keywords.

[ritwik12]

I see. will look into it. thanks for suggesting.