search

grpc / grpc-java

The Java gRPC implementation. HTTP/2 based RPC
https://grpc.io/docs/languages/java/
Apache License 2.0
11.4k stars 3.83k forks source link

ServerCallStreamObserver#isReady seems to be reporting incorrectly causing crashes #10905

Closed Jrashid1998 closed 7 months ago

Jrashid1998 commented 8 months ago

Currently on 1.60.2 for io.grpc:grpc-stub, io.grpc:grpc-protobuf, io.grpc:grpc-netty. Running a GRPC Java Server routing to a react frontend (using nice-grpc 2.1.4 and nice-grpc-web 1.1.0) via grpc-webproxy v0.15.0

I'm using bidi streaming and we send messages quite heavily from server to client. This is regulated using the isReady() conditional but I've found the stream will crash with a nondescript CLIENT_CANCELLED error unless I manually rate limit what we send through. If I log true/false against the isReady() conditional, I can see it's always true until the stream dies implying there is no regulation.

I am querying the responseObserver for the isReady() condition and I can see a ticket open around how that could be a check for server readiness instead of client however it's still open and the comments seem ambiguous. https://github.com/grpc/grpc-java/pull/8910

If it is only an indication of sending side readiness, is there a way to check client readiness from the server? Happy to provide any more info if needed

sergiitk commented 7 months ago

Hey @Jrashid1998. Could you please provide the full response you're getting? CLIENT_CANCELLED is not found in our source. Would be great to have the stack trace of the crash too.

ejona86 commented 7 months ago

I would figure that most likely one the other components you are using (e.g., grpc-webproxy) isn't applying flow control/backoff, and when it buffers to much it kills the RPC. grpc-java doesn't behave that way. If the proxy is "near" the server, then gRPC might be sending to it as fast as you are generating messages, so isReady() stays true.

sergiitk commented 7 months ago

No response to provide more information, so closing. More information can be provided and then we can reopen.