Zeebe Client closes grpc stream to the gateway leading to warnings in gateway

[shahamit]

When running this tool against a zeebe cluster having 6 brokers (12 partitions) I see info messages on the console like this io.grpc.internal.AbstractClientStream : Received data on closed stream At the same time there is a warning logged in the zeebe-gateway.

2023-02-15 10:30:13.522 [] [grpc-default-worker-ELG-4-2] WARN
      io.grpc.netty.NettyServerHandler - Stream Error
io.netty.handler.codec.http2.Http2Exception$StreamException: Stream closed before write could take place
        at io.netty.handler.codec.http2.Http2Exception.streamError(Http2Exception.java:173) ~[netty-codec-http2-4.1.82.Final.jar:4.1.82.Final]

Surprising this warning does not appear when running against a zeebe cluster with a lower configuration - 2 brokers (2 partitions). With the earlier config the completed PI/s go above 200 while with 2 brokers, we get around 50 PI/s.

Assuming the zeebe client connection to the gateway is closed by the benchmarking tool, I tried increasing the timeouts - zeebe.client.job.timeout and requestTimeout to higher values but there was no change in the behaviour.

Also note that there is 0 backpressure reported when these warnings appear which also seemed strange.

Any idea why does the tool close the connection?

Thanks.

[chDame]

Hi @shahamit, did you check that your java client version is the same as the targeted cluster ?

[shahamit]

hi @chDame - sorry for the delayed response. It seems I missed your reply.

did you check that your java client version is the same as the targeted cluster ?

No the java versions are different. Zeebe is running on java 17 while the benchmarking tool is on java 11. Just curious, why could that cause a connection close issue? In parallel I will try to deploy the benchmarking tool on java 17 and see if the error persists

Thanks.