RST_STREAM 1 error with GRPC and ESP

[gombasp]

We have following setup:

Postman -> ESP -> (conversion to grpc) -> backend

Using io.grpc java version 1.44.

When sending request i see in Postman following response:

"message": "Received RST_STREAM with error code 1",

Backend log:

febr. 21, 2022 2:41:02 DU. io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2FrameLogger logHeaders FINE: [id: 0x1ab4b995, L:/[0:0:0:0:0:0:0:1%0]:8089 - R:/[0:0:0:0:0:0:0:1%0]:35372] INBOUND HEADERS: streamId=13 headers=GrpcHttp2RequestHeaders[:path: /HelloService/hello, :authority: localhost:8089, :method: POST, :scheme: http, te: trailers, accept: /, accept-encoding: gzip, deflate, br, content-length: 45, postman-token: 7c1b4fdb-4fe4-4f1f-a431-4032e8790a6f, x-api-key: AIzaSyCvhWZ2ixEhuBV96VcOiXPUFYx7WTYCV1Y, x-endpoint-api-project-id: 247678761469, x-forwarded-user-agent: PostmanRuntime/7.29.0, content-type: application/grpc, user-agent: grpc-c++/1.22.0-dev grpc-c/7.0.0 (linux; chttp2; gale), grpc-accept-encoding: identity,deflate,gzip] padding=0 endStream=false febr. 21, 2022 2:41:02 DU. io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2FrameLogger logData FINE: [id: 0x1ab4b995, L:/[0:0:0:0:0:0:0:1%0]:8089 - R:/[0:0:0:0:0:0:0:1%0]:35372] INBOUND DATA: streamId=13 padding=0 endStream=true length=15 bytes=000000000a0a080a02616112026262 febr. 21, 2022 2:41:02 DU. io.grpc.netty.shaded.io.grpc.netty.NettyServerHandler onStreamError

WARNING: Stream Error io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2Exception$StreamException: Received amount of data 15 does not match content-length header 45"

It seems that ESP is forwarding the content-length header with the value of 45 (coming directly from Postman) however in grpc terms the correct size would be only 15. Please see ticket https://github.com/grpc/grpc-java/issues/9019 where i was advised to report the issue to you.
Would it be possible to do a filtering in ESP (or send the correct content-length)?

[qiwzhang]

Yes, ESP indeed pass content-type to Grpc server.

[gombasp]

@qiwzhang just to be sure: we are talking about the content-length, not the content-type.

[qiwzhang]

Sorry, it was a typo. I mean "content-length" too. Here is the code of filtering the headers from client to grpc server. It did not remove content-length.

[gombasp]

@qiwzhang im wondering when can be expect a bug fix? Currently this makes us unable upgrade the grpc libs - however there are newer versions already (not urgent, but after a point it would make sense). Can you give us any info?

[qiwzhang]

Let me summarize the issue: grpc should not use "content-length" in its header, but ESPv1 sending it. Java grpc-server starts to enforce this in the version 1.44.0, the requests proxyed by ESPv1 starts to fail if Java gRPC server upgrades to new version.

Is that correct?

We are estimating the effort to fix this. Will update you.

[qiwzhang]

BTW, is it possible to upgrade your system to use ESPv2?

[gombasp]

@qiwzhang your summary is correct. My only small remark is that it fails already from grpc version 1.42.0.

About ESPv2: we will consider if this would be possible. Are you recommending it because the bug is already solved in v2?

[gombasp]

@qiwzhang i tested my original scenario with espv2. Unfortunately i only get a http 415 in response, regardless of the grpc lib i use. Maybe i need to add some extra options in case of esp v2? I started it locally similar to this:

sudo docker run \ --detach \ --name="esp_hello12" \ --net="host" \ --volume=/home/user/projects/:/esp \ --publish=8087 \ gcr.io/endpoints-release/endpoints-runtime:2 \ --service=helloservice.endpoints.projectx.cloud.goog \ --rollout_strategy=managed \ --http_port=8087 \ --backend=grpc://localhost:8089 \ --service_account_key=/esp/hello.json --non_gcp

Basically i added the non_gcp flag to the example what i found on the page: https://cloud.google.com/endpoints/docs/openapi/running-esp-localdev Note that the documentation doesnt say its needed, it just states that v1 and v2 works the same in this regards, however the v2 does not work without the --non_gcp. Just a remark from my side, maybe add to the doc.

After starting it like above and sending the same json to v2 as i did with v1 i see the following log in backend:

ápr. 21, 2022 11:14:09 DE. io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2FrameLogger logSettings FINE: [id: 0x25d37ba8, L:/[0:0:0:0:0:0:0:1%0]:8089 - R:/[0:0:0:0:0:0:0:1%0]:50958] OUTBOUND SETTINGS: ack=false settings={MAX_CONCURRENT_STREAMS=2147483647, INITIAL_WINDOW_SIZE=1048576, MAX_HEADER_LIST_SIZE=8192} ápr. 21, 2022 11:14:09 DE. io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2FrameLogger logWindowsUpdate FINE: [id: 0x25d37ba8, L:/[0:0:0:0:0:0:0:1%0]:8089 - R:/[0:0:0:0:0:0:0:1%0]:50958] OUTBOUND WINDOW_UPDATE: streamId=0 windowSizeIncrement=983041 ápr. 21, 2022 11:14:09 DE. io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2FrameLogger logSettings FINE: [id: 0x25d37ba8, L:/[0:0:0:0:0:0:0:1%0]:8089 - R:/[0:0:0:0:0:0:0:1%0]:50958] INBOUND SETTINGS: ack=false settings={HEADER_TABLE_SIZE=4096, ENABLE_PUSH=0, MAX_CONCURRENT_STREAMS=2147483647, INITIAL_WINDOW_SIZE=268435456,=0} ápr. 21, 2022 11:14:09 DE. io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2FrameLogger logSettingsAck FINE: [id: 0x25d37ba8, L:/[0:0:0:0:0:0:0:1%0]:8089 - R:/[0:0:0:0:0:0:0:1%0]:50958] OUTBOUND SETTINGS: ack=true ápr. 21, 2022 11:14:09 DE. io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2FrameLogger logWindowsUpdate FINE: [id: 0x25d37ba8, L:/[0:0:0:0:0:0:0:1%0]:8089 - R:/[0:0:0:0:0:0:0:1%0]:50958] INBOUND WINDOW_UPDATE: streamId=0 windowSizeIncrement=268369921 ápr. 21, 2022 11:14:09 DE. io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2FrameLogger logHeaders FINE: [id: 0x25d37ba8, L:/[0:0:0:0:0:0:0:1%0]:8089 - R:/[0:0:0:0:0:0:0:1%0]:50958] INBOUND HEADERS: streamId=1 headers=GrpcHttp2RequestHeaders[:path: /helloservice/hello, :authority: localhost:8087, :method: POST, :scheme: http, x-api-key: AIzaSyCvhWZ2ixEhuBV96VcOiXPUFYx7WTYCV1Y, content-type: application/json, user-agent: PostmanRuntime/7.29.0, accept: /, postman-token: 2bc9b280-e18a-485c-aeba-80944121cbca, accept-encoding: gzip, deflate, br, content-length: 45, x-forwarded-proto: http, x-request-id: 3b613dd4-3d38-4198-b6de-63f9b438905e, x-endpoint-api-consumer-type: PROJECT, x-endpoint-api-consumer-number: 247678761469] padding=0 endStream=false ápr. 21, 2022 11:14:09 DE. io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2FrameLogger logHeaders FINE: [id: 0x25d37ba8, L:/[0:0:0:0:0:0:0:1%0]:8089 - R:/[0:0:0:0:0:0:0:1%0]:50958] OUTBOUND HEADERS: streamId=1 headers=DefaultHttp2Headers[:status: 415, content-type: text/plain; charset=utf-8, grpc-status: 13, grpc-message: Content-Type 'application/json' is not supported] padding=0 endStream=false ápr. 21, 2022 11:14:09 DE. io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2FrameLogger logData FINE: [id: 0x25d37ba8, L:/[0:0:0:0:0:0:0:1%0]:8089 - R:/[0:0:0:0:0:0:0:1%0]:50958] INBOUND DATA: streamId=1 padding=0 endStream=true length=45 bytes=7b2275736572223a7b2266697273744e616d65223a20226161222c226c6173744e616d65223a20226262227d7d ápr. 21, 2022 11:14:09 DE. io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2FrameLogger logPing FINE: [id: 0x25d37ba8, L:/[0:0:0:0:0:0:0:1%0]:8089 - R:/[0:0:0:0:0:0:0:1%0]:50958] OUTBOUND PING: ack=false bytes=1234 ápr. 21, 2022 11:14:09 DE. io.grpc.netty.shaded.io.grpc.netty.NettyServerHandler onDataRead WARNING: Exception in onDataRead() java.lang.NullPointerException: Cannot invoke "io.grpc.netty.shaded.io.grpc.netty.NettyServerStream$TransportState.tag()" because "stream" is null at io.grpc.netty.shaded.io.grpc.netty.NettyServerHandler.onDataRead(NettyServerHandler.java:506) at io.grpc.netty.shaded.io.grpc.netty.NettyServerHandler.access$900(NettyServerHandler.java:106) at io.grpc.netty.shaded.io.grpc.netty.NettyServerHandler$FrameListener.onDataRead(NettyServerHandler.java:840) at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onDataRead(DefaultHttp2ConnectionDecoder.java:307) at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$PrefaceFrameListener.onDataRead(DefaultHttp2ConnectionDecoder.java:691) at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onDataRead(Http2InboundFrameLogger.java:48) at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.readDataFrame(DefaultHttp2FrameReader.java:415) at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:250) at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:159) at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41) at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:173) at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:378) at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.decode(Http2ConnectionHandler.java:242) at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:438) at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795) at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:833)

ápr. 21, 2022 11:14:09 DE. io.grpc.netty.shaded.io.grpc.netty.NettyServerHandler onStreamError WARNING: Stream Error io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2Exception$StreamException: Cannot invoke "io.grpc.netty.shaded.io.grpc.netty.NettyServerStream$TransportState.tag()" because "stream" is null at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2Exception.streamError(Http2Exception.java:172) at io.grpc.netty.shaded.io.grpc.netty.NettyServerHandler.newStreamException(NettyServerHandler.java:817) at io.grpc.netty.shaded.io.grpc.netty.NettyServerHandler.onDataRead(NettyServerHandler.java:515) at io.grpc.netty.shaded.io.grpc.netty.NettyServerHandler.access$900(NettyServerHandler.java:106) at io.grpc.netty.shaded.io.grpc.netty.NettyServerHandler$FrameListener.onDataRead(NettyServerHandler.java:840) at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onDataRead(DefaultHttp2ConnectionDecoder.java:307) at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$PrefaceFrameListener.onDataRead(DefaultHttp2ConnectionDecoder.java:691) at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onDataRead(Http2InboundFrameLogger.java:48) at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.readDataFrame(DefaultHttp2FrameReader.java:415) at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:250) at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:159) at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41) at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:173) at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:378) at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.decode(Http2ConnectionHandler.java:242) at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:438) at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795) at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: java.lang.NullPointerException: Cannot invoke "io.grpc.netty.shaded.io.grpc.netty.NettyServerStream$TransportState.tag()" because "stream" is null at io.grpc.netty.shaded.io.grpc.netty.NettyServerHandler.onDataRead(NettyServerHandler.java:506) ... 30 more

ápr. 21, 2022 11:14:09 DE. io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2FrameLogger logRstStream FINE: [id: 0x25d37ba8, L:/[0:0:0:0:0:0:0:1%0]:8089 - R:/[0:0:0:0:0:0:0:1%0]:50958] OUTBOUND RST_STREAM: streamId=1 errorCode=2 ápr. 21, 2022 11:14:09 DE. io.grpc.netty.shaded.io.grpc.netty.NettyServerHandler onStreamError FINE: Stream Error io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2Exception$StreamException: Stream closed before write could take place at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2Exception.streamError(Http2Exception.java:172) at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2RemoteFlowController$FlowState.cancel(DefaultHttp2RemoteFlowController.java:481) at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2RemoteFlowController$1.onStreamClosed(DefaultHttp2RemoteFlowController.java:105) at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2Connection.notifyClosed(DefaultHttp2Connection.java:357) at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.removeFromActiveStreams(DefaultHttp2Connection.java:1007) at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.deactivate(DefaultHttp2Connection.java:963) at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultStream.close(DefaultHttp2Connection.java:515) at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultStream.close(DefaultHttp2Connection.java:521) at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.closeStream(Http2ConnectionHandler.java:613) at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.processRstStreamWriteResult(Http2ConnectionHandler.java:894) at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.access$1000(Http2ConnectionHandler.java:66) at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$4.operationComplete(Http2ConnectionHandler.java:809) at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$4.operationComplete(Http2ConnectionHandler.java:806) at io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578) at io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:552) at io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491) at io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616) at io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:605) at io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104) at io.grpc.netty.shaded.io.netty.util.internal.PromiseNotificationUtil.trySuccess(PromiseNotificationUtil.java:48) at io.grpc.netty.shaded.io.netty.channel.ChannelOutboundBuffer.safeSuccess(ChannelOutboundBuffer.java:717) at io.grpc.netty.shaded.io.netty.channel.ChannelOutboundBuffer.remove(ChannelOutboundBuffer.java:272) at io.grpc.netty.shaded.io.netty.channel.ChannelOutboundBuffer.removeBytes(ChannelOutboundBuffer.java:352) at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollStreamChannel.writeBytesMultiple(AbstractEpollStreamChannel.java:305) at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollStreamChannel.doWriteMultiple(AbstractEpollStreamChannel.java:510) at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollStreamChannel.doWrite(AbstractEpollStreamChannel.java:422) at io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:949) at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.flush0(AbstractEpollChannel.java:557) at io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:913) at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1372) at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750) at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:742) at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:728) at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.onError(Http2ConnectionHandler.java:643) at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:380) at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.decode(Http2ConnectionHandler.java:242) at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:438) at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795) at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:833)

ápr. 21, 2022 11:14:09 DE. io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2FrameLogger logSettingsAck FINE: [id: 0x25d37ba8, L:/[0:0:0:0:0:0:0:1%0]:8089 - R:/[0:0:0:0:0:0:0:1%0]:50958] INBOUND SETTINGS: ack=true ápr. 21, 2022 11:14:09 DE. io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2FrameLogger logPingAck FINE: [id: 0x25d37ba8, L:/[0:0:0:0:0:0:0:1%0]:8089 - R:/[0:0:0:0:0:0:0:1%0]:50958] INBOUND PING: ack=true bytes=1234 ápr. 21, 2022 11:14:09 DE. io.grpc.netty.shaded.io.grpc.netty.NettyServerHandler$FrameListener onPingAckRead FINE: Window: 1048576

Do i need some extra setting to get it work or is this some kind of a bug? As said i tested with libs 1.41.0 and 1.44.0, and it did not work with either of them.

[qiwzhang]

Since you are running in localhost, it doesn't GCP metadata server, yes, "--non_gcp" is required.

Could you raise this issue in ESPv2 repo with following info:

• add --enable_debug flag in ESPv2 container arguments
• get the debug log from the ESPv2 container

[qiwzhang]

BTW, ESPv2 is based on Envoy, it should NOT have this issue. It is currently supported, if there are any bugs, we will fix it.

[gombasp]

Hi,

any news on this topic? Earlier you wrote about espv2 that "if there are any bugs, we will fix it". Is this not the case for espv1?? We use it still extensively in our projects so a bugfix would be nice to have. Thanks,

Peter

[qiwzhang]

Unfortunately, many ESPv1 release components, e.g. CI/CT are broken. Most likely they are outdated. It will take some effort to restore them. We don't have resource to make new releases for ESPv1 for now.

[gombasp]

I tested espv2 and it seems to be working without the above mentioned issue. So we will consider the upgrade. Thank you.