Constantly getting Http2ConnectionTerminatingException using KCL 2.2.7

[shaozhaoyang]

Hi,

I'm using KCL and enhanced fan out to build a consumer. The consumer does some heavy lift and sometimes it could take over 30 seconds to process a batch of records. I found that when the record batch is large, I'm constantly getting Http2ConnectionTerminatingException. Just wonder if there is any configuration I can do to get rid of this exception.

full stack trace: io.netty.channel.DefaultChannelPipeline: An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception. software.amazon.awssdk.http.nio.netty.internal.http2.Http2ConnectionTerminatingException: An I/O error occurred on an associated Http2 stream [id: 0x00ddcc50, L:/192.168.15.97:61772 ! R:kinesis.us-west-2.amazonaws.com/34.223.45.98:443](H2 - 5) at software.amazon.awssdk.http.nio.netty.internal.http2.Http2StreamExceptionHandler.exceptionCaught(Http2StreamExceptionHandler.java:50) [AwsJavaSdk-HttpClient-NettyNioClient-2.0.jar:?]

(aws-java-sdk-NettyEventLoop-1-1) io.netty.channel.DefaultChannelPipeline: An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception. software.amazon.awssdk.http.nio.netty.internal.http2.Http2ConnectionTerminatingException: An I/O error occurred on an associated Http2 stream [id: 0x00ddcc50, L:/192.168.15.97:61772 ! R:kinesis.us-west-2.amazonaws.com/34.223.45.98:443](H2 - 5) at software.amazon.awssdk.http.nio.netty.internal.http2.Http2StreamExceptionHandler.exceptionCaught(Http2StreamExceptionHandler.java:50) [AwsJavaSdk-HttpClient-NettyNioClient-2.0.jar:?] at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:302) [netty-all-4.1.jar:4.1.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:281) [netty-all-4.1.jar:4.1.48.Final] at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:273) [netty-all-4.1.jar:4.1.48.Final] at software.amazon.awssdk.http.nio.netty.internal.UnusedChannelExceptionHandler.exceptionCaught(UnusedChannelExceptionHandler.java:52) [AwsJavaSdk-HttpClient-NettyNioClient-2.0.jar:?] at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:302) [netty-all-4.1.jar:4.1.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:281) [netty-all-4.1.jar:4.1.48.Final] at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:273) [netty-all-4.1.jar:4.1.48.Final] at io.netty.handler.timeout.ReadTimeoutHandler.readTimedOut(ReadTimeoutHandler.java:98) [netty-all-4.1.jar:4.1.48.Final] at io.netty.handler.timeout.ReadTimeoutHandler.channelIdle(ReadTimeoutHandler.java:90) [netty-all-4.1.jar:4.1.48.Final] at io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) [netty-all-4.1.jar:4.1.48.Final] at io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) [netty-all-4.1.jar:4.1.48.Final] at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) [netty-all-4.1.jar:4.1.48.Final] at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) [netty-all-4.1.jar:4.1.48.Final] at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [netty-all-4.1.jar:4.1.48.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [netty-all-4.1.jar:4.1.48.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) [netty-all-4.1.jar:4.1.48.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-all-4.1.jar:4.1.48.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-all-4.1.jar:4.1.48.Final] at java.lang.Thread.run(Thread.java:834) [?:

[ashwing]

Is this affecting your propagation delay of the messages or the millisbehindlatest of the shards?

[Bas83]

I have the same issue, last 24 hours I've seen this message 12 times, before upgrading from 2.2.5 to 2.3.3 it never happened.

[super9user]

Any updates? I'm seeing the same exception

[nocmk2]

the same

[itsmesrds]

+1. I'm also getting the same exception.

[MihaiBogdanEugen]

We're having the same issue, any updates?

[nylund]

We also see this issue. Restarting the consumers does not help. KCL: 2.3.10 SDK: 2.17.101

[gsharon2022]

wow, I am using latest version 2.4.1 and also seeing this issue... looks like this is open since Nov/2020 and no fix or solution.

[leicoop]

Got the same exception running on version 2.4.2, so will it casuse any data lose or some other potential data integrity issues