shamanthchandra-yb
commented
1 year ago Also seen here: http://stress.dev.yugabyte.com/stress_test/334b4b66-305b-4ca7-8401-a39354a1919a
Apart from crash and cores, in connector log
2023-04-04 04:40:01,389 ERROR || WorkerSourceTask{id=ybconnector_cdc_ad02d2_test_cdc_7e06d9_test_cdc_07701c-4} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted [org.apache.kafka.connect.runtime.WorkerTask]
org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped.
at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:50)
at io.debezium.connector.yugabytedb.YugabyteDBStreamingChangeEventSource.execute(YugabyteDBStreamingChangeEventSource.java:144)
at io.debezium.connector.yugabytedb.YugabyteDBStreamingChangeEventSource.execute(YugabyteDBStreamingChangeEventSource.java:47)
at io.debezium.pipeline.ChangeEventSourceCoordinator.streamEvents(ChangeEventSourceCoordinator.java:174)
at io.debezium.connector.yugabytedb.YugabyteDBChangeEventSourceCoordinator.executeChangeEventSources(YugabyteDBChangeEventSourceCoordinator.java:138)
at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:109)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.yb.client.NonRecoverableException: Time out: YRpc(method=GetChanges, service=yb.cdc.CDCService, tablet=eed57e66d46d495daebffb765de9e9d1, attempt=23, maxAttempts=100, maxTimeoutMs=600000, elapsedTimeMs=600001). Server[23f50084c7d54e80870df4f461cf6093] TIMED_OUT[code 14]: GetChanges RPC (request call id 16556) to 172.151.19.160:9100 timed out after 0.252s
at org.yb.client.AsyncYBClient.tooManyAttemptsOrTimeout(AsyncYBClient.java:2127)
at org.yb.client.AsyncYBClient.delayedSendRpcToTablet(AsyncYBClient.java:2335)
at org.yb.client.AsyncYBClient.handleRetryableError(AsyncYBClient.java:2315)
at org.yb.client.TabletClient.dispatchCDCErrorOrReturnException(TabletClient.java:518)
at org.yb.client.TabletClient.decode(TabletClient.java:437)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510)
at io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:366)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
... 3 more
Jira Link: DB-6053
Description
http://stress.dev.yugabyte.com/stress_test/3c884bbf-459e-4ade-98f8-3ffa6fab6566
Source connector version
1.9.5.y.18
Connector configuration
add connector connector_name='ybconnector_cdc_3f369f_test_cdc_030e90_test_cdc_276c3f' stream_id='bf29c867452a468aac48e8c6aeef2c69' db_name='cdc_3f369f' connector_host='172.151.21.65' table_list=['test_cdc_030e90', 'test_cdc_276c3f'] {'name': 'ybconnector_cdc_3f369f_test_cdc_030e90_test_cdc_276c3f', 'config': {'connector.class': 'io.debezium.connector.yugabytedb.YugabyteDBConnector', 'database.hostname': '172.151.25.55', 'database.master.addresses': '172.151.27.64:7100,172.151.25.55:7100,172.151.21.32:7100', 'database.port': 5433, 'database.masterhost': '172.151.25.55', 'database.masterport': '7100', 'database.user': 'yugabyte', 'database.password': 'yugabyte', 'database.dbname': 'cdc_3f369f', 'database.server.name': 'db_cdc', 'database.streamid': 'bf29c867452a468aac48e8c6aeef2c69', 'snapshot.mode': 'never', 'admin.operation.timeout.ms': 600000, 'socket.read.timeout.ms': 600000, 'max.connector.retries': '10', 'operation.timeout.ms': 600000, 'topic.creation.default.compression.type': 'lz4', 'topic.creation.default.cleanup.policy': 'delete', 'topic.creation.default.partitions': 2, 'topic.creation.default.replication.factor': '1', 'tasks.max': '5', 'table.include.list': 'public.test_cdc_030e90,public.test_cdc_276c3f'}}YugabyteDB version
2.17.4.0-b50
Warning: Please confirm that this issue does not contain any sensitive information