search

JoeShi / kafka-realtime-data-processing

MIT License
0 stars 2 forks source link

Kinesis Connect 退出报错 #3

Open JoeShi opened 5 years ago

JoeShi commented 5 years ago

启动 kinesis connect 之后，如果使用 control + c 退出的话，有可能会报错；这个现象在 kafka-connect-datagen 不存在

担心 Kinesis Connect 退出的时候是不是会导致数据丢失

error message

^C[2019-11-10 14:47:10,514] INFO Kafka Connect stopping (org.apache.kafka.connect.runtime.Connect:66)
[2019-11-10 14:47:10,558] INFO Stopping REST server (org.apache.kafka.connect.runtime.rest.RestServer:241)
[2019-11-10 14:47:10,567] INFO Stopped http_8083@72c927f1{HTTP/1.1,[http/1.1]}{0.0.0.0:8083} (org.eclipse.jetty.server.AbstractConnector:341)
[2019-11-10 14:47:10,568] INFO node0 Stopped scavenging (org.eclipse.jetty.server.session:167)
[2019-11-10 14:47:10,575] INFO REST server stopped (org.apache.kafka.connect.runtime.rest.RestServer:258)
[2019-11-10 14:47:10,575] INFO Herder stopping (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:98)
[2019-11-10 14:47:10,579] INFO Stopping task connect-kinesis-shiheng-0 (org.apache.kafka.connect.runtime.Worker:684)
[2019-11-10 14:47:10,583] WARN WorkerSinkTask{id=connect-kinesis-shiheng-0} Offset commit failed during close (org.apache.kafka.connect.runtime.WorkerSinkTask:381)
[2019-11-10 14:47:10,584] ERROR WorkerSinkTask{id=connect-kinesis-shiheng-0} Commit of offsets threw an unexpected exception for sequence number 19: null (org.apache.kafka.connect.runtime.WorkerSinkTask:259)
com.amazonaws.services.kinesis.producer.DaemonException: The child process has been shutdown and can no longer accept messages.
    at com.amazonaws.services.kinesis.producer.Daemon.add(Daemon.java:176)
    at com.amazonaws.services.kinesis.producer.KinesisProducer.flush(KinesisProducer.java:784)
    at com.amazonaws.services.kinesis.producer.KinesisProducer.flush(KinesisProducer.java:804)
    at com.amazonaws.services.kinesis.producer.KinesisProducer.flushSync(KinesisProducer.java:830)
    at com.amazon.kinesis.kafka.AmazonKinesisSinkTask.lambda$flush$0(AmazonKinesisSinkTask.java:101)
    at java.util.HashMap$Values.forEach(HashMap.java:981)
    at com.amazon.kinesis.kafka.AmazonKinesisSinkTask.flush(AmazonKinesisSinkTask.java:99)
    at org.apache.kafka.connect.sink.SinkTask.preCommit(SinkTask.java:125)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.commitOffsets(WorkerSinkTask.java:378)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.closePartitions(WorkerSinkTask.java:590)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:196)
    at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:177)
    at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:227)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
[2019-11-10 14:47:11,475] WARN Exception during updateCredentials (com.amazonaws.services.kinesis.producer.Daemon:323)
java.lang.InterruptedException: sleep interrupted
    at java.lang.Thread.sleep(Native Method)
    at com.amazonaws.services.kinesis.producer.Daemon$5.run(Daemon.java:321)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
[2019-11-10 14:47:11,495] WARN Exception during updateCredentials (com.amazonaws.services.kinesis.producer.Daemon:323)
java.lang.InterruptedException: sleep interrupted
    at java.lang.Thread.sleep(Native Method)
    at com.amazonaws.services.kinesis.producer.Daemon$5.run(Daemon.java:321)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
[2019-11-10 14:47:11,499] ERROR Error in child process (com.amazonaws.services.kinesis.producer.KinesisProducer:152)
java.lang.RuntimeException: EOF reached during read
    at com.amazonaws.services.kinesis.producer.Daemon.fatalError(Daemon.java:521)
    at com.amazonaws.services.kinesis.producer.Daemon.fatalError(Daemon.java:497)
    at com.amazonaws.services.kinesis.producer.Daemon.fatalError(Daemon.java:493)
    at com.amazonaws.services.kinesis.producer.Daemon.readSome(Daemon.java:542)
    at com.amazonaws.services.kinesis.producer.Daemon.receiveMessage(Daemon.java:246)
    at com.amazonaws.services.kinesis.producer.Daemon.access$500(Daemon.java:63)
    at com.amazonaws.services.kinesis.producer.Daemon$3.run(Daemon.java:301)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
[2019-11-10 14:47:11,501] INFO Asking for trace (com.amazonaws.services.kinesis.producer.Daemon:421)
[2019-11-10 14:47:11,499] WARN Exception during updateCredentials (com.amazonaws.services.kinesis.producer.Daemon:323)
java.lang.InterruptedException: sleep interrupted
    at java.lang.Thread.sleep(Native Method)
    at com.amazonaws.services.kinesis.producer.Daemon$5.run(Daemon.java:321)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
[2019-11-10 14:47:11,523] INFO [Consumer clientId=connector-consumer-connect-kinesis-shiheng-0, groupId=connect-connect-kinesis-shiheng] Member connector-consumer-connect-kinesis-shiheng-0-16fe1a95-f7b7-475c-a23e-e88339b46a71 sending LeaveGroup request to coordinator ip-172-31-31-118.cn-northwest-1.compute.internal:9092 (id: 2147483645 rack: null) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:879)
[2019-11-10 14:47:11,529] INFO Asking for trace (com.amazonaws.services.kinesis.producer.Daemon:421)
[2019-11-10 14:47:11,532] INFO Asking for trace (com.amazonaws.services.kinesis.producer.Daemon:421)
[2019-11-10 14:47:11,534] INFO [2019-11-10 14:47:11.526577] [0x00001276][0x00007f40322ad7c0] [info] [logging.cc:83] Set boost log level to info (com.amazonaws.services.kinesis.producer.LogInputStreamReader:59)
[2019-11-10 14:47:11,535] INFO [2019-11-10 14:47:11.526643] [0x00001276][0x00007f40322ad7c0] [info] [logging.cc:170] Set AWS Log Level to WARN (com.amazonaws.services.kinesis.producer.LogInputStreamReader:59)
[2019-11-10 14:47:11,538] INFO [2019-11-10 14:47:11.534070] [0x00001276][0x00007f40322ad7c0] [info] [main.cc:346] Setting CA path to /tmp/amazon-kinesis-producer-native-binaries/cacerts (com.amazonaws.services.kinesis.producer.LogInputStreamReader:59)
[2019-11-10 14:47:11,538] INFO [2019-11-10 14:47:11.534088] [0x00001276][0x00007f40322ad7c0] [info] [main.cc:382] Starting up main producer (com.amazonaws.services.kinesis.producer.LogInputStreamReader:59)
[2019-11-10 14:47:11,538] INFO [2019-11-10 14:47:11.534107] [0x00001276][0x00007f40322ad7c0] [info] [kinesis_producer.cc:101] Using Region: cn-northwest-1 (com.amazonaws.services.kinesis.producer.LogInputStreamReader:59)
[2019-11-10 14:47:11,539] INFO [2019-11-10 14:47:11.534115] [0x00001276][0x00007f40322ad7c0] [info] [kinesis_producer.cc:120] Using per request threading model. (com.amazonaws.services.kinesis.producer.LogInputStreamReader:59)
[2019-11-10 14:47:11,539] INFO [2019-11-10 14:47:11.534124] [0x00001276][0x00007f40322ad7c0] [info] [kinesis_producer.cc:171] Using Kinesis endpoint kinesis.cn-northwest-1.amazonaws.com.cn:443 (com.amazonaws.services.kinesis.producer.LogInputStreamReader:59)
[2019-11-10 14:47:11,539] INFO [2019-11-10 14:47:11.534148] [0x00001276][0x00007f40322ad7c0] [info] [kinesis_producer.cc:101] Using Region: cn-northwest-1 (com.amazonaws.services.kinesis.producer.LogInputStreamReader:59)
[2019-11-10 14:47:11,539] INFO [2019-11-10 14:47:11.534155] [0x00001276][0x00007f40322ad7c0] [info] [kinesis_producer.cc:120] Using per request threading model. (com.amazonaws.services.kinesis.producer.LogInputStreamReader:59)
[2019-11-10 14:47:11,539] INFO [2019-11-10 14:47:11.534162] [0x00001276][0x00007f40322ad7c0] [info] [kinesis_producer.cc:186] Using CloudWatch endpoint monitoring.cn-northwest-1.amazonaws.com.cn:443 (com.amazonaws.services.kinesis.producer.LogInputStreamReader:59)
[2019-11-10 14:47:11,539] INFO [2019-11-10 14:47:11.534216] [0x00001276][0x00007f40322ad7c0] [info] [main.cc:393] Entering join (com.amazonaws.services.kinesis.producer.LogInputStreamReader:59)
[2019-11-10 14:47:11,551] INFO Stopping connector connect-kinesis-shiheng (org.apache.kafka.connect.runtime.Worker:353)
[2019-11-10 14:47:11,555] INFO Stopped connector connect-kinesis-shiheng (org.apache.kafka.connect.runtime.Worker:369)
[2019-11-10 14:47:11,555] INFO Worker stopping (org.apache.kafka.connect.runtime.Worker:194)
[2019-11-10 14:47:11,569] INFO Stopped FileOffsetBackingStore (org.apache.kafka.connect.storage.FileOffsetBackingStore:66)
[2019-11-10 14:47:11,570] INFO Worker stopped (org.apache.kafka.connect.runtime.Worker:215)
[2019-11-10 14:47:11,570] INFO Herder stopped (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:115)
[2019-11-10 14:47:11,570] INFO Kafka Connect stopped (org.apache.kafka.connect.runtime.Connect:71)
[2019-11-10 14:47:11,571] INFO [2019-11-10 14:47:11.553595] [0x00001281][0x00007f82883817c0] [info] [logging.cc:83] Set boost log level to info (com.amazonaws.services.kinesis.producer.LogInputStreamReader:59)
[2019-11-10 14:47:11,572] INFO [2019-11-10 14:47:11.553670] [0x00001281][0x00007f82883817c0] [info] [logging.cc:170] Set AWS Log Level to WARN (com.amazonaws.services.kinesis.producer.LogInputStreamReader:59)
[2019-11-10 14:47:11,572] INFO [2019-11-10 14:47:11.558380] [0x00001281][0x00007f82883817c0] [info] [main.cc:346] Setting CA path to /tmp/amazon-kinesis-producer-native-binaries/cacerts (com.amazonaws.services.kinesis.producer.LogInputStreamReader:59)
[2019-11-10 14:47:11,572] INFO [2019-11-10 14:47:11.558397] [0x00001281][0x00007f82883817c0] [info] [main.cc:382] Starting up main producer (com.amazonaws.services.kinesis.producer.LogInputStreamReader:59)
[2019-11-10 14:47:11,572] INFO [2019-11-10 14:47:11.558412] [0x00001281][0x00007f82883817c0] [info] [kinesis_producer.cc:101] Using Region: cn-northwest-1 (com.amazonaws.services.kinesis.producer.LogInputStreamReader:59)
[2019-11-10 14:47:11,572] INFO [2019-11-10 14:47:11.558419] [0x00001281][0x00007f82883817c0] [info] [kinesis_producer.cc:120] Using per request threading model. (com.amazonaws.services.kinesis.producer.LogInputStreamReader:59)
[2019-11-10 14:47:11,572] INFO [2019-11-10 14:47:11.558428] [0x00001281][0x00007f82883817c0] [info] [kinesis_producer.cc:171] Using Kinesis endpoint kinesis.cn-northwest-1.amazonaws.com.cn:443 (com.amazonaws.services.kinesis.producer.LogInputStreamReader:59)
[2019-11-10 14:47:11,572] INFO [2019-11-10 14:47:11.558450] [0x00001281][0x00007f82883817c0] [info] [kinesis_producer.cc:101] Using Region: cn-northwest-1 (com.amazonaws.services.kinesis.producer.LogInputStreamReader:59)
[2019-11-10 14:47:11,572] INFO [2019-11-10 14:47:11.558456] [0x00001281][0x00007f82883817c0] [info] [kinesis_producer.cc:120] Using per request threading model. (com.amazonaws.services.kinesis.producer.LogInputStreamReader:59)
[2019-11-10 14:47:11,572] INFO [2019-11-10 14:47:11.558463] [0x00001281][0x00007f82883817c0] [info] [kinesis_producer.cc:186] Using CloudWatch endpoint monitoring.cn-northwest-1.amazonaws.com.cn:443 (com.amazonaws.services.kinesis.producer.LogInputStreamReader:59)
[2019-11-10 14:47:11,573] INFO [2019-11-10 14:47:11.558507] [0x00001281][0x00007f82883817c0] [info] [main.cc:393] Entering join (com.amazonaws.services.kinesis.producer.LogInputStreamReader:59)
[2019-11-10 14:47:11,575] INFO [2019-11-10 14:47:11.550443] [0x00001280][0x00007fb73a53c7c0] [info] [logging.cc:83] Set boost log level to info (com.amazonaws.services.kinesis.producer.LogInputStreamReader:59)
[2019-11-10 14:47:11,575] INFO [2019-11-10 14:47:11.550505] [0x00001280][0x00007fb73a53c7c0] [info] [logging.cc:170] Set AWS Log Level to WARN (com.amazonaws.services.kinesis.producer.LogInputStreamReader:59)
[2019-11-10 14:47:11,575] INFO [2019-11-10 14:47:11.553850] [0x00001280][0x00007fb73a53c7c0] [info] [main.cc:346] Setting CA path to /tmp/amazon-kinesis-producer-native-binaries/cacerts (com.amazonaws.services.kinesis.producer.LogInputStreamReader:59)
[2019-11-10 14:47:11,575] INFO [2019-11-10 14:47:11.553867] [0x00001280][0x00007fb73a53c7c0] [info] [main.cc:382] Starting up main producer (com.amazonaws.services.kinesis.producer.LogInputStreamReader:59)
[2019-11-10 14:47:11,575] INFO [2019-11-10 14:47:11.553883] [0x00001280][0x00007fb73a53c7c0] [info] [kinesis_producer.cc:101] Using Region: cn-northwest-1 (com.amazonaws.services.kinesis.producer.LogInputStreamReader:59)
[2019-11-10 14:47:11,575] INFO [2019-11-10 14:47:11.553891] [0x00001280][0x00007fb73a53c7c0] [info] [kinesis_producer.cc:120] Using per request threading model. (com.amazonaws.services.kinesis.producer.LogInputStreamReader:59)
[2019-11-10 14:47:11,575] INFO [2019-11-10 14:47:11.553900] [0x00001280][0x00007fb73a53c7c0] [info] [kinesis_producer.cc:171] Using Kinesis endpoint kinesis.cn-northwest-1.amazonaws.com.cn:443 (com.amazonaws.services.kinesis.producer.LogInputStreamReader:59)
[2019-11-10 14:47:11,576] INFO [2019-11-10 14:47:11.553924] [0x00001280][0x00007fb73a53c7c0] [info] [kinesis_producer.cc:101] Using Region: cn-northwest-1 (com.amazonaws.services.kinesis.producer.LogInputStreamReader:59)
[2019-11-10 14:47:11,576] INFO [2019-11-10 14:47:11.553931] [0x00001280][0x00007fb73a53c7c0] [info] [kinesis_producer.cc:120] Using per request threading model. (com.amazonaws.services.kinesis.producer.LogInputStreamReader:59)
[2019-11-10 14:47:11,576] INFO [2019-11-10 14:47:11.553938] [0x00001280][0x00007fb73a53c7c0] [info] [kinesis_producer.cc:186] Using CloudWatch endpoint monitoring.cn-northwest-1.amazonaws.com.cn:443 (com.amazonaws.services.kinesis.producer.LogInputStreamReader:59)
[2019-11-10 14:47:11,576] INFO [2019-11-10 14:47:11.553985] [0x00001280][0x00007fb73a53c7c0] [info] [main.cc:393] Entering join (com.amazonaws.services.kinesis.producer.LogInputStreamReader:59)