search

confluentinc / kafka-connect-hdfs

Kafka Connect HDFS connector
Other
12 stars 397 forks source link

Problem renewing Kerberos ticket #225

Open talnicolas opened 7 years ago

talnicolas commented 7 years ago

Hi,

I am using the following docker image to run my workers: confluentinc/cp-kafka-connect:3.2.2.

I looked at the existing issues and it does look a lot like 178, but it seems to me that this problem would have been fixed in this version.

So what happens is that my worker is running fine until the Kerberos ticket expires. Once it is not valid anymore, the worker is not able to renew the ticket, stop working and just ouptut a bunch of those in the logs:

[2017-08-31 23:14:14,462] WARN Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] (org.apache.hadoop.ipc.Client)
[2017-08-31 23:14:14,463] INFO Exception while invoking renewLease of class ClientNamenodeProtocolTranslatorPB over namenode-fqdn/namenode-ip:8020 after 10 fail over attempts. Trying to fail over immediately. (org.apache.hadoop.io.retry.RetryInvocationHandler)
java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "worker-fqdn/worker-ip"; destination host is: "namenode-fqdn":8020; 
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:776)
        at org.apache.hadoop.ipc.Client.call(Client.java:1479)
        at org.apache.hadoop.ipc.Client.call(Client.java:1412)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
        at com.sun.proxy.$Proxy49.renewLease(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:590)
        at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com.sun.proxy.$Proxy50.renewLease(Unknown Source)
        at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:892)
        at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:423)
        at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:448)
        at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
        at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
        at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:687)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
        at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:650)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:737)
        at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:375)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1528)
        at org.apache.hadoop.ipc.Client.call(Client.java:1451)
        ... 16 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
        at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
        at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:414)
        at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:560)
        at org.apache.hadoop.ipc.Client$Connection.access$1900(Client.java:375)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:729)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:725)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:725)
        ... 19 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
        at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
        at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
        at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
        at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
        at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
        ... 28 more

In my configuration I do use the FQDN for hive.metastore.uris and hdfs.namenode.principal, and I use the HA url for hdfs.url (hdfs://ha-url/).

What if find concerning is that there is no reference to the Connect code in the Stacktrace.

koertkuipers commented 7 years ago

i am seeing the same issue, however it seems the re-login code did work correctly in the logs the process for initiating logout and initiating re-login with kerberos seems OK, and i see a successful ticket from TGT is logged, but after this i get similar errors to above nonetheless. i am using version 3.3.0

koertkuipers commented 7 years ago

out setup is HA hdfs, we dont use hive. below are logs from one connect worker. the process runs fine for about 3 days and then we start getting these errors.

[2017-10-10 18:39:59,551] INFO Initiating logout for kafka/box12.ourcompany.com@OURCOMPANY.COM (org.apache.kafka.common.security.kerberos.KerberosLogin:355)
[2017-10-10 18:39:59,552] INFO Initiating re-login for kafka/box12.ourcompany.com@OURCOMPANY.COM (org.apache.kafka.common.security.kerberos.KerberosLogin:365)
[2017-10-10 18:39:59,557] INFO [Principal=kafka/box12.ourcompany.com@OURCOMPANY.COM]: TGT valid starting at: Tue Oct 10 18:39:59 EDT 2017 (org.apache.kafka.common.security.kerberos.KerberosLogin:305)
[2017-10-10 18:39:59,558] INFO [Principal=kafka/box12.ourcompany.com@OURCOMPANY.COM]: TGT expires: Wed Oct 11 18:39:59 EDT 2017 (org.apache.kafka.common.security.kerberos.KerberosLogin:306)
[2017-10-10 18:39:59,558] INFO [Principal=kafka/box12.ourcompany.com@OURCOMPANY.COM]: TGT refresh sleeping until: Wed Oct 11 13:56:12 EDT 2017 (org.apache.kafka.common.security.kerberos.KerberosLogin:188)
[2017-10-10 18:40:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1163: {twitter-0=OffsetAndMetadata{offset=751086, metadata=''}, test-0=OffsetAndMetadata{offset=4599695, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 18:41:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1164: {twitter-0=OffsetAndMetadata{offset=751117, metadata=''}, test-0=OffsetAndMetadata{offset=4599714, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 18:42:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1165: {twitter-0=OffsetAndMetadata{offset=751173, metadata=''}, test-0=OffsetAndMetadata{offset=4599738, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 18:43:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1166: {twitter-0=OffsetAndMetadata{offset=751205, metadata=''}, test-0=OffsetAndMetadata{offset=4599757, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 18:44:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1167: {twitter-0=OffsetAndMetadata{offset=751249, metadata=''}, test-0=OffsetAndMetadata{offset=4599779, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 18:45:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1168: {twitter-0=OffsetAndMetadata{offset=751314, metadata=''}, test-0=OffsetAndMetadata{offset=4599807, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 18:46:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1169: {twitter-0=OffsetAndMetadata{offset=751375, metadata=''}, test-0=OffsetAndMetadata{offset=4599832, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 18:47:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1170: {twitter-0=OffsetAndMetadata{offset=751441, metadata=''}, test-0=OffsetAndMetadata{offset=4599866, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 18:48:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1171: {twitter-0=OffsetAndMetadata{offset=751477, metadata=''}, test-0=OffsetAndMetadata{offset=4599887, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 18:49:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1172: {twitter-0=OffsetAndMetadata{offset=751507, metadata=''}, test-0=OffsetAndMetadata{offset=4599903, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 18:50:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1173: {twitter-0=OffsetAndMetadata{offset=751527, metadata=''}, test-0=OffsetAndMetadata{offset=4599914, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 18:51:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1174: {twitter-0=OffsetAndMetadata{offset=751575, metadata=''}, test-0=OffsetAndMetadata{offset=4599937, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 18:52:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1175: {twitter-0=OffsetAndMetadata{offset=751609, metadata=''}, test-0=OffsetAndMetadata{offset=4599951, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 18:53:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1176: {twitter-0=OffsetAndMetadata{offset=751627, metadata=''}, test-0=OffsetAndMetadata{offset=4599962, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 18:54:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1177: {twitter-0=OffsetAndMetadata{offset=751669, metadata=''}, test-0=OffsetAndMetadata{offset=4599986, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 18:55:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1178: {twitter-0=OffsetAndMetadata{offset=751715, metadata=''}, test-0=OffsetAndMetadata{offset=4600002, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 18:56:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1179: {twitter-0=OffsetAndMetadata{offset=751751, metadata=''}, test-0=OffsetAndMetadata{offset=4600018, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 18:57:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1180: {twitter-0=OffsetAndMetadata{offset=751795, metadata=''}, test-0=OffsetAndMetadata{offset=4600039, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 18:58:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1181: {twitter-0=OffsetAndMetadata{offset=751821, metadata=''}, test-0=OffsetAndMetadata{offset=4600055, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 18:59:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1182: {twitter-0=OffsetAndMetadata{offset=751857, metadata=''}, test-0=OffsetAndMetadata{offset=4600070, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:00:01,835] INFO Starting commit and rotation for topic partition twitter-0 with start offsets {year=2017/month=10/day=10/hour=23/=751897, year=2017/month=10/day=10/hour=22/=749458} and end offsets {year=2017/month=10/day=10/hour=07/=708077, year=2017/month=10/day=10/hour=18/=741506, year=2017/month=10/day=10/hour=06/=705988, year=2017/month=10/day=10/hour=17/=738750, year=2017/month=10/day=10/hour=09/=712088, year=2017/month=10/day=10/hour=08/=710134, year=2017/month=10/day=10/hour=19/=744248, year=2017/month=10/day=10/hour=03/=699192, year=2017/month=10/day=10/hour=14/=729204, year=2017/month=10/day=10/hour=13/=725310, year=2017/month=10/day=10/hour=05/=703904, year=2017/month=10/day=10/hour=16/=736202, year=2017/month=10/day=10/hour=04/=701868, year=2017/month=10/day=10/hour=15/=732942, year=2017/month=10/day=10/hour=10/=714308, year=2017/month=10/day=10/hour=21/=749456, year=2017/month=10/day=10/hour=20/=746978, year=2017/month=10/day=10/hour=12/=720930, year=2017/month=10/day=10/hour=23/=751897, year=2017/month=10/day=10/hour=11/=717390, year=2017/month=10/day=10/hour=22/=751896} (io.confluent.connect.hdfs.TopicPartitionWriter:297)
[2017-10-10 19:00:01,865] INFO Committed hdfs://dev//data/kafka/topics/twitter/year=2017/month=10/day=10/hour=23//twitter+0+0000751897+0000751897.txt for twitter-0 (io.confluent.connect.hdfs.TopicPartitionWriter:625)
[2017-10-10 19:00:01,868] INFO Committed hdfs://dev//data/kafka/topics/twitter/year=2017/month=10/day=10/hour=22//twitter+0+0000749458+0000751896.txt for twitter-0 (io.confluent.connect.hdfs.TopicPartitionWriter:625)
[2017-10-10 19:00:01,868] INFO committing files after waiting for rotateIntervalMs time but less than flush.size records available. (io.confluent.connect.hdfs.TopicPartitionWriter:333)
[2017-10-10 19:00:01,881] INFO Committed hdfs://dev//data/kafka/topics/test/year=2017/month=10/day=10/hour=22//test+0+0004598921+0004600091.txt for test-0 (io.confluent.connect.hdfs.TopicPartitionWriter:625)
[2017-10-10 19:00:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1183: {twitter-0=OffsetAndMetadata{offset=751909, metadata=''}, test-0=OffsetAndMetadata{offset=4600092, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:01:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1184: {twitter-0=OffsetAndMetadata{offset=751981, metadata=''}, test-0=OffsetAndMetadata{offset=4600131, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:02:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1185: {twitter-0=OffsetAndMetadata{offset=752057, metadata=''}, test-0=OffsetAndMetadata{offset=4600167, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:03:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1186: {twitter-0=OffsetAndMetadata{offset=752091, metadata=''}, test-0=OffsetAndMetadata{offset=4600187, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:04:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1187: {twitter-0=OffsetAndMetadata{offset=752121, metadata=''}, test-0=OffsetAndMetadata{offset=4600204, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:05:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1188: {twitter-0=OffsetAndMetadata{offset=752167, metadata=''}, test-0=OffsetAndMetadata{offset=4600221, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:06:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1189: {twitter-0=OffsetAndMetadata{offset=752187, metadata=''}, test-0=OffsetAndMetadata{offset=4600230, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:07:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1190: {twitter-0=OffsetAndMetadata{offset=752217, metadata=''}, test-0=OffsetAndMetadata{offset=4600246, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:08:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1191: {twitter-0=OffsetAndMetadata{offset=752239, metadata=''}, test-0=OffsetAndMetadata{offset=4600253, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:09:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1192: {twitter-0=OffsetAndMetadata{offset=752281, metadata=''}, test-0=OffsetAndMetadata{offset=4600275, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:10:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1193: {twitter-0=OffsetAndMetadata{offset=752313, metadata=''}, test-0=OffsetAndMetadata{offset=4600294, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:11:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1194: {twitter-0=OffsetAndMetadata{offset=752353, metadata=''}, test-0=OffsetAndMetadata{offset=4600314, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:12:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1195: {twitter-0=OffsetAndMetadata{offset=752399, metadata=''}, test-0=OffsetAndMetadata{offset=4600334, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:13:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1196: {twitter-0=OffsetAndMetadata{offset=752429, metadata=''}, test-0=OffsetAndMetadata{offset=4600351, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:14:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1197: {twitter-0=OffsetAndMetadata{offset=752483, metadata=''}, test-0=OffsetAndMetadata{offset=4600374, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:15:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1198: {twitter-0=OffsetAndMetadata{offset=752529, metadata=''}, test-0=OffsetAndMetadata{offset=4600394, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:16:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1199: {twitter-0=OffsetAndMetadata{offset=752555, metadata=''}, test-0=OffsetAndMetadata{offset=4600409, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:17:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1200: {twitter-0=OffsetAndMetadata{offset=752593, metadata=''}, test-0=OffsetAndMetadata{offset=4600427, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:18:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1201: {twitter-0=OffsetAndMetadata{offset=752615, metadata=''}, test-0=OffsetAndMetadata{offset=4600436, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:19:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1202: {twitter-0=OffsetAndMetadata{offset=752637, metadata=''}, test-0=OffsetAndMetadata{offset=4600445, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:20:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1203: {twitter-0=OffsetAndMetadata{offset=752673, metadata=''}, test-0=OffsetAndMetadata{offset=4600462, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:21:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1204: {twitter-0=OffsetAndMetadata{offset=752705, metadata=''}, test-0=OffsetAndMetadata{offset=4600480, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:22:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1205: {twitter-0=OffsetAndMetadata{offset=752739, metadata=''}, test-0=OffsetAndMetadata{offset=4600500, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:23:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1206: {twitter-0=OffsetAndMetadata{offset=752793, metadata=''}, test-0=OffsetAndMetadata{offset=4600519, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:24:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1207: {twitter-0=OffsetAndMetadata{offset=752835, metadata=''}, test-0=OffsetAndMetadata{offset=4600538, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:25:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1208: {twitter-0=OffsetAndMetadata{offset=752873, metadata=''}, test-0=OffsetAndMetadata{offset=4600556, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:26:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1209: {twitter-0=OffsetAndMetadata{offset=752923, metadata=''}, test-0=OffsetAndMetadata{offset=4600581, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:27:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1210: {twitter-0=OffsetAndMetadata{offset=752969, metadata=''}, test-0=OffsetAndMetadata{offset=4600603, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:28:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1211: {twitter-0=OffsetAndMetadata{offset=753015, metadata=''}, test-0=OffsetAndMetadata{offset=4600626, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:29:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1212: {twitter-0=OffsetAndMetadata{offset=753055, metadata=''}, test-0=OffsetAndMetadata{offset=4600646, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:30:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1213: {twitter-0=OffsetAndMetadata{offset=753095, metadata=''}, test-0=OffsetAndMetadata{offset=4600662, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:31:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1214: {twitter-0=OffsetAndMetadata{offset=753141, metadata=''}, test-0=OffsetAndMetadata{offset=4600688, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:32:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1215: {twitter-0=OffsetAndMetadata{offset=753166, metadata=''}, test-0=OffsetAndMetadata{offset=4600699, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:33:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1216: {twitter-0=OffsetAndMetadata{offset=753227, metadata=''}, test-0=OffsetAndMetadata{offset=4600728, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:34:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1217: {twitter-0=OffsetAndMetadata{offset=753249, metadata=''}, test-0=OffsetAndMetadata{offset=4600742, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:35:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1218: {twitter-0=OffsetAndMetadata{offset=753275, metadata=''}, test-0=OffsetAndMetadata{offset=4600752, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:36:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1219: {twitter-0=OffsetAndMetadata{offset=753293, metadata=''}, test-0=OffsetAndMetadata{offset=4600762, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:37:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1220: {twitter-0=OffsetAndMetadata{offset=753337, metadata=''}, test-0=OffsetAndMetadata{offset=4600781, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:38:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1221: {twitter-0=OffsetAndMetadata{offset=753371, metadata=''}, test-0=OffsetAndMetadata{offset=4600801, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:39:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1222: {twitter-0=OffsetAndMetadata{offset=753409, metadata=''}, test-0=OffsetAndMetadata{offset=4600819, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:40:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1223: {twitter-0=OffsetAndMetadata{offset=753429, metadata=''}, test-0=OffsetAndMetadata{offset=4600828, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:41:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1224: {twitter-0=OffsetAndMetadata{offset=753457, metadata=''}, test-0=OffsetAndMetadata{offset=4600843, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:42:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1225: {twitter-0=OffsetAndMetadata{offset=753493, metadata=''}, test-0=OffsetAndMetadata{offset=4600859, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:43:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1226: {twitter-0=OffsetAndMetadata{offset=753525, metadata=''}, test-0=OffsetAndMetadata{offset=4600872, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:44:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1227: {twitter-0=OffsetAndMetadata{offset=753553, metadata=''}, test-0=OffsetAndMetadata{offset=4600886, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:45:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1228: {twitter-0=OffsetAndMetadata{offset=753591, metadata=''}, test-0=OffsetAndMetadata{offset=4600901, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:46:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1229: {twitter-0=OffsetAndMetadata{offset=753627, metadata=''}, test-0=OffsetAndMetadata{offset=4600919, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:47:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1230: {twitter-0=OffsetAndMetadata{offset=753663, metadata=''}, test-0=OffsetAndMetadata{offset=4600938, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:48:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1231: {twitter-0=OffsetAndMetadata{offset=753687, metadata=''}, test-0=OffsetAndMetadata{offset=4600951, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:49:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1232: {twitter-0=OffsetAndMetadata{offset=753717, metadata=''}, test-0=OffsetAndMetadata{offset=4600965, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:50:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1233: {twitter-0=OffsetAndMetadata{offset=753743, metadata=''}, test-0=OffsetAndMetadata{offset=4600976, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:51:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1234: {twitter-0=OffsetAndMetadata{offset=753771, metadata=''}, test-0=OffsetAndMetadata{offset=4600989, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:52:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1235: {twitter-0=OffsetAndMetadata{offset=753807, metadata=''}, test-0=OffsetAndMetadata{offset=4601008, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:53:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1236: {twitter-0=OffsetAndMetadata{offset=753835, metadata=''}, test-0=OffsetAndMetadata{offset=4601023, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:54:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1237: {twitter-0=OffsetAndMetadata{offset=753871, metadata=''}, test-0=OffsetAndMetadata{offset=4601040, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:55:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1238: {twitter-0=OffsetAndMetadata{offset=753923, metadata=''}, test-0=OffsetAndMetadata{offset=4601061, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:56:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1239: {twitter-0=OffsetAndMetadata{offset=753965, metadata=''}, test-0=OffsetAndMetadata{offset=4601084, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:57:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1240: {twitter-0=OffsetAndMetadata{offset=753993, metadata=''}, test-0=OffsetAndMetadata{offset=4601100, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:58:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1241: {twitter-0=OffsetAndMetadata{offset=754013, metadata=''}, test-0=OffsetAndMetadata{offset=4601110, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 19:59:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1242: {twitter-0=OffsetAndMetadata{offset=754053, metadata=''}, test-0=OffsetAndMetadata{offset=4601128, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:00:00,456] INFO Starting commit and rotation for topic partition twitter-0 with start offsets {year=2017/month=10/day=11/hour=00/=754077, year=2017/month=10/day=10/hour=23/=751898} and end offsets {year=2017/month=10/day=10/hour=07/=708077, year=2017/month=10/day=10/hour=18/=741506, year=2017/month=10/day=10/hour=06/=705988, year=2017/month=10/day=10/hour=17/=738750, year=2017/month=10/day=10/hour=09/=712088, year=2017/month=10/day=10/hour=08/=710134, year=2017/month=10/day=10/hour=19/=744248, year=2017/month=10/day=11/hour=00/=754077, year=2017/month=10/day=10/hour=03/=699192, year=2017/month=10/day=10/hour=14/=729204, year=2017/month=10/day=10/hour=13/=725310, year=2017/month=10/day=10/hour=05/=703904, year=2017/month=10/day=10/hour=16/=736202, year=2017/month=10/day=10/hour=04/=701868, year=2017/month=10/day=10/hour=15/=732942, year=2017/month=10/day=10/hour=10/=714308, year=2017/month=10/day=10/hour=21/=749456, year=2017/month=10/day=10/hour=20/=746978, year=2017/month=10/day=10/hour=12/=720930, year=2017/month=10/day=10/hour=23/=754076, year=2017/month=10/day=10/hour=11/=717390, year=2017/month=10/day=10/hour=22/=751896} (io.confluent.connect.hdfs.TopicPartitionWriter:297)
[2017-10-10 20:00:00,483] INFO Committed hdfs://dev//data/kafka/topics/twitter/year=2017/month=10/day=11/hour=00//twitter+0+0000754077+0000754077.txt for twitter-0 (io.confluent.connect.hdfs.TopicPartitionWriter:625)
[2017-10-10 20:00:00,486] INFO Committed hdfs://dev//data/kafka/topics/twitter/year=2017/month=10/day=10/hour=23//twitter+0+0000751898+0000754076.txt for twitter-0 (io.confluent.connect.hdfs.TopicPartitionWriter:625)
[2017-10-10 20:00:00,486] INFO committing files after waiting for rotateIntervalMs time but less than flush.size records available. (io.confluent.connect.hdfs.TopicPartitionWriter:333)
[2017-10-10 20:00:00,498] INFO Committed hdfs://dev//data/kafka/topics/test/year=2017/month=10/day=10/hour=23//test+0+0004600092+0004601139.txt for test-0 (io.confluent.connect.hdfs.TopicPartitionWriter:625)
[2017-10-10 20:00:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1243: {twitter-0=OffsetAndMetadata{offset=754085, metadata=''}, test-0=OffsetAndMetadata{offset=4601141, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:01:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1244: {twitter-0=OffsetAndMetadata{offset=754153, metadata=''}, test-0=OffsetAndMetadata{offset=4601175, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:02:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1245: {twitter-0=OffsetAndMetadata{offset=754197, metadata=''}, test-0=OffsetAndMetadata{offset=4601202, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:03:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1246: {twitter-0=OffsetAndMetadata{offset=754233, metadata=''}, test-0=OffsetAndMetadata{offset=4601220, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:04:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1247: {twitter-0=OffsetAndMetadata{offset=754259, metadata=''}, test-0=OffsetAndMetadata{offset=4601232, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:05:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1248: {twitter-0=OffsetAndMetadata{offset=754291, metadata=''}, test-0=OffsetAndMetadata{offset=4601250, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:06:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1249: {twitter-0=OffsetAndMetadata{offset=754317, metadata=''}, test-0=OffsetAndMetadata{offset=4601263, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:07:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1250: {twitter-0=OffsetAndMetadata{offset=754345, metadata=''}, test-0=OffsetAndMetadata{offset=4601272, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:08:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1251: {twitter-0=OffsetAndMetadata{offset=754375, metadata=''}, test-0=OffsetAndMetadata{offset=4601290, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:09:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1252: {twitter-0=OffsetAndMetadata{offset=754401, metadata=''}, test-0=OffsetAndMetadata{offset=4601304, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:10:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1253: {twitter-0=OffsetAndMetadata{offset=754443, metadata=''}, test-0=OffsetAndMetadata{offset=4601319, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:11:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1254: {twitter-0=OffsetAndMetadata{offset=754465, metadata=''}, test-0=OffsetAndMetadata{offset=4601330, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:12:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1255: {twitter-0=OffsetAndMetadata{offset=754493, metadata=''}, test-0=OffsetAndMetadata{offset=4601342, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:13:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1256: {twitter-0=OffsetAndMetadata{offset=754523, metadata=''}, test-0=OffsetAndMetadata{offset=4601355, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:14:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1257: {twitter-0=OffsetAndMetadata{offset=754561, metadata=''}, test-0=OffsetAndMetadata{offset=4601371, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:15:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1258: {twitter-0=OffsetAndMetadata{offset=754617, metadata=''}, test-0=OffsetAndMetadata{offset=4601397, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:16:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1259: {twitter-0=OffsetAndMetadata{offset=754667, metadata=''}, test-0=OffsetAndMetadata{offset=4601421, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:17:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1260: {twitter-0=OffsetAndMetadata{offset=754719, metadata=''}, test-0=OffsetAndMetadata{offset=4601446, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:18:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1261: {twitter-0=OffsetAndMetadata{offset=754759, metadata=''}, test-0=OffsetAndMetadata{offset=4601466, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:19:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1262: {twitter-0=OffsetAndMetadata{offset=754793, metadata=''}, test-0=OffsetAndMetadata{offset=4601486, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:20:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1263: {twitter-0=OffsetAndMetadata{offset=754837, metadata=''}, test-0=OffsetAndMetadata{offset=4601504, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:21:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1264: {twitter-0=OffsetAndMetadata{offset=754869, metadata=''}, test-0=OffsetAndMetadata{offset=4601519, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:22:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1265: {twitter-0=OffsetAndMetadata{offset=754903, metadata=''}, test-0=OffsetAndMetadata{offset=4601532, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:23:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1266: {twitter-0=OffsetAndMetadata{offset=754933, metadata=''}, test-0=OffsetAndMetadata{offset=4601548, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:24:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1267: {twitter-0=OffsetAndMetadata{offset=754977, metadata=''}, test-0=OffsetAndMetadata{offset=4601568, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:25:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1268: {twitter-0=OffsetAndMetadata{offset=755025, metadata=''}, test-0=OffsetAndMetadata{offset=4601596, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:26:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1269: {twitter-0=OffsetAndMetadata{offset=755083, metadata=''}, test-0=OffsetAndMetadata{offset=4601623, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:27:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1270: {twitter-0=OffsetAndMetadata{offset=755129, metadata=''}, test-0=OffsetAndMetadata{offset=4601647, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:28:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1271: {twitter-0=OffsetAndMetadata{offset=755167, metadata=''}, test-0=OffsetAndMetadata{offset=4601665, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:29:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1272: {twitter-0=OffsetAndMetadata{offset=755189, metadata=''}, test-0=OffsetAndMetadata{offset=4601679, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:30:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1273: {twitter-0=OffsetAndMetadata{offset=755233, metadata=''}, test-0=OffsetAndMetadata{offset=4601697, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:31:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1274: {twitter-0=OffsetAndMetadata{offset=755267, metadata=''}, test-0=OffsetAndMetadata{offset=4601715, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:32:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1275: {twitter-0=OffsetAndMetadata{offset=755303, metadata=''}, test-0=OffsetAndMetadata{offset=4601732, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:33:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1276: {twitter-0=OffsetAndMetadata{offset=755345, metadata=''}, test-0=OffsetAndMetadata{offset=4601753, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:34:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1277: {twitter-0=OffsetAndMetadata{offset=755377, metadata=''}, test-0=OffsetAndMetadata{offset=4601768, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:35:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1278: {twitter-0=OffsetAndMetadata{offset=755399, metadata=''}, test-0=OffsetAndMetadata{offset=4601782, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:36:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1279: {twitter-0=OffsetAndMetadata{offset=755455, metadata=''}, test-0=OffsetAndMetadata{offset=4601806, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:37:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1280: {twitter-0=OffsetAndMetadata{offset=755471, metadata=''}, test-0=OffsetAndMetadata{offset=4601815, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:38:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1281: {twitter-0=OffsetAndMetadata{offset=755501, metadata=''}, test-0=OffsetAndMetadata{offset=4601829, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:39:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1282: {twitter-0=OffsetAndMetadata{offset=755525, metadata=''}, test-0=OffsetAndMetadata{offset=4601841, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:40:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1283: {twitter-0=OffsetAndMetadata{offset=755573, metadata=''}, test-0=OffsetAndMetadata{offset=4601863, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:41:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1284: {twitter-0=OffsetAndMetadata{offset=755613, metadata=''}, test-0=OffsetAndMetadata{offset=4601883, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:42:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1285: {twitter-0=OffsetAndMetadata{offset=755633, metadata=''}, test-0=OffsetAndMetadata{offset=4601895, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:43:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1286: {twitter-0=OffsetAndMetadata{offset=755657, metadata=''}, test-0=OffsetAndMetadata{offset=4601904, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:44:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1287: {twitter-0=OffsetAndMetadata{offset=755669, metadata=''}, test-0=OffsetAndMetadata{offset=4601913, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:45:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1288: {twitter-0=OffsetAndMetadata{offset=755695, metadata=''}, test-0=OffsetAndMetadata{offset=4601924, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:46:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1289: {twitter-0=OffsetAndMetadata{offset=755725, metadata=''}, test-0=OffsetAndMetadata{offset=4601938, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:47:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1290: {twitter-0=OffsetAndMetadata{offset=755777, metadata=''}, test-0=OffsetAndMetadata{offset=4601955, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:48:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1291: {twitter-0=OffsetAndMetadata{offset=755825, metadata=''}, test-0=OffsetAndMetadata{offset=4601988, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:49:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1292: {twitter-0=OffsetAndMetadata{offset=755865, metadata=''}, test-0=OffsetAndMetadata{offset=4602007, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:50:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1293: {twitter-0=OffsetAndMetadata{offset=755913, metadata=''}, test-0=OffsetAndMetadata{offset=4602026, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:51:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1294: {twitter-0=OffsetAndMetadata{offset=755945, metadata=''}, test-0=OffsetAndMetadata{offset=4602043, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:52:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1295: {twitter-0=OffsetAndMetadata{offset=755983, metadata=''}, test-0=OffsetAndMetadata{offset=4602060, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:53:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1296: {twitter-0=OffsetAndMetadata{offset=756013, metadata=''}, test-0=OffsetAndMetadata{offset=4602076, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:54:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1297: {twitter-0=OffsetAndMetadata{offset=756063, metadata=''}, test-0=OffsetAndMetadata{offset=4602101, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:55:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1298: {twitter-0=OffsetAndMetadata{offset=756109, metadata=''}, test-0=OffsetAndMetadata{offset=4602122, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:56:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1299: {twitter-0=OffsetAndMetadata{offset=756143, metadata=''}, test-0=OffsetAndMetadata{offset=4602137, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:57:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1300: {twitter-0=OffsetAndMetadata{offset=756183, metadata=''}, test-0=OffsetAndMetadata{offset=4602158, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:58:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1301: {twitter-0=OffsetAndMetadata{offset=756217, metadata=''}, test-0=OffsetAndMetadata{offset=4602174, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 20:59:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1302: {twitter-0=OffsetAndMetadata{offset=756255, metadata=''}, test-0=OffsetAndMetadata{offset=4602194, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:00:02,142] INFO Starting commit and rotation for topic partition twitter-0 with start offsets {year=2017/month=10/day=11/hour=01/=756301, year=2017/month=10/day=11/hour=00/=754078} and end offsets {year=2017/month=10/day=11/hour=01/=756301, year=2017/month=10/day=10/hour=07/=708077, year=2017/month=10/day=10/hour=18/=741506, year=2017/month=10/day=10/hour=06/=705988, year=2017/month=10/day=10/hour=17/=738750, year=2017/month=10/day=10/hour=09/=712088, year=2017/month=10/day=10/hour=08/=710134, year=2017/month=10/day=10/hour=19/=744248, year=2017/month=10/day=11/hour=00/=756300, year=2017/month=10/day=10/hour=03/=699192, year=2017/month=10/day=10/hour=14/=729204, year=2017/month=10/day=10/hour=13/=725310, year=2017/month=10/day=10/hour=05/=703904, year=2017/month=10/day=10/hour=16/=736202, year=2017/month=10/day=10/hour=04/=701868, year=2017/month=10/day=10/hour=15/=732942, year=2017/month=10/day=10/hour=10/=714308, year=2017/month=10/day=10/hour=21/=749456, year=2017/month=10/day=10/hour=20/=746978, year=2017/month=10/day=10/hour=12/=720930, year=2017/month=10/day=10/hour=23/=754076, year=2017/month=10/day=10/hour=11/=717390, year=2017/month=10/day=10/hour=22/=751896} (io.confluent.connect.hdfs.TopicPartitionWriter:297)
[2017-10-10 21:00:02,170] INFO Committed hdfs://dev//data/kafka/topics/twitter/year=2017/month=10/day=11/hour=01//twitter+0+0000756301+0000756301.txt for twitter-0 (io.confluent.connect.hdfs.TopicPartitionWriter:625)
[2017-10-10 21:00:02,172] INFO Committed hdfs://dev//data/kafka/topics/twitter/year=2017/month=10/day=11/hour=00//twitter+0+0000754078+0000756300.txt for twitter-0 (io.confluent.connect.hdfs.TopicPartitionWriter:625)
[2017-10-10 21:00:02,173] INFO committing files after waiting for rotateIntervalMs time but less than flush.size records available. (io.confluent.connect.hdfs.TopicPartitionWriter:333)
[2017-10-10 21:00:02,184] INFO Committed hdfs://dev//data/kafka/topics/test/year=2017/month=10/day=11/hour=00//test+0+0004601140+0004602207.txt for test-0 (io.confluent.connect.hdfs.TopicPartitionWriter:625)
[2017-10-10 21:00:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1303: {twitter-0=OffsetAndMetadata{offset=756303, metadata=''}, test-0=OffsetAndMetadata{offset=4602208, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:01:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1304: {twitter-0=OffsetAndMetadata{offset=756359, metadata=''}, test-0=OffsetAndMetadata{offset=4602236, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:02:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1305: {twitter-0=OffsetAndMetadata{offset=756419, metadata=''}, test-0=OffsetAndMetadata{offset=4602265, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:03:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1306: {twitter-0=OffsetAndMetadata{offset=756480, metadata=''}, test-0=OffsetAndMetadata{offset=4602287, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:04:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1307: {twitter-0=OffsetAndMetadata{offset=756525, metadata=''}, test-0=OffsetAndMetadata{offset=4602318, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:05:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1308: {twitter-0=OffsetAndMetadata{offset=756563, metadata=''}, test-0=OffsetAndMetadata{offset=4602336, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:06:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1309: {twitter-0=OffsetAndMetadata{offset=756613, metadata=''}, test-0=OffsetAndMetadata{offset=4602361, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:07:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1310: {twitter-0=OffsetAndMetadata{offset=756635, metadata=''}, test-0=OffsetAndMetadata{offset=4602370, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:08:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1311: {twitter-0=OffsetAndMetadata{offset=756667, metadata=''}, test-0=OffsetAndMetadata{offset=4602390, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:09:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1312: {twitter-0=OffsetAndMetadata{offset=756705, metadata=''}, test-0=OffsetAndMetadata{offset=4602408, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:10:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1313: {twitter-0=OffsetAndMetadata{offset=756759, metadata=''}, test-0=OffsetAndMetadata{offset=4602433, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:11:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1314: {twitter-0=OffsetAndMetadata{offset=756797, metadata=''}, test-0=OffsetAndMetadata{offset=4602452, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:12:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1315: {twitter-0=OffsetAndMetadata{offset=756875, metadata=''}, test-0=OffsetAndMetadata{offset=4602475, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:13:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1316: {twitter-0=OffsetAndMetadata{offset=756911, metadata=''}, test-0=OffsetAndMetadata{offset=4602495, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:14:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1317: {twitter-0=OffsetAndMetadata{offset=756945, metadata=''}, test-0=OffsetAndMetadata{offset=4602509, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:15:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1318: {twitter-0=OffsetAndMetadata{offset=756971, metadata=''}, test-0=OffsetAndMetadata{offset=4602524, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:16:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1319: {twitter-0=OffsetAndMetadata{offset=756993, metadata=''}, test-0=OffsetAndMetadata{offset=4602533, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:17:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1320: {twitter-0=OffsetAndMetadata{offset=757035, metadata=''}, test-0=OffsetAndMetadata{offset=4602555, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:18:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1321: {twitter-0=OffsetAndMetadata{offset=757055, metadata=''}, test-0=OffsetAndMetadata{offset=4602563, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:19:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1322: {twitter-0=OffsetAndMetadata{offset=757085, metadata=''}, test-0=OffsetAndMetadata{offset=4602579, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:20:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1323: {twitter-0=OffsetAndMetadata{offset=757111, metadata=''}, test-0=OffsetAndMetadata{offset=4602588, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:21:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1324: {twitter-0=OffsetAndMetadata{offset=757125, metadata=''}, test-0=OffsetAndMetadata{offset=4602597, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:22:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1325: {twitter-0=OffsetAndMetadata{offset=757163, metadata=''}, test-0=OffsetAndMetadata{offset=4602613, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:23:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1326: {twitter-0=OffsetAndMetadata{offset=757219, metadata=''}, test-0=OffsetAndMetadata{offset=4602641, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:24:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1327: {twitter-0=OffsetAndMetadata{offset=757243, metadata=''}, test-0=OffsetAndMetadata{offset=4602657, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:25:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1328: {twitter-0=OffsetAndMetadata{offset=757291, metadata=''}, test-0=OffsetAndMetadata{offset=4602674, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:26:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1329: {twitter-0=OffsetAndMetadata{offset=757315, metadata=''}, test-0=OffsetAndMetadata{offset=4602691, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:27:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1330: {twitter-0=OffsetAndMetadata{offset=757347, metadata=''}, test-0=OffsetAndMetadata{offset=4602706, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:28:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1331: {twitter-0=OffsetAndMetadata{offset=757377, metadata=''}, test-0=OffsetAndMetadata{offset=4602721, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:29:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1332: {twitter-0=OffsetAndMetadata{offset=757405, metadata=''}, test-0=OffsetAndMetadata{offset=4602733, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:30:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1333: {twitter-0=OffsetAndMetadata{offset=757447, metadata=''}, test-0=OffsetAndMetadata{offset=4602748, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:31:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1334: {twitter-0=OffsetAndMetadata{offset=757591, metadata=''}, test-0=OffsetAndMetadata{offset=4602810, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:32:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1335: {twitter-0=OffsetAndMetadata{offset=757731, metadata=''}, test-0=OffsetAndMetadata{offset=4602895, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:33:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1336: {twitter-0=OffsetAndMetadata{offset=757757, metadata=''}, test-0=OffsetAndMetadata{offset=4602908, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:34:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1337: {twitter-0=OffsetAndMetadata{offset=757795, metadata=''}, test-0=OffsetAndMetadata{offset=4602925, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:35:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1338: {twitter-0=OffsetAndMetadata{offset=757821, metadata=''}, test-0=OffsetAndMetadata{offset=4602936, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:36:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1339: {twitter-0=OffsetAndMetadata{offset=757845, metadata=''}, test-0=OffsetAndMetadata{offset=4602951, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:37:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1340: {twitter-0=OffsetAndMetadata{offset=757889, metadata=''}, test-0=OffsetAndMetadata{offset=4602969, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:38:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1341: {twitter-0=OffsetAndMetadata{offset=757913, metadata=''}, test-0=OffsetAndMetadata{offset=4602984, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:39:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1342: {twitter-0=OffsetAndMetadata{offset=757949, metadata=''}, test-0=OffsetAndMetadata{offset=4603001, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:40:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1343: {twitter-0=OffsetAndMetadata{offset=757983, metadata=''}, test-0=OffsetAndMetadata{offset=4603019, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:41:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1344: {twitter-0=OffsetAndMetadata{offset=758013, metadata=''}, test-0=OffsetAndMetadata{offset=4603035, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:42:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1345: {twitter-0=OffsetAndMetadata{offset=758059, metadata=''}, test-0=OffsetAndMetadata{offset=4603056, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:43:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1346: {twitter-0=OffsetAndMetadata{offset=758095, metadata=''}, test-0=OffsetAndMetadata{offset=4603071, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:44:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1347: {twitter-0=OffsetAndMetadata{offset=758119, metadata=''}, test-0=OffsetAndMetadata{offset=4603087, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:45:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1348: {twitter-0=OffsetAndMetadata{offset=758137, metadata=''}, test-0=OffsetAndMetadata{offset=4603097, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:46:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1349: {twitter-0=OffsetAndMetadata{offset=758163, metadata=''}, test-0=OffsetAndMetadata{offset=4603106, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:47:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1350: {twitter-0=OffsetAndMetadata{offset=758199, metadata=''}, test-0=OffsetAndMetadata{offset=4603124, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:48:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1351: {twitter-0=OffsetAndMetadata{offset=758225, metadata=''}, test-0=OffsetAndMetadata{offset=4603138, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:49:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1352: {twitter-0=OffsetAndMetadata{offset=758249, metadata=''}, test-0=OffsetAndMetadata{offset=4603150, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:50:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1353: {twitter-0=OffsetAndMetadata{offset=758287, metadata=''}, test-0=OffsetAndMetadata{offset=4603167, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:51:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1354: {twitter-0=OffsetAndMetadata{offset=758321, metadata=''}, test-0=OffsetAndMetadata{offset=4603181, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:52:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1355: {twitter-0=OffsetAndMetadata{offset=758349, metadata=''}, test-0=OffsetAndMetadata{offset=4603196, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:53:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1356: {twitter-0=OffsetAndMetadata{offset=758385, metadata=''}, test-0=OffsetAndMetadata{offset=4603214, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:54:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1357: {twitter-0=OffsetAndMetadata{offset=758413, metadata=''}, test-0=OffsetAndMetadata{offset=4603228, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:55:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1358: {twitter-0=OffsetAndMetadata{offset=758431, metadata=''}, test-0=OffsetAndMetadata{offset=4603237, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:56:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1359: {twitter-0=OffsetAndMetadata{offset=758455, metadata=''}, test-0=OffsetAndMetadata{offset=4603249, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:57:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1360: {twitter-0=OffsetAndMetadata{offset=758483, metadata=''}, test-0=OffsetAndMetadata{offset=4603264, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:58:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1361: {twitter-0=OffsetAndMetadata{offset=758513, metadata=''}, test-0=OffsetAndMetadata{offset=4603279, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 21:59:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1362: {twitter-0=OffsetAndMetadata{offset=758543, metadata=''}, test-0=OffsetAndMetadata{offset=4603291, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:00:00,377] INFO committing files after waiting for rotateIntervalMs time but less than flush.size records available. (io.confluent.connect.hdfs.TopicPartitionWriter:333)
[2017-10-10 22:00:00,398] INFO Committed hdfs://dev//data/kafka/topics/twitter/year=2017/month=10/day=11/hour=01//twitter+0+0000756302+0000758578.txt for twitter-0 (io.confluent.connect.hdfs.TopicPartitionWriter:625)
[2017-10-10 22:00:00,401] INFO Starting commit and rotation for topic partition test-0 with start offsets {year=2017/month=10/day=11/hour=01/=4602208, year=2017/month=10/day=11/hour=02/=4603306} and end offsets {year=2017/month=10/day=11/hour=01/=4603305, year=2017/month=10/day=11/hour=02/=4603306, year=2017/month=10/day=10/hour=07/=4579400, year=2017/month=10/day=10/hour=18/=4595130, year=2017/month=10/day=10/hour=06/=4578412, year=2017/month=10/day=10/hour=17/=4593831, year=2017/month=10/day=10/hour=09/=4581310, year=2017/month=10/day=10/hour=08/=4580382, year=2017/month=10/day=10/hour=19/=4596436, year=2017/month=10/day=11/hour=00/=4602207, year=2017/month=10/day=10/hour=03/=4575105, year=2017/month=10/day=10/hour=14/=4589286, year=2017/month=10/day=10/hour=13/=4587411, year=2017/month=10/day=10/hour=05/=4577404, year=2017/month=10/day=10/hour=16/=4592594, year=2017/month=10/day=10/hour=04/=4576416, year=2017/month=10/day=10/hour=15/=4591042, year=2017/month=10/day=10/hour=10/=4582380, year=2017/month=10/day=10/hour=21/=4598920, year=2017/month=10/day=10/hour=20/=4597736, year=2017/month=10/day=10/hour=12/=4585288, year=2017/month=10/day=10/hour=23/=4601139, year=2017/month=10/day=10/hour=11/=4583711, year=2017/month=10/day=10/hour=22/=4600091} (io.confluent.connect.hdfs.TopicPartitionWriter:297)
[2017-10-10 22:00:00,430] INFO Committed hdfs://dev//data/kafka/topics/test/year=2017/month=10/day=11/hour=01//test+0+0004602208+0004603305.txt for test-0 (io.confluent.connect.hdfs.TopicPartitionWriter:625)
[2017-10-10 22:00:00,434] INFO Committed hdfs://dev//data/kafka/topics/test/year=2017/month=10/day=11/hour=02//test+0+0004603306+0004603306.txt for test-0 (io.confluent.connect.hdfs.TopicPartitionWriter:625)
[2017-10-10 22:00:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1363: {twitter-0=OffsetAndMetadata{offset=758581, metadata=''}, test-0=OffsetAndMetadata{offset=4603309, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:01:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1364: {twitter-0=OffsetAndMetadata{offset=758637, metadata=''}, test-0=OffsetAndMetadata{offset=4603331, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:02:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1365: {twitter-0=OffsetAndMetadata{offset=758687, metadata=''}, test-0=OffsetAndMetadata{offset=4603361, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:03:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1366: {twitter-0=OffsetAndMetadata{offset=758711, metadata=''}, test-0=OffsetAndMetadata{offset=4603374, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:04:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1367: {twitter-0=OffsetAndMetadata{offset=758735, metadata=''}, test-0=OffsetAndMetadata{offset=4603386, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:05:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1368: {twitter-0=OffsetAndMetadata{offset=758769, metadata=''}, test-0=OffsetAndMetadata{offset=4603404, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:06:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1369: {twitter-0=OffsetAndMetadata{offset=758801, metadata=''}, test-0=OffsetAndMetadata{offset=4603420, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:07:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1370: {twitter-0=OffsetAndMetadata{offset=758839, metadata=''}, test-0=OffsetAndMetadata{offset=4603434, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:08:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1371: {twitter-0=OffsetAndMetadata{offset=758873, metadata=''}, test-0=OffsetAndMetadata{offset=4603456, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:09:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1372: {twitter-0=OffsetAndMetadata{offset=758907, metadata=''}, test-0=OffsetAndMetadata{offset=4603473, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:10:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1373: {twitter-0=OffsetAndMetadata{offset=758945, metadata=''}, test-0=OffsetAndMetadata{offset=4603491, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:11:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1374: {twitter-0=OffsetAndMetadata{offset=758985, metadata=''}, test-0=OffsetAndMetadata{offset=4603510, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:12:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1375: {twitter-0=OffsetAndMetadata{offset=759017, metadata=''}, test-0=OffsetAndMetadata{offset=4603525, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:13:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1376: {twitter-0=OffsetAndMetadata{offset=759045, metadata=''}, test-0=OffsetAndMetadata{offset=4603541, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:14:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1377: {twitter-0=OffsetAndMetadata{offset=759071, metadata=''}, test-0=OffsetAndMetadata{offset=4603553, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:15:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1378: {twitter-0=OffsetAndMetadata{offset=759093, metadata=''}, test-0=OffsetAndMetadata{offset=4603564, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:16:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1379: {twitter-0=OffsetAndMetadata{offset=759223, metadata=''}, test-0=OffsetAndMetadata{offset=4603593, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:17:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1380: {twitter-0=OffsetAndMetadata{offset=759257, metadata=''}, test-0=OffsetAndMetadata{offset=4603613, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:18:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1381: {twitter-0=OffsetAndMetadata{offset=759297, metadata=''}, test-0=OffsetAndMetadata{offset=4603631, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:19:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1382: {twitter-0=OffsetAndMetadata{offset=759349, metadata=''}, test-0=OffsetAndMetadata{offset=4603654, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:20:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1383: {twitter-0=OffsetAndMetadata{offset=759379, metadata=''}, test-0=OffsetAndMetadata{offset=4603673, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:21:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1384: {twitter-0=OffsetAndMetadata{offset=759409, metadata=''}, test-0=OffsetAndMetadata{offset=4603685, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:22:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1385: {twitter-0=OffsetAndMetadata{offset=759429, metadata=''}, test-0=OffsetAndMetadata{offset=4603694, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:23:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1386: {twitter-0=OffsetAndMetadata{offset=759459, metadata=''}, test-0=OffsetAndMetadata{offset=4603710, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:24:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1387: {twitter-0=OffsetAndMetadata{offset=759491, metadata=''}, test-0=OffsetAndMetadata{offset=4603727, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:25:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1388: {twitter-0=OffsetAndMetadata{offset=759541, metadata=''}, test-0=OffsetAndMetadata{offset=4603745, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:26:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1389: {twitter-0=OffsetAndMetadata{offset=759563, metadata=''}, test-0=OffsetAndMetadata{offset=4603756, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:27:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1390: {twitter-0=OffsetAndMetadata{offset=759609, metadata=''}, test-0=OffsetAndMetadata{offset=4603779, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:28:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1391: {twitter-0=OffsetAndMetadata{offset=759639, metadata=''}, test-0=OffsetAndMetadata{offset=4603790, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:29:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1392: {twitter-0=OffsetAndMetadata{offset=759687, metadata=''}, test-0=OffsetAndMetadata{offset=4603811, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:30:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1393: {twitter-0=OffsetAndMetadata{offset=759725, metadata=''}, test-0=OffsetAndMetadata{offset=4603832, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:31:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1394: {twitter-0=OffsetAndMetadata{offset=759783, metadata=''}, test-0=OffsetAndMetadata{offset=4603859, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:32:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1395: {twitter-0=OffsetAndMetadata{offset=759807, metadata=''}, test-0=OffsetAndMetadata{offset=4603871, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:33:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1396: {twitter-0=OffsetAndMetadata{offset=759839, metadata=''}, test-0=OffsetAndMetadata{offset=4603886, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:34:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1397: {twitter-0=OffsetAndMetadata{offset=759857, metadata=''}, test-0=OffsetAndMetadata{offset=4603897, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:35:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1398: {twitter-0=OffsetAndMetadata{offset=759877, metadata=''}, test-0=OffsetAndMetadata{offset=4603908, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:36:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1399: {twitter-0=OffsetAndMetadata{offset=759905, metadata=''}, test-0=OffsetAndMetadata{offset=4603919, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:37:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1400: {twitter-0=OffsetAndMetadata{offset=759933, metadata=''}, test-0=OffsetAndMetadata{offset=4603935, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:38:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1401: {twitter-0=OffsetAndMetadata{offset=759959, metadata=''}, test-0=OffsetAndMetadata{offset=4603949, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:39:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1402: {twitter-0=OffsetAndMetadata{offset=759977, metadata=''}, test-0=OffsetAndMetadata{offset=4603955, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:40:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1403: {twitter-0=OffsetAndMetadata{offset=760047, metadata=''}, test-0=OffsetAndMetadata{offset=4603977, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:41:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1404: {twitter-0=OffsetAndMetadata{offset=760087, metadata=''}, test-0=OffsetAndMetadata{offset=4604001, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:42:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1405: {twitter-0=OffsetAndMetadata{offset=760107, metadata=''}, test-0=OffsetAndMetadata{offset=4604012, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:43:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1406: {twitter-0=OffsetAndMetadata{offset=760129, metadata=''}, test-0=OffsetAndMetadata{offset=4604024, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:44:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1407: {twitter-0=OffsetAndMetadata{offset=760163, metadata=''}, test-0=OffsetAndMetadata{offset=4604039, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:45:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1408: {twitter-0=OffsetAndMetadata{offset=760185, metadata=''}, test-0=OffsetAndMetadata{offset=4604050, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:46:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1409: {twitter-0=OffsetAndMetadata{offset=760211, metadata=''}, test-0=OffsetAndMetadata{offset=4604064, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:47:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1410: {twitter-0=OffsetAndMetadata{offset=760237, metadata=''}, test-0=OffsetAndMetadata{offset=4604077, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:48:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1411: {twitter-0=OffsetAndMetadata{offset=760269, metadata=''}, test-0=OffsetAndMetadata{offset=4604092, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:49:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1412: {twitter-0=OffsetAndMetadata{offset=760297, metadata=''}, test-0=OffsetAndMetadata{offset=4604104, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:50:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1413: {twitter-0=OffsetAndMetadata{offset=760325, metadata=''}, test-0=OffsetAndMetadata{offset=4604119, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:51:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1414: {twitter-0=OffsetAndMetadata{offset=760351, metadata=''}, test-0=OffsetAndMetadata{offset=4604127, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:52:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1415: {twitter-0=OffsetAndMetadata{offset=760371, metadata=''}, test-0=OffsetAndMetadata{offset=4604141, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:53:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1416: {twitter-0=OffsetAndMetadata{offset=760389, metadata=''}, test-0=OffsetAndMetadata{offset=4604148, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:54:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1417: {twitter-0=OffsetAndMetadata{offset=760403, metadata=''}, test-0=OffsetAndMetadata{offset=4604156, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:55:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1418: {twitter-0=OffsetAndMetadata{offset=760455, metadata=''}, test-0=OffsetAndMetadata{offset=4604182, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:56:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1419: {twitter-0=OffsetAndMetadata{offset=760485, metadata=''}, test-0=OffsetAndMetadata{offset=4604193, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:57:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1420: {twitter-0=OffsetAndMetadata{offset=760509, metadata=''}, test-0=OffsetAndMetadata{offset=4604207, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:58:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1421: {twitter-0=OffsetAndMetadata{offset=760547, metadata=''}, test-0=OffsetAndMetadata{offset=4604225, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 22:59:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1422: {twitter-0=OffsetAndMetadata{offset=760573, metadata=''}, test-0=OffsetAndMetadata{offset=4604240, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 23:00:00,425] INFO Starting commit and rotation for topic partition twitter-0 with start offsets {year=2017/month=10/day=11/hour=02/=758579, year=2017/month=10/day=11/hour=03/=760601} and end offsets {year=2017/month=10/day=11/hour=02/=760600, year=2017/month=10/day=10/hour=07/=708077, year=2017/month=10/day=10/hour=17/=738750, year=2017/month=10/day=10/hour=09/=712088, year=2017/month=10/day=10/hour=19/=744248, year=2017/month=10/day=11/hour=00/=756300, year=2017/month=10/day=10/hour=03/=699192, year=2017/month=10/day=10/hour=13/=725310, year=2017/month=10/day=10/hour=05/=703904, year=2017/month=10/day=10/hour=15/=732942, year=2017/month=10/day=10/hour=21/=749456, year=2017/month=10/day=10/hour=23/=754076, year=2017/month=10/day=10/hour=11/=717390, year=2017/month=10/day=11/hour=01/=758578, year=2017/month=10/day=11/hour=03/=760601, year=2017/month=10/day=10/hour=18/=741506, year=2017/month=10/day=10/hour=06/=705988, year=2017/month=10/day=10/hour=08/=710134, year=2017/month=10/day=10/hour=14/=729204, year=2017/month=10/day=10/hour=16/=736202, year=2017/month=10/day=10/hour=04/=701868, year=2017/month=10/day=10/hour=10/=714308, year=2017/month=10/day=10/hour=20/=746978, year=2017/month=10/day=10/hour=12/=720930, year=2017/month=10/day=10/hour=22/=751896} (io.confluent.connect.hdfs.TopicPartitionWriter:297)
[2017-10-10 23:00:00,454] INFO Committed hdfs://dev//data/kafka/topics/twitter/year=2017/month=10/day=11/hour=02//twitter+0+0000758579+0000760600.txt for twitter-0 (io.confluent.connect.hdfs.TopicPartitionWriter:625)
[2017-10-10 23:00:00,459] INFO Committed hdfs://dev//data/kafka/topics/twitter/year=2017/month=10/day=11/hour=03//twitter+0+0000760601+0000760601.txt for twitter-0 (io.confluent.connect.hdfs.TopicPartitionWriter:625)
[2017-10-10 23:00:00,459] INFO committing files after waiting for rotateIntervalMs time but less than flush.size records available. (io.confluent.connect.hdfs.TopicPartitionWriter:333)
[2017-10-10 23:00:00,472] INFO Committed hdfs://dev//data/kafka/topics/test/year=2017/month=10/day=11/hour=02//test+0+0004603307+0004604249.txt for test-0 (io.confluent.connect.hdfs.TopicPartitionWriter:625)
[2017-10-10 23:00:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1423: {twitter-0=OffsetAndMetadata{offset=760611, metadata=''}, test-0=OffsetAndMetadata{offset=4604252, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 23:01:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1424: {twitter-0=OffsetAndMetadata{offset=760651, metadata=''}, test-0=OffsetAndMetadata{offset=4604275, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 23:02:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1425: {twitter-0=OffsetAndMetadata{offset=760707, metadata=''}, test-0=OffsetAndMetadata{offset=4604300, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 23:03:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1426: {twitter-0=OffsetAndMetadata{offset=760731, metadata=''}, test-0=OffsetAndMetadata{offset=4604315, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 23:04:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1427: {twitter-0=OffsetAndMetadata{offset=760749, metadata=''}, test-0=OffsetAndMetadata{offset=4604324, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 23:05:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1428: {twitter-0=OffsetAndMetadata{offset=760767, metadata=''}, test-0=OffsetAndMetadata{offset=4604335, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 23:06:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1429: {twitter-0=OffsetAndMetadata{offset=760791, metadata=''}, test-0=OffsetAndMetadata{offset=4604346, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 23:07:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1430: {twitter-0=OffsetAndMetadata{offset=760835, metadata=''}, test-0=OffsetAndMetadata{offset=4604364, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 23:08:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1431: {twitter-0=OffsetAndMetadata{offset=760863, metadata=''}, test-0=OffsetAndMetadata{offset=4604381, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 23:09:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1432: {twitter-0=OffsetAndMetadata{offset=760881, metadata=''}, test-0=OffsetAndMetadata{offset=4604390, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 23:10:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1433: {twitter-0=OffsetAndMetadata{offset=760913, metadata=''}, test-0=OffsetAndMetadata{offset=4604405, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 23:11:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1434: {twitter-0=OffsetAndMetadata{offset=760943, metadata=''}, test-0=OffsetAndMetadata{offset=4604419, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 23:12:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1435: {twitter-0=OffsetAndMetadata{offset=760973, metadata=''}, test-0=OffsetAndMetadata{offset=4604435, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 23:13:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1436: {twitter-0=OffsetAndMetadata{offset=761021, metadata=''}, test-0=OffsetAndMetadata{offset=4604458, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 23:14:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1437: {twitter-0=OffsetAndMetadata{offset=761075, metadata=''}, test-0=OffsetAndMetadata{offset=4604481, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 23:15:03,687] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1438: {twitter-0=OffsetAndMetadata{offset=761101, metadata=''}, test-0=OffsetAndMetadata{offset=4604498, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 23:16:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1439: {twitter-0=OffsetAndMetadata{offset=761145, metadata=''}, test-0=OffsetAndMetadata{offset=4604517, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 23:17:03,688] INFO WorkerSinkTask{id=hdfs-sink-5} Committing offsets asynchronously using sequence number 1440: {twitter-0=OffsetAndMetadata{offset=761167, metadata=''}, test-0=OffsetAndMetadata{offset=4604530, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:288)
[2017-10-10 23:17:13,509] WARN Not attempting to re-login since the last re-login was attempted less than 600 seconds before. (org.apache.hadoop.security.UserGroupInformation:1127)
[2017-10-10 23:17:16,099] WARN Not attempting to re-login since the last re-login was attempted less than 600 seconds before. (org.apache.hadoop.security.UserGroupInformation:1127)
[2017-10-10 23:17:20,763] WARN Not attempting to re-login since the last re-login was attempted less than 600 seconds before. (org.apache.hadoop.security.UserGroupInformation:1127)
[2017-10-10 23:17:24,791] WARN Not attempting to re-login since the last re-login was attempted less than 600 seconds before. (org.apache.hadoop.security.UserGroupInformation:1127)
[2017-10-10 23:17:29,495] WARN Couldn't setup connection for kafka/box12.ourcompany.com@OURCOMPANY.COM to box01.ourcompany.com/192.168.14.51:8020 (org.apache.hadoop.ipc.Client:671)
[2017-10-10 23:17:29,500] INFO Exception while invoking renewLease of class ClientNamenodeProtocolTranslatorPB over box01.ourcompany.com/192.168.14.51:8020. Trying to fail over immediately. (org.apache.hadoop.io.retry.RetryInvocationHandler:140)
java.io.IOException: Failed on local exception: java.io.IOException: Couldn't setup connection for kafka/box12.ourcompany.com@OURCOMPANY.COM to box01.ourcompany.com/192.168.14.51:8020; Host Details : local host is: "box12.ourcompany.com/192.168.14.62"; destination host is: "box01.ourcompany.com":8020; 
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
    at org.apache.hadoop.ipc.Client.call(Client.java:1472)
    at org.apache.hadoop.ipc.Client.call(Client.java:1399)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
    at com.sun.proxy.$Proxy47.renewLease(Unknown Source)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:571)
    at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
    at com.sun.proxy.$Proxy48.renewLease(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:879)
    at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417)
    at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442)
    at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
    at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Couldn't setup connection for kafka/box12.ourcompany.com@OURCOMPANY.COM to box01.ourcompany.com/192.168.14.51:8020
    at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:672)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)
    at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:730)
    at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
    at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
    at org.apache.hadoop.ipc.Client.call(Client.java:1438)
    ... 16 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
    at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
    at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
    at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)
    at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)
    at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)
    at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)
    ... 19 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
    at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
    at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
    at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
    at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
    at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
    at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
    at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
    ... 28 more
kaushiksrinivas commented 5 years ago

Any update on this behavior ? Facing similar issue.

mstaffa commented 5 years ago

Bump. Also seeing this issue.

heaje commented 4 years ago

I'm running into this issue currently with Confluent Connect 5.4.1 as well. I've been spending quite some time trying to track down where the issue is. I'm not a developer, but it does appear this problem might lie somewhere in the Hadoop client libraries (hadoop-common UserGroupInformation class). Following the debug logging, I've found the following:

  1. The HDFS sink calls for renewal of the keytab using a refresh interval (defaults to one hour). This is the UGI reloginFromKeytab() call.
  2. The Hadoop UGI library then does properly go through all the necessary checks of validating enough time has passed to warrant a renewal (80% of the ticket lifetime). That function is located here.
  3. In the event renewal is necessary, the UGI library does go through all the proper steps according to the code that is there. All the expected DEBUG log entries do show up.

The strange thing about this is in my case, I could see in my logs that it was renewing exactly as expected. Then suddenly 1.5 hours after the previous renewal I started getting the errors that I didn't have valid credentials. My ticket lifetime is 8 hours for each renewal. This leads me to believe something in the Hadoop client libraries is NOT actually renewing the ticket as expected, but it only fails sometimes. Other times things DO renew as expected and I don't get the errors.

Maybe this bug is related? Complete guess -> https://issues.apache.org/jira/browse/HADOOP-13433

heaje commented 4 years ago

I attempted running the connector using Hadoop 2.7.7 (I just updated the pom.xml to use Hadoop 2.7.7 and then rebuilt it with the new dependencies). We still saw this issue even after using Hadoop 2.7.7 (which includes the fix for the bug I gave in my last comment).

Me and @jkt628 continued debugging further, and we think there could be some kind of race condition occurring in the getTGT() call of the Hadoop common code. We think that potentially this function is finding the wrong TGT. We haven't found a definitive way to prove that.

That being the case, we think there is some weird scenario occurring where the Hadoop code is renewing a ticket that is not actually being used by the connect sink. As a result, the sink starts throwing errors after the original ticket expires.

We are testing a one-line change in the DataWriter class that forces the HDFS sink to instead use an externally managed keytab (on the system using kinit). The change is simply this on line 173: // UserGroupInformation.loginUserFromKeytab(principal, keytab);

I'll report back after we've gone through our set of tests.

heaje commented 4 years ago

This one change led to the HDFS sink using an external kerberos ticket and all of our tests proved successful. Here is what we tested:

  1. Connector functions normally across writes as expected.
  2. Connector functions normally after renewal.
  3. Connector functions normally even after someone re-initializes the ticket using "kinit -kt" from the command line.
  4. Connector continues functioning even after the ticket has been destroyed until the ticket has expired. Assuming a valid ticket still exists in the Kerberos cache, the continue functions just fine after expiry of the original ticket.
  5. Connector recovers as expected if the ticket has been destroyed, the renewal period has lapsed, and then a user re-initializes the ticket using "kinit -kt" from the command line. Errors were thrown in the logs as expected until this was done.

We are running this modified version of the connector in production at this point. In 6+ days, we have not seen a single instance of this kerberos authentication issue recur with hosts using the updated code.

UPDATE: We have continued running this modified code in production and we have not had any recurrences of this issue. This really looks like the hadoop-common library is somehow getting things mixed up somehow when each task controls its own ticket.

dosvath commented 3 years ago

I attempted running the connector using Hadoop 2.7.7 (I just updated the pom.xml to use Hadoop 2.7.7 and then rebuilt it with the new dependencies). We still saw this issue even after using Hadoop 2.7.7 (which includes the fix for the bug I gave in my last comment).

Me and @jkt628 continued debugging further, and we think there could be some kind of race condition occurring in the getTGT() call of the Hadoop common code. We think that potentially this function is finding the wrong TGT. We haven't found a definitive way to prove that.

That being the case, we think there is some weird scenario occurring where the Hadoop code is renewing a ticket that is not actually being used by the connect sink. As a result, the sink starts throwing errors after the original ticket expires.

We are testing a one-line change in the DataWriter class that forces the HDFS sink to instead use an externally managed keytab (on the system using kinit). The change is simply this on line 173: // UserGroupInformation.loginUserFromKeytab(principal, keytab);

I'll report back after we've gone through our set of tests.

Confirming that this is a viable workaround while the Hadoop library issue is fixed/investigated.

vdesabou commented 3 years ago

FYI https://issues.apache.org/jira/browse/HDFS-16165 was opened to track this issue and reproduction model has been made to consistently reproduce the issue.

dinegri commented 3 years ago

This one change led to the HDFS sink using an external kerberos ticket and all of our tests proved successful. Here is what we tested:

1. Connector functions normally across writes as expected.

2. Connector functions normally after renewal.

3. Connector functions normally even after someone re-initializes the ticket using "kinit -kt" from the command line.

4. Connector continues functioning even after the ticket has been destroyed until the ticket has expired.  Assuming a valid ticket still exists in the Kerberos cache, the continue functions just fine after expiry of the original ticket.

5. Connector recovers as expected if the ticket has been destroyed, the renewal period has lapsed, and then a user re-initializes the ticket using "kinit -kt" from the command line.  Errors were thrown in the logs as expected until this was done.

We are running this modified version of the connector in production at this point. In 6+ days, we have not seen a single instance of this kerberos authentication issue recur with hosts using the updated code.

UPDATE: We have continued running this modified code in production and we have not had any recurrences of this issue. This really looks like the hadoop-common library is somehow getting things mixed up somehow when each task controls its own ticket.

Hi @heaje, we have been running more than 200 HDFS Sink connectors using your suggestion for 1 week and we had zero issues of kerberos.

We tried migrate to HDFS sink 3 but we still got kerberos issue about ticket renew

Thank you very much

fallen-up commented 2 years ago

@heaje hi, do you have an actual lifehack for 10.1.x versions? disabling of https://github.com/confluentinc/kafka-connect-hdfs/blob/v10.1.4/src/main/java/io/confluent/connect/hdfs/DataWriter.java#L265 doesn't work in my tests (there are fatal errors during start connector)

heaje commented 2 years ago

@fallen-up,

Funny enough, I just started working today on making a patch on version 10.1.11. I should hopefully have something within a week.

fallen-up commented 2 years ago

@heaje i will expand the question: did you have the experience of launching your patch in the strimzi-operator environment (example-basic-manifest)? in this case we can't overwrite running options (kinit/k5start) in yaml, only by patching start-up-scripts -> too much pain :(

thanks a lot

heaje commented 2 years ago

@heaje i will expand the question: did you have the experience of launching your patch in the strimzi-operator environment (example-basic-manifest)? in this case we can't overwrite running options (kinit/k5start) in yaml, only by patching start-up-scripts -> too much pain :(

thanks a lot

I haven't ever run anything in strimzi-operator. We just run this on bare-metal.

heaje commented 2 years ago

@fallen-up,

A draft of the patch can be found at https://gist.github.com/heaje/924d070daedea904f7b4172b487e2433. I HAVE NOT TESTED THIS. I have no environment to test this in for at least a couple of weeks. You are free to give it a shot, but again, this is UNTESTED.

I threw the patch in a Gist because this is in no way something that should ever be upstreamed into the HDFS sink code.

lciolecki commented 1 year ago

@heaje are you sure there is need to remove: ticketRenewThread = new Thread(() -> renewKerberosTicket(ugi));? What about the renewing Kerbero's ticket?

heaje commented 1 year ago

@heaje are you sure there is need to remove: ticketRenewThread = new Thread(() -> renewKerberosTicket(ugi));? What about the renewing Kerbero's ticket?

@lciolecki - Yes, that must be removed. The entire purpose of this code removal is to force the connector to use an externally managed keytab. Since the keytab is externally managed, that means some external service needs to take care of renewing the ticket.

That said, I have been using the code from that draft for the past 3 months without any issues.

lciolecki commented 1 year ago

Ahh, I understood now, e.g. we can renew Kerberos's ticket by the cron? Thanks, man!

heaje commented 1 year ago

Ahh, I understood now, e.g. we can renew Kerberos's ticket by the cron? Thanks, man!

Yes, that is correct. However you go about renewing the ticket, something external to Kafka Connect will need to do it.

orinciog commented 1 year ago

@heaje Thank you for the v10 code . I have tested using hdfs-connect v10.1.15 and it works ok. The renewing of kerberos ticket is done by a cron job, which is launched every hour. The cronjob does only kinit -kt <keytab> <principal>

Thank you again for this brilliant code.

NikolayTsvetkov commented 9 months ago

@heaje Thank you for the investigation and the proposed workaround! I have applied the proposed changes, but the connector cannot find Kerberos ticket even though it is available in the cache. Have you applied any additional JAAS config instructing the connector how the ticket should be obtained or it's simply working for you? I'm using version 10.2.0.