search

Azure / kafka-sink-azure-kusto

Kafka sink for Kusto
MIT License
48 stars 24 forks source link

Connect worker fails with failed to create new directory error #44

Closed jojokoshy closed 4 years ago

jojokoshy commented 4 years ago

Kafka Kusto Sink connector fails with an error after running for few minutes. For smaller topics it seemed to work fine.

{ "id": 8, "state": "FAILED", "worker_id": "1.1.1.1:8083", "trace": "org.apache.kafka.connect.errors.ConnectException: Exiting WorkerSinkTask due to unrecoverable exception.\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:568)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:326)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:228)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:196)\n\tat org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:184)\n\tat org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:234)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: org.apache.kafka.connect.errors.ConnectException: Failed to create file or write record into file for ingestion.\n\tat com.microsoft.azure.kusto.kafka.connect.sink.TopicPartitionWriter.handleErrors(TopicPartitionWriter.java:171)\n\tat com.microsoft.azure.kusto.kafka.connect.sink.TopicPartitionWriter.writeRecord(TopicPartitionWriter.java:162)\n\tat com.microsoft.azure.kusto.kafka.connect.sink.KustoSinkTask.put(KustoSinkTask.java:401)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:546)\n\t... 10 more\nCaused by: java.io.IOException: Failed to create new directory /var/tmp/kusto-sink-connector-c98a74ad-8dc7-4f39-8ef7-ab8af064aeff\n\tat com.microsoft.azure.kusto.kafka.connect.sink.FileWriter.openFile(FileWriter.java:103)\n\tat com.microsoft.azure.kusto.kafka.connect.sink.FileWriter.writeData(FileWriter.java:243)\n\tat com.microsoft.azure.kusto.kafka.connect.sink.TopicPartitionWriter.writeRecord(TopicPartitionWriter.java:160)\n\t... 12 more\n" },

anagha-microsoft commented 4 years ago

Thanks Jojo. We will review and get back to you.

anagha-microsoft commented 4 years ago

Kafka Kusto Sink connector fails with an error after running for few minutes. For smaller topics it seemed to work fine.

{ "id": 8, "state": "FAILED", "worker_id": "1.1.1.1:8083", "trace": "org.apache.kafka.connect.errors.ConnectException: Exiting WorkerSinkTask due to unrecoverable exception.\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:568)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:326)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:228)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:196)\n\tat org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:184)\n\tat org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:234)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: org.apache.kafka.connect.errors.ConnectException: Failed to create file or write record into file for ingestion.\n\tat com.microsoft.azure.kusto.kafka.connect.sink.TopicPartitionWriter.handleErrors(TopicPartitionWriter.java:171)\n\tat com.microsoft.azure.kusto.kafka.connect.sink.TopicPartitionWriter.writeRecord(TopicPartitionWriter.java:162)\n\tat com.microsoft.azure.kusto.kafka.connect.sink.KustoSinkTask.put(KustoSinkTask.java:401)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:546)\n\t... 10 more\nCaused by: java.io.IOException: Failed to create new directory /var/tmp/kusto-sink-connector-c98a74ad-8dc7-4f39-8ef7-ab8af064aeff\n\tat com.microsoft.azure.kusto.kafka.connect.sink.FileWriter.openFile(FileWriter.java:103)\n\tat com.microsoft.azure.kusto.kafka.connect.sink.FileWriter.writeData(FileWriter.java:243)\n\tat com.microsoft.azure.kusto.kafka.connect.sink.TopicPartitionWriter.writeRecord(TopicPartitionWriter.java:160)\n\t... 12 more\n" },

Hello Jojo- The latest release version of the connector has fix for the issues reported. Do give it a try in an environment where you have multiple pods on the same AKS node, with each pod running multiple tasks to reproduce the environment where the issue was identified. Do let us know if you run into any issues.

We will close the issue.

yihezkel commented 4 years ago

Hi @jojokoshy ,

As @anagha-microsoft mentioned, your issue has been fixed in the latest release (https://github.com/Azure/kafka-sink-azure-kusto/releases/tag/v1.0.4).

We are closing this issue. Please let us know if you have any further questions or issues!

jojokoshy commented 4 years ago

Yes. I tested this and this issue is fixed.