search

qubole / streamx

kafka-connect-s3 : Ingest data from Kafka to Object Stores(s3)
Apache License 2.0
96 stars 54 forks source link

org.apache.kafka.connect.errors.ConnectException: java.io.FileNotFoundException: File /logs/test-streamx/0/log does not exist #31

Closed zzbennett closed 7 years ago

zzbennett commented 7 years ago

I was following the quickstart and when I tried starting the connector for the first time, I got this exception:

[2017-01-24 00:48:41,877] ERROR Recovery failed at state WAL_APPLIED (io.confluent.connect.hdfs.TopicPartitionWriter:219)
org.apache.kafka.connect.errors.ConnectException: java.io.FileNotFoundException: File /logs/test-streamx/0/log does not exist
    at io.confluent.connect.hdfs.utils.MemoryWAL.truncate(MemoryWAL.java:87)
    at io.confluent.connect.hdfs.TopicPartitionWriter.truncateWAL(TopicPartitionWriter.java:457)
    at io.confluent.connect.hdfs.TopicPartitionWriter.recover(TopicPartitionWriter.java:205)
    at io.confluent.connect.hdfs.TopicPartitionWriter.write(TopicPartitionWriter.java:232)
    at io.confluent.connect.hdfs.DataWriter.write(DataWriter.java:237)
    at io.confluent.connect.hdfs.HdfsSinkTask.put(HdfsSinkTask.java:92)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:384)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:240)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:172)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:143)
    at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:140)
    at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:175)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.FileNotFoundException: File /logs/test-streamx/0/log does not exist
    at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:609)
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:822)
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:599)
    at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337)
    at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289)
    at org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:372)
    at org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:513)
    at com.qubole.streamx.s3.S3Storage.renameFile(S3Storage.java:140)
    at com.qubole.streamx.s3.S3Storage.commit(S3Storage.java:96)
    at io.confluent.connect.hdfs.utils.MemoryWAL.truncate(MemoryWAL.java:84)
    ... 16 more

I found this statement the HDFS connector doc file:

note: You need to make sure the connector user have write access to the directories specified in topics.dir and logs.dir. The default value of topics.dir is /topics and the default value of logs.dir is /logs, if you don't specify the two configurations, make sure that the connector user has write access to /topics and /logs. You may need to create /topics and /logs before running the connector as the connector usually don't have write access to /.

I did this, however, the error still persists. I then created the file manually and it seems to have resolved the problem. Possibly a bug? I'm using the NativeS3FileSystem, but other than that I just followed the quick start.

zzbennett commented 7 years ago

Shoot, I realized I had actually forgotten to set the s3.url and hadoop.conf.dir properties so that could have something to do with it.

PraveenSeluka commented 7 years ago

Hi @zzbennett Did you get past this issue ?

zzbennett commented 7 years ago

I did, but I manually created the file. I'm still not sure if forgetting to set the s3.url and hadoop.conf.dir had something to do with it. If not, then the bug still persists, and if so, then more error handling would be great.

PraveenSeluka commented 7 years ago

Hi @zzbennett This is the side-effect of not passing "s3.url". I will add some config checks to ensure, s3.url and hadoop.conf.dir as mandatory and log and shut down if its not provided.

PraveenSeluka commented 7 years ago

Closing this issue, this should not happen anymore. StreamX now uses DummyWAL by default (no WAL), and it can be configured to use RDSWAL instead.