search

alexholmes / hdfs-file-slurper

Utility to easily copy files into HDFS
Apache License 2.0
69 stars 35 forks source link

Slurper is not detecting copying file #13

Open nagarajasn opened 9 years ago

nagarajasn commented 9 years ago

slurper is start copying the file as soon as it detect the file in local file system. its not waiting till full copy.

I am setting below property in the configuration file VERIFY = true Please let me know if i'm missing anything.

Note : I am using winscp tool for uploading files into slurper in directory

alexholmes commented 9 years ago

There can be a race condition if a file is being written-to when the slurper picks it up. The general recommendation is to either perform an atomic move into the directory of a file, or if you're writing to a file, make it "hidden" (prefix with a period "."), and once the file's been written then rename it.

On Mon, Sep 14, 2015 at 9:42 AM, nagarajasn notifications@github.com wrote:

slurper is start copying the file as soon as it detect the file in local file system. its not waiting till full copy.

I am setting below property in the configuration file VERIFY = true Please let me know if i'm missing anything.

Note : I am using winscp tool for uploading files into slurper in directory

— Reply to this email directly or view it on GitHub https://github.com/alexholmes/hdfs-file-slurper/issues/13.