copy large files(over 1GB)

[floodico]

Hey. I set up Listen to listen to my folder, where are copying a lot of large files. And the next step I have to do is move these file to some processing. BUT the big problem is that the added callback works almost immediately and this file(larger than 1GB) is still copying to the folder at this time(for 10 seconds). I need to avoid the way the file is processed during this time. Maybe have i to use rb-inotify to handle the created event and then i will know for sure that the file is completely copied?!

[ColinDKelley]

@floodico It makes sense that you get notified when the file is first opened. listen is built on inotify and that notifies when the filename is first created in the folder, not when it's closed.

There is a fix for this, but it involves changing the file writer. Assuming you have access to it, change it to follow the "atomic file write" pattern:

1. Write the file to a temporary place on the same device **.
2. When the write is complete, mv (rename) it into its final destination. This way any code that monitors the final destination has no chance of seeing a partially written file. Even if the writing code crashes.

** Note that this folder location must be on the same device as the final folder, so that the rename can be atomic--on different devices, Linux will copy/delete which won't be atomic. To be sure the device matches, you can always write to the exact same folder, but to a pathname that is ignored by listen. For example:

File.open("/folder/file.tmp", "w") do |file|
  # write file here
end
FileUtils.mv("/folder/file.tmp", "/folder.file.dat")

When listening, ignore: the .tmp files:

listener = Listen.to("/folder", ignore: /\.tmp/) { |modified, added, removed| ... }
listener.start

or only: listen to the .dat files:

listener = Listen.to("/folder", only: /\.dat/) { |modified, added, removed| ... }
listener.start

[ColinDKelley]

@floodico I hope the above is helpful. Closing this ticket since I believe the fix needs to be outside of the listen gem.