spring-projects / spring-integration-extensions

The Spring Integration Extensions project provides extension components for Spring Integration
http://www.springintegration.org/
279 stars 265 forks source link

Inbound poller breaks when file gets modified while reading #177

Closed ghost closed 6 years ago

ghost commented 7 years ago

we have a use case, where we are reading the file from windows remote location with the following steps:

  1. Read file from windows share location say input folder.
  2. Usually files are very large in size ( > 100mb ) and poller takes time to download it.
  3. When poller is downloading files, and at the same time if user updates/modifies ( copy & paste the same file ) the source file then poller breaks and moves to next step in flow with half downloaded file.

Please see this link for more details http://stackoverflow.com/questions/43389062/spring-integration-inbound-poller-breaks-when-file-gets-modified-while-reading?noredirect=1#comment74369732_43389062

garyrussell commented 7 years ago

It is not a good design (using any technology) to fetch a large file that might be modified during the fetch. The results are indeterminate.

Plus, even if you could handle it, you'd be wasting bandwidth by fetching the same data over and over again.

Is there any way where we can append some ext to file say .processing before reading and start reading, one done with reading revert back to original name. I guess we can use SmbRemoteFileTemplate rt?

But what happens then if the user wants to write while the file has a temporary name?

It would be much better to have the writer "close" the file (by giving it a new name, perhaps with a timestamp) and start a new one from time-to-time and have your code only look for "closed" files.