search

fast-data-transfer / fdt

FDT is an Application for Efficient Data Transfers which is capable of reading and writing at disk speed over wide area networks (with standard TCP). It is written in Java, runs an all major platforms and it is easy to use. FDT is based on an asynchronous, flexible multithreaded system and is using the capabilities of the Java NIO libraries.
https://fast-data-transfer.github.io/
Apache License 2.0
201 stars 45 forks source link

FDT exits when trying to pull a file that no longer exists on the source #55

Open sabbene opened 3 years ago

sabbene commented 3 years ago

It would be really cool if you could tell FDT to continue syncing other files even if some of the files in the file list do not exist. This would help with syncing dynamic data like home dirs, etc.

[root@test-02 tmp]# java -jar -XX:MaxDirectMemorySize=100000m /tmp/fdt.jar -pull -wCount 8 -rCount 8 -c test-01 -fl /tmp/file_list -d /output
...
...
FDTWriterSession ( d342e05b-d937-43b1-bff8-15a4b5787251 ) final stats:
 Started: Tue Apr 06 12:05:26 PDT 2021
 Ended:   Tue Apr 06 12:05:29 PDT 2021
 Transfer period:   03s
 TotalBytes: 0
 TotalNetworkBytes: 0
 Exit Status: Not OK

2021-04-06 12:05:29 INFO lia.util.net.copy.FDTWriterSession doPostProcessing [ FDTWriterSession ] Post Processing started
2021-04-06 12:05:29 INFO lia.util.net.copy.FDTWriterSession doPostProcessing [ FDTWriterSession ] No post processing filters defined/processed.
2021-04-06 12:05:30 INFO lia.util.net.copy.transport.ControlChannel sendMsgImpl  [ ControlChannel ] sending message tag ( 10 ): END_SESSION msg: null
2021-04-06 12:05:30 INFO lia.util.net.copy.FDT doWork  [ Tue Apr 06 12:05:30 PDT 2021 ] - GracefulStopper hook started ... Waiting for the cleanup to finish
2021-04-06 12:05:30 INFO lia.util.net.copy.FDT doWork  [ Tue Apr 06 12:05:30 PDT 2021 ]  - GracefulStopper hook finished!
2021-04-06 12:05:30 WARNING lia.util.net.copy.FDT doWork
 [ Tue Apr 06 12:05:30 PDT 2021 ]  FDT Session finished with errors:
2021-04-06 12:05:30 WARNING lia.util.net.copy.FDT doWork Down message: Got exception trying to process
Down cause:
java.io.FileNotFoundException: File does not exist! /tmp/sabbene3
        at lia.util.net.copy.FDTReaderSession.internalInit(FDTReaderSession.java:250)
        at lia.util.net.copy.FDTReaderSession.handleInitFDTSessionConf(FDTReaderSession.java:803)
        at lia.util.net.copy.FDTSession.notifyCtrlMsg(FDTSession.java:429)
        at lia.util.net.copy.FDTSessionManager.notifyCtrlMsg(FDTSessionManager.java:188)
        at lia.util.net.copy.transport.ControlChannel$1.run(ControlChannel.java:479)