Support optional AXL configuration that deletes source files when transfer completes

[adammoody]

One potential future AXL user has requested the ability to have AXL automatically delete a source file after it has successfully transferred the file to its destination. This feature should be optional, since some users want to also keep the source files around.

An open question is how to handle the case where a transfer moves more than one file, and a subset of those succeed and others error out. Do we require an all or nothing when considering whether to delete the source files?

[tonyhutter]

I think should copy the behaviour of mv. I did a little experiment where I tried to move four 3MB files (testfile[1-4]) into a 10MB mount to see what would happen. I got an ENOSPC while copying the 3rd file. Here's what files I saw afterwards:

Source:

-rw-r----- 1 hutter2 hutter2 3145728 Jul 16 08:52 testfile3
-rw-r----- 1 hutter2 hutter2 3145728 Jul 16 08:52 testfile4

Dest:

-rw-r----- 1 hutter2 hutter2 3145728 Jul 16 08:52 testfile1
-rw-r----- 1 hutter2 hutter2 3145728 Jul 16 08:52 testfile2
-rw------- 1 root    root    2420736 Jul 16 08:52 testfile3
-rw------- 1 root    root          0 Jul 16 08:52 testfile4

So it looks like mv doesn't remove the source file, nor set the permissions on the destination file, until the file is fully transferred.

[bnicolae]

Hmm, "mv" should not have permission to run as root. What you probably are observing is the behavior of the file system, which is something beyond the control of mv. I think the most appropriate approach for AXL would be to emulate how web browsers are downloading files: create a temp name on the destination mount point, try to complete the transfer, if successful rename the file and remove the source. On failure, nobody will work with incomplete files at any point, a subsequent transfer of the same file can detect partial progress and resume (if temp names are not random) and garbage collection is also easy

[tonyhutter]

@bnicolae just for some context - the source files were owned by hutter2, but I was moving them to an ext4 mount that was owned by root (specifically, a file that I formatted ext4 and mounted loopback). So I just did a sudo mv ... because I was too lazy to change the permissions of the mount before copying.

[tonyhutter]

@bnicolae forgot to mention that as of https://github.com/ECP-VeloC/AXL/commit/d738fb7a973367f57d9480cba9b2b63717920374 AXL will use a temp file name while transferring.