Recursive transfers including empty directories

[fscheiner]

When using the gt option -r, files and sub directories are transferred recursively from the source to the destination. When the globus-url-copy (guc) option -cd is used (which should be done by default in all dparams to cover recursive cases and non-recursive cases where directories need to be created on the destination side), guc creates non-existing directories on the destination side if needed. Transfer lists contain commented lines for directories in this case:

[...]
"gsiftp://gridftp1.domain.tld:2811/~/tmp2/file3" "gsiftp://gridftp2.domain.tld:2811/~/tmp2/file3" 0,-1 size=1024;modify=1453792575;mode=0664;
"gsiftp://gridftp1.domain.tld:2811/~/tmp2/file4" "gsiftp://gridftp2.domain.tld:2811/~/tmp2/file4" 0,-1 size=1024;modify=1453792576;mode=0664;
#"gsiftp://gridftp1.domain.tld:2811/~/tmp2/subdir1/" "gsiftp://gridftp2.domain.tld:2811/~/tmp2/subdir1/" 0,-1 modify=1453792587;mode=0775;
#"gsiftp://gridftp1.domain.tld:2811/~/tmp2/subdir2/" "gsiftp://gridftp2.domain.tld:2811/~/tmp2/subdir2/" 0,-1 modify=1453792590;mode=0775;
"gsiftp://gridftp1.domain.tld:2811/~/tmp2/subdir1/file1" "gsiftp://gridftp2.domain.tld:2811/~/tmp2/subdir1/file1" 0,-1 size=1024;modify=1453792587;mode=0664;
"gsiftp://gridftp1.domain.tld:2811/~/tmp2/subdir1/file2" "gsiftp://gridftp2.domain.tld:2811/~/tmp2/subdir1/file2" 0,-1 size=1024;modify=1453792587;mode=0664;
"gsiftp://gridftp1.domain.tld:2811/~/tmp2/subdir1/file3" "gsiftp://gridftp2.domain.tld:2811/~/tmp2/subdir1/file3" 0,-1 size=1024;modify=1453792587;mode=0664;
"gsiftp://gridftp1.domain.tld:2811/~/tmp2/subdir1/file4" "gsiftp://gridftp2.domain.tld:2811/~/tmp2/subdir1/file4" 0,-1 size=1024;modify=1453792587;mode=0664;
#"gsiftp://gridftp1.domain.tld:2811/~/tmp2/subdir1/subsubddir1/" "gsiftp://gridftp2.domain.tld:2811/~/tmp2/subdir1/subsubddir1/" 0,-1 modify=1453792584;mode=0775;
"gsiftp://gridftp1.domain.tld:2811/~/tmp2/subdir2/file1" "gsiftp://gridftp2.domain.tld:2811/~/tmp2/subdir2/file1" 0,-1 size=1024;modify=1453792590;mode=0664;
"gsiftp://gridftp1.domain.tld:2811/~/tmp2/subdir2/file2" "gsiftp://gridftp2.domain.tld:2811/~/tmp2/subdir2/file2" 0,-1 size=1024;modify=1453792590;mode=0664;
"gsiftp://gridftp1.domain.tld:2811/~/tmp2/subdir2/file3" "gsiftp://gridftp2.domain.tld:2811/~/tmp2/subdir2/file3" 0,-1 size=1024;modify=1453792590;mode=0664;
"gsiftp://gridftp1.domain.tld:2811/~/tmp2/subdir2/file4" "gsiftp://gridftp2.domain.tld:2811/~/tmp2/subdir2/file4" 0,-1 size=1024;modify=1453792590;mode=0664;
[...]

The problem now is, that gt filters those comments before forwarding the transfer list to internal code and tgftp (and therefore also to guc), see line 742 in bin/gtransfer.bash and line 1402 in lib/gtransfer/listTransfer.bashlib. This is no problem if those directories actually contain files, as then there exist other non-commented URLs including the directories and guc then also creates those directories when using the -cd option. But if the directories are empty, guc does not create the directories, because the commented lines with the directory information were filtered by gt beforehand.

NOTICE: Due to this bug only empty directory structures are lost at the destination, but files are in any case transferred to the destination and hence aren't lost if removed locally.

[fscheiner]

I just found out that guc (at least until v9.18) actually ignores all commented lines in transfer lists. So commented lines for directories are not evaluated by guc at all. I was on the wrong track here. Hence empty directories are not created on the destination side, but only if they contain a file or sub directory.

It's even worse: When using a transfer list:

$ globus-url-copy -cd -r -cc 2 -tcp-bs 4M -p 2 -sync -sync-level 1 -f transfer.list

...guc only creates dirs on the destination that include empty dirs on the source side if the directory lines in the transfer list are uncommented.

When using URLs only:

$ globus-url-copy -cd -r -cc 2 -tcp-bs 4M -p 2 -sync -sync-level 1 gsiftp://gridftp1.domain.tld:2811/~/files/* gsiftp://gridftp3.domain.tld:2811/~/files/

...guc creates all dirs on the destination side, even dirs that are empty on the source side.

Uncommenting the commented lines for directories does not help, but also does not hurt then. And it makes sure that at least dirs that include other empty dirs are created on the destination side.

If such behaviour - i.e. creating empty dirs on the destination side - is useful at all, well that depends. But the fact that guc shows different behaviour between URL transfers and list transfers is questionable.

It looks like this cannot be fixed within gtransfer but needs a fix in guc.

[fscheiner]

WON'T FIX

But the changes made with 4d9586cfc8bb989fc7d8ef7d962cbb4341250fde stay in, because transfer lists don't need to be modified by gtransfer neither for transfer lists provided by users nor for transfer lists created with guc.