ReproNim / reproman

ReproMan (AKA NICEMAN, AKA ReproNim TRD3)
https://reproman.readthedocs.io
Other
24 stars 14 forks source link

get from resource (EC3) on a directory fails with FileNotFoundError #555

Open yarikoptic opened 3 years ago

yarikoptic commented 3 years ago
$> reproman execute -r my-nitrc-gpu2 -- ls -l abcd-bids-test4
2020-11-10 12:35:58,438 [INFO   ] Executed the ls command in the environment my-nitrc-gpu2 
total 52
lrwxrwxrwx  1 ubuntu ubuntu  110 Nov 10 17:16 CHANGES -> .git/annex/objects/p7/m8/MD5E-s43--a12b06190d4ec20e98e5a2d8e03d9061/MD5E-s43--a12b06190d4ec20e98e5a2d8e03d9061
lrwxrwxrwx  1 ubuntu ubuntu  114 Nov 10 17:16 README -> .git/annex/objects/JM/7M/MD5E-s1358--a9bb7dd9d20474e16305a0640e4243ea/MD5E-s1358--a9bb7dd9d20474e16305a0640e4243ea
lrwxrwxrwx  1 ubuntu ubuntu  124 Nov 10 17:16 dataset_description.json -> .git/annex/objects/wG/zW/MD5E-s3417--0c03dba61aa23ad565e76db232b61fe1.json/MD5E-s3417--0c03dba61aa23ad565e76db232b61fe1.json
drwxrwxr-x  3 ubuntu ubuntu 4096 Nov 10 17:07 derivatives
drwxrwxr-x 23 ubuntu ubuntu 4096 Nov 10 17:22 sourcedata
drwxrwxr-x  5 ubuntu ubuntu 4096 Nov 10 17:07 sub-NDARINV137Y0WR8
drwxrwxr-x  5 ubuntu ubuntu 4096 Nov 10 17:12 sub-NDARINV41U1CNA9
drwxrwxr-x  5 ubuntu ubuntu 4096 Nov 10 17:07 sub-NDARINV6WH2N8P4
drwxrwxr-x  5 ubuntu ubuntu 4096 Nov 10 17:08 sub-NDARINVF15Z4WLL
drwxrwxr-x  5 ubuntu ubuntu 4096 Nov 10 17:12 sub-NDARINVPH0R4T6M
drwxrwxr-x  5 ubuntu ubuntu 4096 Nov 10 17:11 sub-NDARINVWHXTCM8L
lrwxrwxrwx  1 ubuntu ubuntu  122 Nov 10 17:12 task-MID_bold.json -> .git/annex/objects/38/FM/MD5E-s214--7ebfaf4ac323c1b36f970708087ce860.json/MD5E-s214--7ebfaf4ac323c1b36f970708087ce860.json
lrwxrwxrwx  1 ubuntu ubuntu  122 Nov 10 17:08 task-nback_bold.json -> .git/annex/objects/PZ/9w/MD5E-s247--e52424297b9c080d2c20971da4ad9f2e.json/MD5E-s247--e52424297b9c080d2c20971da4ad9f2e.json
(dev3) 1 36924.....................................:Tue 10 Nov 2020 12:35:58 PM EST:.
lena:/tmp
$> reproman execute --internal -r my-nitrc-gpu2 get abcd-bids-test4       
2020-11-10 12:36:18,817 [ERROR  ] [Errno 2] No such file [sftp_client.py:_convert_status:894] (FileNotFoundError) 
(dev3) 1 36925 ->1.....................................:Tue 10 Nov 2020 12:36:18 PM EST:.
lena:/tmp
$> reproman -l debug execute --internal -r my-nitrc-gpu2 get abcd-bids-test4
2020-11-10 12:36:24,818 [DEBUG  ] SSH connecting to ubuntu@54.197.2.255:22, authenticating with /home/yoh/.local/share/reproman/ec2_keys/my-nitrc-gpu2.pem 
2020-11-10 12:36:25,392 [ERROR  ] [Errno 17] File exists: 'abcd-bids-test4' [session.py:transfer_recursive:546] (FileExistsError) 
(dev3) 1 36926 ->1.....................................:Tue 10 Nov 2020 12:36:25 PM EST:.
lena:/tmp
$> rmdir abcd-bids-test4                                                    
(dev3) 1 36927.....................................:Tue 10 Nov 2020 12:36:28 PM EST:.
lena:/tmp
$> reproman -l debug execute --internal -r my-nitrc-gpu2 get abcd-bids-test4
2020-11-10 12:36:30,698 [DEBUG  ] SSH connecting to ubuntu@54.197.2.255:22, authenticating with /home/yoh/.local/share/reproman/ec2_keys/my-nitrc-gpu2.pem 
2020-11-10 12:36:31,383 [ERROR  ] [Errno 2] No such file [sftp_client.py:_convert_status:894] (FileNotFoundError) 

in above you can see that it does create the directory locally first (so that part seems to work) but then some file (no logging which could tell what it is trying to transfer even) is "not found" -- may be it is looking for target subdirectory or smth like that?