search

NDAR / nda-tools

Python package for interacting with NDA web services. Used to validate, submit, and download data to and from NDA.
MIT License
48 stars 22 forks source link

format of S3_path_list #29

Closed petralenzini closed 3 years ago

petralenzini commented 3 years ago

I'm trying to download just a subset of S3 links from the datastructure_manifest.txt of a particular package, per the downloadcmd -h instructions pasted here for reference in their entirety:

(abcd3.5.1) ubuntu@ip-172-31-47-9:~/spacedir$ downloadcmd -h Running NDATools Version 0.2.3 Opening log: /home/ubuntu/NDAValidationResults/debug_log_20210122T165559.txt usage: downloadcmd

This application allows you to enter a list of aws S3 paths and will download the files to your drive in your home folder. Alternatively, you may enter a packageID, an NDA data structure file or a text file with s3 links, and the client will download all files from the S3 links listed. Please note, the maximum transfer limit of data is 5TB at one time.

positional arguments:

Will download all S3 files to your local drive optional arguments: -h, --help show this help message and exit -dp, --package Flags to download all S3 files in package. -t, --txt Flags that a text file has been entered from where to download S3 files. -ds, --datastructure Flags that a data structure text file has been entered from where to download S3 files. -u , --username NDA username -p , --password NDA password -r , --resume Flags to restart a download process. If you already have some files downloaded, you must enter the directory where they are saved. -d , --directory Enter an alternate full directory path where you would like your files to be saved. -wt , --workerThreads Number of worker threads -v, --verbose Option to print out more detailed messages as the program runs. My S3_list in the command below is a text file that contains a list of S3 links: s3://NDAR_Central_3/submission_32142/4088_01_MR/unprocessed/T1w_MPR/4088_01_MR_T1w_MPR.json s3://NDAR_Central_3/submission_32142/4088_01_MR/unprocessed/T1w_MPR/4088_01_MR_T1w_MPR.nii.gz s3://NDAR_Central_3/submission_32142/4088_01_MR/unprocessed/T1w_MPR/OTHER_FILES/4088_01_MR_BIAS_64CH.json **Using such a list results in a downloaded folder of nothing, and a 'list index out of range' message. Any ideas on what might be missing?** (abcd3.5.1) ubuntu@ip-172-31-47-9:~/spacedir$ downloadcmd S3_list -d HCPEPReleaseDLS3listtest -v -u (myusername) -p (mypassword) Running NDATools Version 0.2.3 Opening log: /home/ubuntu/NDAValidationResults/debug_log_20210122T164810.txt list index out of range Finished downloading all files. (abcd3.5.1) ubuntu@ip-172-31-47-9:~/spacedir$ less /home/ubuntu/NDAValidationResults/debug_log_20210122T164810.txt (abcd3.5.1) ubuntu@ip-172-31-47-9:~/spacedir$ cat /home/ubuntu/NDAValidationResults/debug_log_20210122T164810.txt 2021-01-22 16:48:10,920:INFO:request to generate AWS token (abcd3.5.1) ubuntu@ip-172-31-47-9:~/spacedir$
petralenzini commented 3 years ago

okay, so I guess its not a positional argument...using the -t flag before the S3_list works. Any chance you want to update the the instructions?

petralenzini commented 3 years ago

This works, too. Neither are issues, I suppose. Just a question of understanding the documentation.

downloadcmd s3://NDAR_Central_3/submission_32142/4088_01_MR/unprocessed/T1w_MPR/4088_01_MR_T1w_MPR.json s3://NDAR_Central_3/submission_32142/4088_01_MR/unprocessed/T1w_MPR/4088_01_MR_T1w_MPR.nii.gz -d HCPEPReleaseS3test -v -u plenzini -p