search

NDAR / nda-tools

Python package for interacting with NDA web services. Used to validate, submit, and download data to and from NDA.
MIT License
48 stars 22 forks source link

using `downloadcmd` doesn't work #46

Closed dyhan316 closed 2 years ago

dyhan316 commented 2 years ago

Taking a hint from https://github.com/ABCD-ReproNim/exercises/blob/main/project_month_prep/downloadMRI.md , I have tried two commands below to download a data package in NIHM, but have failed with different errors.

Attempt 1 :

tried : downloadcmd -dp XX -u XXX -p XXX -wt 8 -v (-dp, -u, -p were changed with XXX for security reasons)

got : 

Running NDATools Version 0.2.18
usage: downloadcmd <S3_path_list>
downloadcmd: error: unrecognized arguments: -wt 8 -v

Attempt 2 :

tried : downloadcmd -dp XXX got : 

Running NDATools Version 0.2.18
-u/--username argument not provided. Using default value of 'XXX' which was saved in /home/connectome/dyhan316/.NDATools/settings.cfg
Traceback (most recent call last):
  File "/home/connectome/dyhan316/.local/bin/downloadcmd", line 8, in <module>
    sys.exit(main())
  File "/home/connectome/dyhan316/.local/lib/python3.8/site-packages/NDATools/clientscripts/downloadcmd.py", line 187, in main
    config = configure(args)
  File "/home/connectome/dyhan316/.local/lib/python3.8/site-packages/NDATools/clientscripts/downloadcmd.py", line 177, in configure
    config.read_user_credentials()
  File "/home/connectome/dyhan316/.local/lib/python3.8/site-packages/NDATools/Configuration.py", line 164, in read_user_credentials
    self.password = keyring.get_password(self.SERVICE_NAME, self.username)
  File "/usr/lib/python3/dist-packages/keyring/core.py", line 57, in get_password
    return _keyring_backend.get_password(service_name, username)
  File "/usr/lib/python3/dist-packages/keyring/backends/SecretService.py", line 64, in get_password
    collection = self.get_preferred_collection()
  File "/usr/lib/python3/dist-packages/keyring/backends/SecretService.py", line 58, in get_preferred_collection
    raise KeyringLocked("Failed to unlock the collection!")
keyring.errors.KeyringLocked: Failed to unlock the collection!

I tried to resolve the keyring issue by looking at : https://pypi.org/project/keyring/ and using keyring.set_password('nda-tools', 'YOUR_USERNAME', 'NEW_PASSWORD') command, but got 

>>> import keyring
>>> keyring.set_password('nda-tools', 'XXX', 'XXX')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/keyring/core.py", line 63, in set_password
    _keyring_backend.set_password(service_name, username, password)
  File "/usr/lib/python3/dist-packages/keyring/backends/SecretService.py", line 77, in set_password
    collection = self.get_preferred_collection()
  File "/usr/lib/python3/dist-packages/keyring/backends/SecretService.py", line 58, in get_preferred_collection
    raise KeyringLocked("Failed to unlock the collection!")
keyring.errors.KeyringLocked: Failed to unlock the collection!
>>>

(the usernames and password was replaced with XXX for security reasons)

Additional Information

I realized that when I tired downloading ada-tools, I got the following ERROR:

(download_ABCD) dyhan316@node1:/storage/bigdata/CHA_bigdata/ABCD_release4$ pip install nda-tools
Collecting nda-tools
  Using cached https://files.pythonhosted.org/packages/fc/fb/f4261be968916758294db6336afac6d94b36cca5121c42ce2da3acefe8d3/nda_tools-0.2.18-py3-none-any.whl
Collecting botocore (from nda-tools)
  Using cached https://files.pythonhosted.org/packages/f1/3c/7bf0f6fabc9cb05758b10fe8c688d3f0540bb899abd1cdc4072242ee0b73/botocore-1.27.55-py3-none-any.whl
Requirement already satisfied: packaging in /usr/anaconda3/lib/python3.7/site-packages (from nda-tools) (19.2)
Requirement already satisfied: requests in /usr/anaconda3/lib/python3.7/site-packages (from nda-tools) (2.22.0)
Requirement already satisfied: tqdm in /usr/anaconda3/lib/python3.7/site-packages (from nda-tools) (4.36.1)
Requirement already satisfied: pyyaml in /usr/anaconda3/lib/python3.7/site-packages (from nda-tools) (5.1.2)
Collecting boto3 (from nda-tools)
  Using cached https://files.pythonhosted.org/packages/9e/78/ecc8c65153a427ff952b868af287f0c9d7f4b269edd21316f05e6ce194b0/boto3-1.24.55-py3-none-any.whl
Requirement already satisfied: keyring in /usr/anaconda3/lib/python3.7/site-packages (from nda-tools) (18.0.0)
Requirement already satisfied: mock in /usr/anaconda3/lib/python3.7/site-packages (from nda-tools) (3.0.5)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/anaconda3/lib/python3.7/site-packages (from botocore->nda-tools) (2.8.0)
Collecting jmespath<2.0.0,>=0.7.1 (from botocore->nda-tools)
  Using cached https://files.pythonhosted.org/packages/31/b4/b9b800c45527aadd64d5b442f9b932b00648617eb5d63d2c7a6587b7cafc/jmespath-1.0.1-py3-none-any.whl
Collecting urllib3<1.27,>=1.25.4 (from botocore->nda-tools)
  Using cached https://files.pythonhosted.org/packages/d1/cb/4783c8f1a90f89e260dbf72ebbcf25931f3a28f8f80e2e90f8a589941b19/urllib3-1.26.11-py2.py3-none-any.whl
Requirement already satisfied: six in /usr/anaconda3/lib/python3.7/site-packages (from packaging->nda-tools) (1.12.0)
Requirement already satisfied: pyparsing>=2.0.2 in /usr/anaconda3/lib/python3.7/site-packages (from packaging->nda-tools) (2.4.2)
Requirement already satisfied: certifi>=2017.4.17 in /usr/anaconda3/lib/python3.7/site-packages (from requests->nda-tools) (2019.9.11)
Requirement already satisfied: idna<2.9,>=2.5 in /usr/anaconda3/lib/python3.7/site-packages (from requests->nda-tools) (2.8)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/anaconda3/lib/python3.7/site-packages (from requests->nda-tools) (3.0.4)
Collecting s3transfer<0.7.0,>=0.6.0 (from boto3->nda-tools)
  Using cached https://files.pythonhosted.org/packages/5e/c6/af903b5fab3f9b5b1e883f49a770066314c6dcceb589cf938d48c89556c1/s3transfer-0.6.0-py3-none-any.whl
Requirement already satisfied: entrypoints in /usr/anaconda3/lib/python3.7/site-packages (from keyring->nda-tools) (0.3)
Requirement already satisfied: secretstorage; sys_platform == "linux" and python_version >= "3.5" in /usr/anaconda3/lib/python3.7/site-packages (from keyring->nda-tools) (3.1.1)
Requirement already satisfied: cryptography in /usr/anaconda3/lib/python3.7/site-packages (from secretstorage; sys_platform == "linux" and python_version >= "3.5"->keyring->nda-tools) (2.7)
Requirement already satisfied: jeepney in /usr/anaconda3/lib/python3.7/site-packages (from secretstorage; sys_platform == "linux" and python_version >= "3.5"->keyring->nda-tools) (0.4.1)
Requirement already satisfied: asn1crypto>=0.21.0 in /usr/anaconda3/lib/python3.7/site-packages (from cryptography->secretstorage; sys_platform == "linux" and python_version >= "3.5"->keyring->nda-tools) (1.0.1)
Requirement already satisfied: cffi!=1.11.3,>=1.8 in /usr/anaconda3/lib/python3.7/site-packages (from cryptography->secretstorage; sys_platform == "linux" and python_version >= "3.5"->keyring->nda-tools) (1.12.3)
Requirement already satisfied: pycparser in /usr/anaconda3/lib/python3.7/site-packages (from cffi!=1.11.3,>=1.8->cryptography->secretstorage; sys_platform == "linux" and python_version >= "3.5"->keyring->nda-tools) (2.19)
ERROR: requests 2.22.0 has requirement urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1, but you'll have urllib3 1.26.11 which is incompatible.
Installing collected packages: jmespath, urllib3, botocore, s3transfer, boto3, nda-tools
ERROR: Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/usr/anaconda3/lib/python3.7/site-packages/jmespath-1.0.1.dist-info'
Consider using the `--user` option or check the permissions.

Could this be the reason for the aforementioned errors? Should I ask the server manager to install nda-tools using root user privileges?

Could anyone please lend a help? Thank you!