NDAR / nda-tools

Python package for interacting with NDA web services. Used to validate, submit, and download data to and from NDA.
MIT License
47 stars 22 forks source link

S3 download is broken #67

Closed liningpan closed 7 months ago

liningpan commented 1 year ago

The feature for download data to S3 is broken

The stack trace points to this newly added function. It seems like download_from_s3link transferred the file to S3, but later the script assumes the file is on the local filesystem.

Traceback (most recent call last):
  File "/home/ec2-user/nda-tools/bin/downloadcmd", line 8, in <module>
    sys.exit(main())
  File "/home/ec2-user/nda-tools/lib/python3.7/site-packages/NDATools/clientscripts/downloadcmd.py", line 200, in main
    s3Download.start()
  File "/home/ec2-user/nda-tools/lib/python3.7/site-packages/NDATools/Download.py", line 198, in start
    self.download_package_metadata_file()
  File "/home/ec2-user/nda-tools/lib/python3.7/site-packages/NDATools/Download.py", line 889, in download_package_metadata_file
    with gzip.open(download_location, 'rb') as f_in:
  File "/usr/lib64/python3.7/gzip.py", line 58, in open
    binary_file = GzipFile(filename, gz_mode, compresslevel)
  File "/usr/lib64/python3.7/gzip.py", line 168, in __init__
    fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')

https://github.com/NDAR/nda-tools/blob/aa89ec26de21f04679dec702c054f19ef48726d4/NDATools/Download.py#L884-L889

liningpan commented 11 months ago

Friendly ping here. Downloading to S3 is broken for us for a long time, even though this is the method recommended by NDA. We had to find workarounds, develop our own tool, and stop using nda-tools.

gregmagdits commented 7 months ago

This was fixed in version 0.2.26