Closed perronea closed 8 months ago
Hi Anders. I started looking at this today but I think its going to take some time to come up with a solution. I brought this up with the rest of the team so it's on our radar. I'll post an update when we schedule the work for this issue
Has there been any update on this issue? I am downloading significantly less data with a package size of < 300GB, and I am also facing a similar error:
Download Progress Report [May 02 2023 16:41:17]:
1700/1750 queued files downloaded so far.
Last 50 files contained ~ 1021.68MB bytes and finished in 0:00:32 (Hours:Minutes:Seconds).
Avg download rate (in bits per second) for the last 50 files is ~ 255.42Mbps.
Download has been in progress for 0:11:52 (Hours:Minutes:Seconds).
Traceback (most recent call last):
File "/home/ec2-user/.local/bin/downloadcmd", line 8, in <module>
sys.exit(main())
File "/home/ec2-user/.local/lib/python3.9/site-packages/NDATools/clientscripts/downloadcmd.py", line 200, in main
s3Download.start()
File "/home/ec2-user/.local/lib/python3.9/site-packages/NDATools/Download.py", line 352, in start
file_id_to_cred_list = self.get_presigned_urls(list(package_file_id_list))
File "/home/ec2-user/.local/lib/python3.9/site-packages/NDATools/Download.py", line 1043, in get_presigned_urls
tmp = post_request(url, payload=id_list, auth=self.auth,
File "/home/ec2-user/.local/lib/python3.9/site-packages/NDATools/Utils.py", line 273, in post_request
return _send_prepared_request(req.prepare(), timeout=timeout, deserialize_handler=deserialize_handler, error_handler=error_handler)
File "/home/ec2-user/.local/lib/python3.9/site-packages/NDATools/Utils.py", line 239, in _retry
tmp = func(*args, **kwargs)
File "/home/ec2-user/.local/lib/python3.9/site-packages/NDATools/Utils.py", line 262, in _send_prepared_request
error_handler(tmp)
File "/home/ec2-user/.local/lib/python3.9/site-packages/NDATools/Utils.py", line 84, in reraise_status
response.raise_for_status()
File "/usr/lib/python3.9/site-packages/requests/models.py", line 943, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 504 Server Error: Gateway Time-out for url: https://nda.nih.gov/api/package/1215110/files/batchGeneratePresignedUrls
Is there a fix for this yet?
We are looking into this issue now and should have an update before the end of the week.
We are looking into this issue now and should have an update before the end of the week.
So today, I have tried again, with the only difference being I have set the --workerThreads to a value of 16 where, as before, I was not setting this at all.
Not sure if that had any effect, and it may have worked today without setting this, but I did not try today without setting this, so I do not know.
I feel the issue has nothing todo with this tool and is more an issue relating to the nda.nih.gov.
Maybe it is possible to make it so that if the timeout happens initially, it will retry a few times instead of just throwing the error as it does?
We released a new version of the nda-tools on Friday that should fix these 504 errors (v. 0,2.25) . @perronea I know its been a while since this ticket was first created but with the new version of the tools you should be able to download package 1203969 without encountering 504 errors. Please let us know if this is not the case.
The ABCC dataset (collection 3165) is 168.57TB. If I try to download the entire package with the command
downloadcmd -dp ${package_id}
it will begin downloading, but then cut off after about a minute and return the following error:I don't actually want to download the entire dataset at once, but rather I'd like to be able to download a single subject or a single filetype from all subjects using the
--file-regex
parameter. When I do this I get the same error as above.I've tested this out on a much smaller package of around 80GB (essentially just a single submission from the collection) and it works just fine so I'm assuming this error is likely caused by the size of the data package that I am trying to download from.
Any ideas on how I can get around this issue so that users can easily download the specific data that they want from this collection?