504 Server Error: Gateway Time-out when downloading data from large packages

[perronea]

The ABCC dataset (collection 3165) is 168.57TB. If I try to download the entire package with the command downloadcmd -dp ${package_id} it will begin downloading, but then cut off after about a minute and return the following error:

Traceback (most recent call last):
  File "/home/faird/perr0372/.local/bin/downloadcmd", line 8, in <module>
    sys.exit(main())
  File "/home/faird/perr0372/.local/lib/python3.8/site-packages/NDATools/clientscripts/downloadcmd.py", line 199, in main
    s3Download.start()
  File "/home/faird/perr0372/.local/lib/python3.8/site-packages/NDATools/Download.py", line 327, in start
    for package_file_id_list in self.generate_download_batch_file_ids():
  File "/home/faird/perr0372/.local/lib/python3.8/site-packages/NDATools/Download.py", line 595, in generate_download_batch_file_ids
    files = self.get_package_files_by_page(page, batch_size)
  File "/home/faird/perr0372/.local/lib/python3.8/site-packages/NDATools/Download.py", line 971, in get_package_files_by_page
    raise e
  File "/home/faird/perr0372/.local/lib/python3.8/site-packages/NDATools/Download.py", line 961, in get_package_files_by_page
    tmp = get_request(url, headers=self.request_header(), auth=self.auth, error_handler=HttpErrorHandlingStrategy.reraise_status)
  File "/home/faird/perr0372/.local/lib/python3.8/site-packages/NDATools/Utils.py", line 297, in get_request
    error_handler(tmp)
  File "/home/faird/perr0372/.local/lib/python3.8/site-packages/NDATools/Utils.py", line 74, in reraise_status
    response.raise_for_status()
  File "/panfs/roc/msisoft/anaconda/python3-2020.07-mamba/lib/python3.8/site-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 504 Server Error: Gateway Time-out for url: https://nda.nih.gov/api/package/1203969/files?page=2&size=50

I don't actually want to download the entire dataset at once, but rather I'd like to be able to download a single subject or a single filetype from all subjects using the --file-regex parameter. When I do this I get the same error as above.

I've tested this out on a much smaller package of around 80GB (essentially just a single submission from the collection) and it works just fine so I'm assuming this error is likely caused by the size of the data package that I am trying to download from.

Any ideas on how I can get around this issue so that users can easily download the specific data that they want from this collection?

[gregmagdits]

Hi Anders. I started looking at this today but I think its going to take some time to come up with a solution. I brought this up with the rest of the team so it's on our radar. I'll post an update when we schedule the work for this issue

[RyanZurrin]

Has there been any update on this issue? I am downloading significantly less data with a package size of < 300GB, and I am also facing a similar error:

Download Progress Report [May 02 2023 16:41:17]: 
    1700/1750 queued files downloaded so far. 
    Last 50 files contained ~ 1021.68MB bytes and finished in 0:00:32 (Hours:Minutes:Seconds). 
    Avg download rate (in bits per second) for the last 50 files is ~ 255.42Mbps.
    Download has been in progress for 0:11:52 (Hours:Minutes:Seconds).

Traceback (most recent call last):
  File "/home/ec2-user/.local/bin/downloadcmd", line 8, in <module>
    sys.exit(main())
  File "/home/ec2-user/.local/lib/python3.9/site-packages/NDATools/clientscripts/downloadcmd.py", line 200, in main
    s3Download.start()
  File "/home/ec2-user/.local/lib/python3.9/site-packages/NDATools/Download.py", line 352, in start
    file_id_to_cred_list = self.get_presigned_urls(list(package_file_id_list))
  File "/home/ec2-user/.local/lib/python3.9/site-packages/NDATools/Download.py", line 1043, in get_presigned_urls
    tmp = post_request(url,  payload=id_list, auth=self.auth,
  File "/home/ec2-user/.local/lib/python3.9/site-packages/NDATools/Utils.py", line 273, in post_request
    return _send_prepared_request(req.prepare(), timeout=timeout, deserialize_handler=deserialize_handler, error_handler=error_handler)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/NDATools/Utils.py", line 239, in _retry
    tmp = func(*args, **kwargs)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/NDATools/Utils.py", line 262, in _send_prepared_request
    error_handler(tmp)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/NDATools/Utils.py", line 84, in reraise_status
    response.raise_for_status()
  File "/usr/lib/python3.9/site-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 504 Server Error: Gateway Time-out for url: https://nda.nih.gov/api/package/1215110/files/batchGeneratePresignedUrls

Is there a fix for this yet?

[gregmagdits]

We are looking into this issue now and should have an update before the end of the week.

[RyanZurrin]

We are looking into this issue now and should have an update before the end of the week.

So today, I have tried again, with the only difference being I have set the --workerThreads to a value of 16 where, as before, I was not setting this at all.

Not sure if that had any effect, and it may have worked today without setting this, but I did not try today without setting this, so I do not know.

I feel the issue has nothing todo with this tool and is more an issue relating to the nda.nih.gov.

[RyanZurrin]

Maybe it is possible to make it so that if the timeout happens initially, it will retry a few times instead of just throwing the error as it does?

[gregmagdits]

We released a new version of the nda-tools on Friday that should fix these 504 errors (v. 0,2.25) . @perronea I know its been a while since this ticket was first created but with the new version of the tools you should be able to download package 1203969 without encountering 504 errors. Please let us know if this is not the case.