404 Client Error:

[roohy466]

Hi There ,

I could run the code few days ago! but now gives me this error! I have downloaded some subjects!

      Running NDATools Version 0.2.25

      No value specified for --workerThreads. Using the default option of 19
      Important - You can configure the thread count setting using the --workerThreads argument to maximize your download speed.

      Getting Package Information...

      Package-id: 1184998
      Name: HCPAgingAllFiles
      Has associated files?: Yes
      Number of files in package: 2
      Total Package Size: 22.33TB

      Downloading S3 links from data structure: fmriresults01

      S3 links for files that failed to download will be written out to /Users/user/NDA/nda-tools/downloadcmd/logs/failed_s3_links_file_20230802T175503.txt. You can attempt to download these files later by running: 
          downloadcmd -dp 1184998 --file-regex .*/T1w/HCA8358689_V1_MR/surf/rh.white -u username -d /Users/user/NDA/nda-tools/downloadcmd/packages/1184998 -wt 19 -t "/Users/user/NDA/nda-tools/downloadcmd/logs/failed_s3_links_file_20230802T175503.txt"

      Beginning download of 10 files (27.83MB) matching .*/T1w/HCA8358689_V1_MR/surf/rh.white to /Users/user/NDA/nda-tools/downloadcmd/packages/1184998 using 19 threads
      Adding 10 files to download queue. Queue contains 10 files

      Traceback (most recent call last):
        File "/Users/user/python-envs/fastsurfer/bin/downloadcmd", line 8, in <module>
          sys.exit(main())
        File "/Users/user/python-envs/fastsurfer/lib/python3.9/site-packages/NDATools/clientscripts/downloadcmd.py", line 200, in main
          s3Download.start()
        File "/Users/user/python-envs/fastsurfer/lib/python3.9/site-packages/NDATools/Download.py", line 347, in start
          file_id_to_cred_list = self.get_presigned_urls(list(tmp_id_pkfiles.keys()))
        File "/Users/user/python-envs/fastsurfer/lib/python3.9/site-packages/NDATools/Download.py", line 979, in get_presigned_urls
          response = post_request(url,  payload=id_list, auth=self.auth,
        File "/Users/user/python-envs/fastsurfer/lib/python3.9/site-packages/NDATools/Utils.py", line 276, in post_request
          return _send_prepared_request(req.prepare(), timeout=timeout, deserialize_handler=deserialize_handler, error_handler=error_handler)
        File "/Users/user/python-envs/fastsurfer/lib/python3.9/site-packages/NDATools/Utils.py", line 239, in _retry
          tmp = func(*args, **kwargs)
        File "/Users/user/python-envs/fastsurfer/lib/python3.9/site-packages/NDATools/Utils.py", line 265, in _send_prepared_request
          error_handler(tmp)
        File "/Users/user/python-envs/fastsurfer/lib/python3.9/site-packages/NDATools/Utils.py", line 84, in reraise_status
          response.raise_for_status()
        File "/Users/user/python-envs/fastsurfer/lib/python3.9/site-packages/requests/models.py", line 1021, in raise_for_status
          raise HTTPError(http_error_msg, response=self)
      requests.exceptions.HTTPError: 404 Client Error:  for url: https://nda.nih.gov/api/package/1184998/files/batchGeneratePresignedUrls

[kjamison]

I also get this error. I have downloaded data successfully in the past.

[aporter1350]

I am also experiencing this problem with packages that used to be able to download data @gregmagdits can you help with this issue?

[liningpan]

Have you guys tried creating new data packages instead of using existing, shared ones. When I query the package/<package_id>/files API endpoint for this shared package 1184998, it doesn't return any actual data files, which could explain 404. Otherwise, this API endpoint is working fine for us for new data packages we created recently.

[aporter1350]

I just made a new package but it seems to be frozen on the Getting Package Information. Do you have any recommendations? The package is only 1TB. Using the command downloadcmd -dp 1219122 -ds fmriresults01 -u XXXX -d /Desktop/task_data/ -wt 100 (I have tried varying the worker threads 10-100)

[roohy466]

I have the same problem even with a new package id!

[gregmagdits]

Hello, We believe we have resolved the source of the 404 errors. To continue your download, please try either one of the following 2 options:

1) Delete the files “package_file_metadata.txt” and “package_file_metadata.gz.txt” from the download directory and rerun the downloadcmd. 2) Create a new package in the nda web app and run the downloadcmd with the new package-id.

If you continue to see 404 errors after completing step 1 then we recommend trying step 2 since the cause of the error is likely due to incorrect transcription of the package-file-metadata.txt.gz file during package creation. If you continue to see 404 errors after trying steps 1 and 2, please let us know and we will continue to troubleshoot on our end.

Thanks

[aporter1350]

Hi @gregmagdits

I created a new package yesterday night, made sure all metadata files were removed but I still get a 504 error. This particular package is 1TB, can you help?

[gregmagdits]

Hi Alexis - I will troubleshoot and provide the details on your open Help Desk ticket since the error here seems to be different from the rest of the 404 errors

[roohy466]

I could download the file now! without deleting anything! 👍 Thanks. I can close the issue now! :)