search

NDAR / nda-tools

Python package for interacting with NDA web services. Used to validate, submit, and download data to and from NDA.
MIT License
47 stars 22 forks source link

KeyError from mis-parsed URI during Submission #59

Closed jacob-lee closed 1 year ago

jacob-lee commented 1 year ago

When uploading images and after successful validation of my image03 csv(s), I was getting the following exception:

Traceback (most recent call last):
  File "/home/jacoblee/miniconda3/envs/ndar/bin/vtcmd", line 8, in <module>
    sys.exit(main())
  File "/home/jacoblee/miniconda3/envs/ndar/lib/python3.8/site-packages/NDATools/clientscripts/vtcmd.py", line 442, in main
    submit_package(package_id=package_id, full_file_path=full_file_path,
  File "/home/jacoblee/miniconda3/envs/ndar/lib/python3.8/site-packages/NDATools/clientscripts/vtcmd.py", line 325, in submit_package
    submission.submission_upload(hide_progress=config.hideProgress)
  File "/home/jacoblee/miniconda3/envs/ndar/lib/python3.8/site-packages/NDATools/Submission.py", line 366, in submission_upload
    self.batch_update_status(status=Status.PROCESSING)
  File "/home/jacoblee/miniconda3/envs/ndar/lib/python3.8/site-packages/NDATools/Submission.py", line 326, in batch_update_status
    list_data = self.generate_data_for_request(status)
  File "/home/jacoblee/miniconda3/envs/ndar/lib/python3.8/site-packages/NDATools/Submission.py", line 143, in generate_data_for_request
    size = self.full_file_path[file][1]
KeyError: 'submission_54752/980602.T1I-0000529-001.002-000011-121947.ima'

I ran the code through a debugger and identified the full_file_path dictionary, and the keys were not prefixed with submission_54752/ (the exact number depends on which submission attempt of course). I noticed that the destination_uri in cred was s3://NDAR_Central_2/NDAR_Central_2/submission_54752/980602.T1I-0000529-001.002-000011-121947.ima, and that the code:

file = cred['destination_uri'].split('/')
file = '/'.join(file[4:])

was causing the mismatch. I was able to fix and successfully submit by changing the code to read file[5:].

However, those two lines of code haven't changed in several years, and I was able to submit successfully for another grant only a couple of weeks ago. I can only assume that the destination_uri changed between submissions, with an extra / put in somewhere. I don't think that I did anything differently on my end (I largely re-use the same code).

gregmagdits commented 1 year ago

I created an internal work ticket to investigate the cause of this issue. I did a quick check but nothing in-particular stood out to me for submission 54752. Have you tried resuming the submission yet by running 'vtcmd -r 54752' ? if so, do you get the same KeyError?

kuhnmanu commented 1 year ago

Hi,

Just ran in the same issue, I think (### are replacements for the actual pathnames). I successfully uploaded data last week the same way for a different data collection.

Preparing to upload associated files. Traceback (most recent call last): File "/###/apps/anaconda3/bin/vtcmd", line 8, in sys.exit(main()) File "/###/apps/anaconda3/lib/python3.8/site-packages/NDATools/clientscripts/vtcmd.py", line 442, in main submit_package(package_id=package_id, full_file_path=full_file_path, File "/###/apps/anaconda3/lib/python3.8/site-packages/NDATools/clientscripts/vtcmd.py", line 325, in submit_package submission.submission_upload(hide_progress=config.hideProgress) File "/###/apps/anaconda3/lib/python3.8/site-packages/NDATools/Submission.py", line 364, in submission_upload self.batch_update_status(status=Status.PROCESSING) File "/###/apps/anaconda3/lib/python3.8/site-packages/NDATools/Submission.py", line 324, in batch_update_status list_data = self.generate_data_for_request(status) File "/###/apps/anaconda3/lib/python3.8/site-packages/NDATools/Submission.py", line 141, in generate_data_for_request size = self.full_file_path[file][1] KeyError: 'submission_54823/nifti/###/###/###_26.nii.gz'

gregmagdits commented 1 year ago

Other users are reporting this same error as well in our Help desk. We are going to start an investigation into this now. We'll post updates to this ticket as we make some progress

gregmagdits commented 1 year ago

We believe we found and fixed the source of the issue. @jacob-lee / @kuhnmanu , can you both try to resume your submissions now using the -r flag and post here if you continue to see errors?

kuhnmanu commented 1 year ago

just tried, got the following message:

`An unexpected error was encountered and the program could not continue. Error message from service was: None

Exit signal received, shutting down... `

gregmagdits commented 1 year ago

@kuhnmanu - can you increase the verbosity of the logging by updating your ~/.NDATools/logging.yml file? I attached a zip here that contains a sample logging.yml file that you can use to accomplish this. After you update the logging.yml file, please re-run the command and email the generated log file to ndahelp@mail.nih.gov. Sending an email to that address automatically opens a support ticket in our Help Desk system and someone (likely myself) will be assigned to the ticket and from there we can troubleshoot the issue.

logging.zip

jacob-lee commented 1 year ago

It ran successfully for me. I uninstalled my altered version of nda-tools, and re-installed from PyPi, and everything worked just fine, i.e. my hack was no longer required to submit successfully.

Note, I did not try resuming a previous submission package.

gregmagdits commented 1 year ago

The root cause was a data-issue which was addressed at EOD yesterday. The problem affected all submissions created between 1/6/2022@4:30EST and 1/9/2022@5:00EST. We have fixed the records for the affected submissions and users who were observing this error in the past couple of days should now be able to use the tool without issue.