search

DCAN-Labs / abcd-hcp-pipeline

bids application for processing functional MRI data, robust to scanner, acquisition and age variability.
https://hub.docker.com/r/dcanumn/abcd-hcp-pipeline
BSD 3-Clause "New" or "Revised" License
50 stars 19 forks source link

Files not available for download #35

Closed mriedel56 closed 3 years ago

mriedel56 commented 3 years ago

Sorry if this is the wrong place for this comment. We are downloading files from Collection 3165 - DCAN Labs ABCD-BIDS, and we have alot of instances of files that do not download (for example, event timing files, or ABCD-HCP processed MNI bold images); we also have alot of participants that completely download as expected. I dont think its an issue with the downloader, and I imagine the participants made available in the collection passed the expected output stage. Maybe some links in the manifest are incorrect? I have a text file for each file type with the failed subjects if it would help give some examples of the problem.

arueter1 commented 3 years ago

Thanks for reaching out! We will get back to you about this soon.

mriedel56 commented 3 years ago

What I can tell from a couple examples, the S3 links for some files are not in the manifest.

ericfeczko commented 3 years ago

@perronea Can we take a look into this and see what might be going on here? @mriedel56 The examples would help; unfortunately, we cannot share subject information here. We can reach out via email to communicate the examples so we can take a look from our end. Thanks for catching this!

mriedel56 commented 3 years ago

Thanks for the response! I created a sort of "pseudo-manifest" that contains the missing information/links based on examples in the original. For example, if one participant was missing the *space-MNI_bold.nii.gz derivative, but had the motion.tsv files, I would recreate a line in the manifest using the same dataset ID/collection/manifest/etc. with the appropriate S3 link, but it returned an error as if the file does not exist on the S3 server. Happy to continue working on this off thread. miriedel@fiu.edu.

perronea commented 3 years ago

@mriedel56 it looks like that data is indeed missing. We've had some issues with the NDA upload process that we are currently working on resolving with them. Thank you for bringing this to our attention. I'm going to close the issue here and open a new one on the nda-abcd-s3-downloader (https://github.com/DCAN-Labs/nda-abcd-s3-downloader/issues/24)