Modify crawler behaviours to actually download the files when crawling so MD5hash can be added to the git annex URL

[cmadjar]

Purpose

The crawler code for Zenodo and OSF need to be updated so that files get downloaded before they get annexed. Doing so will allow the storage of the MD5sum hash can be seen in the git annex object and we can later on link easily processed files in the execution records Mandana is working on.

[cmadjar]

The code has already been modified in https://github.com/CONP-PCNO/conp-dataset/pull/627/files#diff-4db1ecdec0c56275d97bd8a7d85769c0ad610d06cab82ef10124fa2cf6c7fab0L81 to download the files. All that is left is recrawling the datasets.

Here is a list of datasets that do not have the MD5 hash in the git annex object URL:

• CFMM_7T__MP2RAGE_T1_mapping
• Calgary_Preschool_MRI_Dataset
• Comparing_Perturbation_Modes_for_Evaluating_Instabilities_in_Neuroimaging__Processed_NKI_RS_Subset__082019
• Learning_Naturalistic_Structure__Processed_fMRI_dataset
• Multimodal_data_with_wide_field_GCaMP_imaging
• Numerically_Perturbed_Structural_Connectomes_from_100_individuals_in_the_NKI_Rockland_Dataset
• Quantifying_Neural_Cognitive_Relationships_Across_the_Brain
• VFA_T1_mapping_RTHawkopenvs_Siemenscommercial_

[joeyzhou98]

@cmadjar is this ok to close?

[cmadjar]

Yep :)