search

AMP-SCZ / lochness

Download your data to a data lake.
Other
3 stars 1 forks source link

Changes from Pronet data aggregation server #113

Closed kcho closed 1 year ago

kcho commented 1 year ago

Summary of changes:

  1. lochness.xnat module to save .zip files rather than the raw dicom directory. Saving a large number of dicom files under PHOENIX significantly delayed aws sync step. AMPSCZ-yaxil was updated along with this change. https://github.com/AMP-SCZ/yaxil/commit/094fd3464e8de2922cfe902b3cc7452a24e19b19

  2. lochness.redcap now has the functions to be used in the future for the repeated instrument measure. These functions are not called in the current sync.py pipeline, but we will use these functions once DPACC decides to move on with the repeated instrument measure.

  3. lochness.transfer module updated to only sync the data files selected by the cmdline options. eg) if box is selected as the data source when executing the sync.py, only the datatypes included in the box will get synced. This update is required for faster testing and parallelization in the future version.

  4. lochness.redcap also saves the redcap metadata as a csv file.

kcho commented 1 year ago

@tashrifbillah This is the working version of lochness at the Pronet aggregation server. It includes all changes not added to the master branch. Your quick review would be very helpful. Thanks.

kcho commented 1 year ago

This PR will be merged to master by the end of next week.