search

fiuneuro / cis-processing

Automated transfer, conversion, and processing of data acquired at the FIU Center for Imaging Science.
Apache License 2.0
4 stars 4 forks source link

Project memory #19

Open mriedel56 opened 6 years ago

mriedel56 commented 6 years ago

Should we keep a running tally of project memory? i.e., raw storage, dset storage, derivatives storage (and subsequent folders within derivatives like mriqc, fmriprep, etc). could be more beneficial for more accurate estimates/reporting for storage needs in grants

tsalo commented 6 years ago

I think that makes sense. What are you thinking? How about:

We can extend to include projects from other labs as well.

mriedel56 commented 6 years ago

I was mainly thinking about totals for each project, but yeah, means would also be good.

tsalo commented 6 years ago

I've got code working for this, and would like to add it to this repository so others can use it, but want to check how other labs plan to organize their datasets and derivatives. I am assuming that the derivatives (MRIQC, Freesurfer, and fMRIPrep) will go outside the BIDS datasets. The cis-processing code outputs the MRIQC derivatives to a folder inside the BIDS dataset (e.g., /home/data/lab/project/dset/derivatives/), but I would like to change that to a folder outside (e.g., /home/data/lab/project/derivatives/). How does that sound? Tagging @anthonystevendick @mattfeld and @adamkimbler for your thoughts.

Given that one current plan is to only download dicoms from XNAT only for conversion to BIDS format (and then to clean up the tar files after), my code currently doesn't look for dicoms.

anthonystevendick commented 6 years ago

It would be better to have the bids folder self-contained because it may need to be uploaded to the NIH data archive. So having the downstream process folders outside the bids folder is better.

Anthony

-- Anthony Steven Dick, Ph.D. Associate Professor Director, Cognitive Neuroscience Program and Graduate Certificate in Cognitive Neuroscience Florida International University MM Campus AHC4 454 Department of Psychology and Center for Children and Familieshttp://www.ccf.fiu.edu/, an FIU Preeminent Programhttps://beyondpossible.fiu.edu/preeminent-programs/ 11200 S.W. 8th Street Miami, FL 33199 Ph: 305-348-4202; Lab Ph: 305-348-9055; Fx: 305-348-3879 Email: adick@fiu.edumailto:adick@fiu.edu; Webpage: http://dcn.fiu.eduhttp://dcn.fiu.edu/; Join the SSHDhttp://www.sshdonline.org/

Check out the new bookhttp://www.routledge.com/9781138960039: Dick, A.S., & Müller, U. (2018). Advancing Developmental Science: Philosophy, Theory, and Method. Taylor & Francis.

From: Taylor Salo notifications@github.com Reply-To: FIU-Neuro/cis-processing reply@reply.github.com Date: Saturday, September 15, 2018 at 2:00 PM To: FIU-Neuro/cis-processing cis-processing@noreply.github.com Cc: Anthony Dick adick@fiu.edu, Mention mention@noreply.github.com Subject: Re: [FIU-Neuro/cis-processing] Project memory (#19)

I've got code working for this, and would like to add it to this repository so others can use it, but want to check how other labs plan to organize their datasets and derivatives. I am assuming that the derivatives (MRIQC, Freesurfer, and fMRIPrep) will go outside the BIDS datasets. The cis-processing code outputs the MRIQC derivatives to a folder inside the BIDS dataset (e.g., /home/data/lab/project/dset/derivatives/), but I would like to change that to a folder outside (e.g., /home/data/lab/project/derivatives/). How does that sound? Tagging @anthonystevendickhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_anthonystevendick&d=DwMFaQ&c=lhMMI368wojMYNABHh1gQQ&r=DmdQWjwCJRFVHJCqLOrjwA&m=ROgQjLKjnMADUTf4SP9Pw0_Z9_2XuWakApmopQI-7Mg&s=9VFxwiK9C7UXEW5AnVOyIEDdXK4M5Zp4pPxukidnLmU&e= @mattfeldhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_mattfeld&d=DwMFaQ&c=lhMMI368wojMYNABHh1gQQ&r=DmdQWjwCJRFVHJCqLOrjwA&m=ROgQjLKjnMADUTf4SP9Pw0_Z9_2XuWakApmopQI-7Mg&s=72aOMl88bYsxTcf5oUoVprWIUFiMQwhfqGQn614Yi6A&e= and @adamkimblerhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_adamkimbler&d=DwMFaQ&c=lhMMI368wojMYNABHh1gQQ&r=DmdQWjwCJRFVHJCqLOrjwA&m=ROgQjLKjnMADUTf4SP9Pw0_Z9_2XuWakApmopQI-7Mg&s=pchHHHVnFFRCsng-z1Ic-mh4DtpLel-o0xLbaVlTfmo&e= for your thoughts.

Given that one current plan is to only download dicoms from XNAT only for conversion to BIDS format (and then to clean up the tar files after), my code currently doesn't look for dicoms.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_FIU-2DNeuro_cis-2Dprocessing_issues_19-23issuecomment-2D421608154&d=DwMFaQ&c=lhMMI368wojMYNABHh1gQQ&r=DmdQWjwCJRFVHJCqLOrjwA&m=ROgQjLKjnMADUTf4SP9Pw0_Z9_2XuWakApmopQI-7Mg&s=GsD7BPEGdTexcg5HHiJQ36g3Y-S4r_u2pwlHEubImaI&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AeBVSe5LNT-2DLZnLBWntratXLxImXAP-5FYks5ubUAWgaJpZM4WhJ0q&d=DwMFaQ&c=lhMMI368wojMYNABHh1gQQ&r=DmdQWjwCJRFVHJCqLOrjwA&m=ROgQjLKjnMADUTf4SP9Pw0_Z9_2XuWakApmopQI-7Mg&s=nBThDU7uihccvKH645UYGOpF86ZsTZ4gzgtFILTeY3E&e=.

tsalo commented 6 years ago

@anthonystevendick That seems like a good rationale to keep them separate. Currently, we output to within the dataset folder, so this will require a change to cis_proc.py, but won't affect the BIDSifier image.

tsalo commented 4 years ago

I think we should move this to a separate repository. Something like cis-tracking. The check could be set up in separate cron jobs.