[x] for thursday: add subclassing for functional qc class. Deliverable: a "class" functional_qc that performs the same behavior as the qc now, but implements a classical class structure in the current set of functions' places. new class. Note that if you run the demo, the qa/ directory will have a "...pkl" file. If you call:
import ndmg
from ndmg.stats.qa_func import qa_func
test = qa_func.load('...pkl')
you will find that test has several attributes that will be of use for group level qc down the line.
[x] for Monday: analyze existing pipelines to compare mnr scores. Particularly, analyze on 5 datasets using EPI-reg method, and 5 datasets using the local-optimisation cost function. LOL I wish. triple threading on cortex (ie, maximum allocation) takes ~3 days just for the BNU1 dataset. Remember from specs that the new pipeline is enormously expensive computationally. Previously we could 10x thread on cortex, and still have ~30 gigs leftover for people to use with half the processing time of the present. Now, we are lucky to be able to 3x thread without cortex crashing. With each scan taking ~1 hour, a 108 scan BNU1 dataset took 2 days to run, or roughly most of the weekend. See here
Plea: need somewhere i can get approx 10x better throughput, so I can actually proceed week to week, as right now dev on the pipeline is relatively impossible (minor altercations to a method, for instance, take ~6 days to verify that they work at scale. This is non-usable).
@ebridge2
you will find that test has several attributes that will be of use for group level qc down the line.
Plea: need somewhere i can get approx 10x better throughput, so I can actually proceed week to week, as right now dev on the pipeline is relatively impossible (minor altercations to a method, for instance, take ~6 days to verify that they work at scale. This is non-usable).
[x] thesis in progress @02agarwalt
[x] FNGS_cloud flow chart: Link
@ewalke31