Use Pydra to manage dataflow

Hi Dan,

In our lab we often have to generate time series and connectomes for huge datasets, which calls for parallelisation of the workflow and keeping track of progress. Currently, niimasker usesmultiprocessing for parallelisation but I think it would be useful to upgrade to using pydra https://github.com/nipype/pydra:

It has built in caching, which is useful to implement hot restarts if a job fails
It is simple to make computation graphs once the code is factored into to composable steps
- Makes it really easy to split the processing for each subject at nodes of the graph
It supports distributed execution, including SLURM

The team that developed it provides an example application pydra-ml https://github.com/nipype/pydra-ml, which is how I would like to structure niimasker:

Create another file called workflow.py that builds a pydra workflow from niimasker functions and runs it
- This would become the entry point into the program from cli.py rather than make_timeseries
Refactor the code into composable steps
- Split up make_timeseries into steps:
- Create a masker
- Create a denoiser
- Make timeseries
- Generally clean up other sections of code and split functions into smaller units

Using pydra workflows to manage the different steps of processing would also be really useful for adding features in the future. Additional steps are really easy to include in the workflow, by tacking them on to the end as another node (e.g. producing connectomes from timeseries) or splitting nodes (e.g. running the whole process with multiple parcellations).

Would you be interested in including this in the project?

danjgale / nii-masker

Use Pydra to manage dataflow #21