Parallelize motion correction

[pcklink]

Currently motion correction is parallelized within a run, while all runs in a job are done in serial fashion. We could potentially speed things up tremendously if runs were parallelized. This is probably not possible within a single job due to memory/cpu restrictions but it will be possible if we create one job per run. [EDIT] Indeed 3dAllineate already takes a lot of CPU. We prob. should not do this within-job.

Note that we can in principle already do this by defining single-run csv files and starting jobs for each of them. However, it would be nice if it were possible with the same multi-run csv.

To establish this and not mess up the workflow too much we would need:

OPTION 1

• a python script that creates new 'split' csv files and generates jobs for them

OPTION 2

• an argument on the bids_preprocessing_workflow.py that overrules the run field from the csv

[pcklink]

Done! Went with Option 1. There's a Python script in NHP-BIDS/code/subcode called bids_preproc_parallel_runs.py that does this.

The old serial way of doing it also still works, but I'd recommend doing it the new way. The script basically does the following:

• Take the original csv-file (no need to create a new one) and parse it.
• For each unique run in the csv-file, it creates a new csv-file that specifies only that run.
• A companion job file is created for each single-run csv-file and made executable (note that we reserve 10h for each job, which should be more than enough).
• This job file specifies the bids_preprocessing_workflow.py and potentially also the bids_warp2nmt_workflow.py for this single run (warping can be excluded, but I suggest you run it immediately).
• Schedule all jobs into SLURM

Few hours later >> RESULTS!