Break down OpenMP jobs across sub-datasets

[jcohenadad]

Because the number of cores on Compute Canada is limited (ie ncore=32 seems to be a good number, in terms of availability and memory), an idea would be to submit multiple jobs of 32 subjects each.

In order to do so, it would be convenient to create pre-made yml config files with the flag include_list including selected 32 subjects.

[jcohenadad]

The idea here would be to have a python script that launches SLURM jobs by chunks of 32 subjects: run_all.py

# Get list of subjects in path data
# Create X sublists of 32 subjects each
# Loop across the sublists
for sublist in sublists:
  # Create temporary job shell script: tmp.job_csa_sublist_i.sh
  # Run it
  os.system('sbatch tmp.job_csa_sublist_i.sh')

and each job file would look like this: tmp.job_csa_sublist_i.sh

#!/bin/sh
#SBATCH --account=def-jcohen
#SBATCH --time=0-08:00        # time (DD-HH:MM)
#SBATCH --nodes=1
#SBATCH --cpus-per-task=32    # number of OpenMP processes
#SBATCH --mem=128G

sct_run_batch -c CONFIG.yml -list_include sub-xxx1 sub-xxx2 ...