Launcher support? - Githubissues

effigies commented 3 years ago

TACC has launcher, does not support job arrays https://github.com/TACC/launcher

See https://frontera-portal.tacc.utexas.edu/user-guide/running/

effigies commented 3 years ago

https://www.tacc.utexas.edu/research-development/tacc-software/the-launcher

yarikoptic commented 3 years ago

Launcher seems to avoid #559 by requesting a single job with multiple nodes/cpus, and then doing it own "scheduling" and distribution of runs across those nodes available. Obviously suboptimal from the point of view of HPC configuration, BUT allows right away to submit those 1000 subjects to be processed.

yarikoptic commented 3 years ago

so a prototypical invocation would likely to be

$ reproman run --follow -r local --sub slurm-launcher --orc datalad-no-remote --jp nodes=10 --jp jobs=10 -bp sub=$(all_bids_subjects) --bp space=MNI,native CMD '{p[sub]} {p[preproc]}'

where we would

[ ] implement slurm-launcher interface to use launcher to avoid messing with #559 since it does internally
[ ] (could be done later)do submit multiple jobs (so somewhat related to #558) so to not fall into 100 "subjobs" waiting on the longest one to complete. Hence I thought it would be that additional --jp jobs=10 .

jdkent commented 3 years ago

was this fixed by #577?

yarikoptic commented 3 years ago

rright! thank you @jdkent, lets close

ReproNim / reproman

Launcher support? #557