[ENH] `babs-status` and `babs-submit` function

[zhao-cy]

• when submitting the job(s), check argument where_to_run: e.g., ${CBICA_TMPDIR} or /cbica/comp_space/$(basename $HOME)

  • this is also the argument of participant_job.sh

• first check out file of status.csv (see below); no need to check successfully jobs any more

• check if a subject's job has been submitted

• if submitted:

  • check job status via qsub
    • if finished: check the output:
      • successfully done: if there is a branch for this subject
      • failed: there is only a log file without other results
        • rerun if --rerun is requested
    • if stuck: rerun if --rerun is requested

Before writing babs-status and babs-submit, check existing effort in:

• bash version by Matt: shoreline paper repo - this script or see qsub_rerun.sh in slack channel
  • check if the branch is created (i.e. finished successfully); or the job is still running (checking qsub); otherwise, the job hasn't been submitted yet: to submit it!
• pyhon version by Sydney: cubic RBC project
  • most udpated version is on cubic RBC project: ~/Curation/RBC/PennLINC/Generic
  • not up-to-date version is on GitHub RBC, e.g., get_qsub_calls_rerun_multises.py
  • scripts name:
  • get_qsub_calls_rerun_multises.py
  • get_qsub_calls_rerun.py
  • there are a few others as well but basically just a broken up version of what matt's does

[zhao-cy]

have a text file of list of expected branches (based on the input dataset: availability of the MRI data in each session). babs-status and babs-submit should refer to this text file.

[zhao-cy]

e.g., class job_status

bottom line: not to write as python pickle series (with multiple functions?)...

[zhao-cy]

Best to have a text (or csv) file (probably not tracked by datalad) Every time of babs-submit or babs-status, check this file and update it. This file should have columns of e.g., :

• Has submitted? Finished? Successful? Amount of Memory requested?

[zhao-cy]

Might try out: Sge array job: an array of jobs, submit only one array but that contains all 1000 jobs; easier to submit and kill and check; still keep the graininess of a job (i.e., can still check each job’s stats)

[zhao-cy]

testing:

• create jobs that run for a long time
• create jobs that will fail

[zhao-cy]

only thing left: where to run? (cubic tmpdir, comp_space, etc) can add it in the config yaml file (e.g., section compute_job_working_dir)