odissei-lifecourse / life-sequencing-dutch

MIT License
0 stars 0 forks source link

Standardize and modularize slurm scripts #39

Open f-hafner opened 5 months ago

f-hafner commented 5 months ago

goal:

Steps:

  1. on OSSC and in this repository, create directory slurm_setup/, and then create files modules.sh and venv.sh
    • modules.sh should call module load xyz in order
    • venv.sh calls source "$PREFIX"/virtual_envs/"$VENV"/bin/activate
  2. on OSSC, move the virtual environments from root into dedicated folder virtual_envs
  3. in this repo, the slurm scripts should follow the same conventions

    • (common location for logs -- in the future) -- see #27
    • common names for log files, ie jobname.jobid.err and *.out as is currently done in some scripts (I'm open to alternatives)
    • declare local variables and load

      declare PREFIX="/gpfs/ostor/ossc9424/homedir/"
      declare VENV="ossc_new" # the name of the virtual environment
      
      source "$PREFIX"/slurm_setup/modules.sh
      source "$PREFIX"/slurm_setup/venv.sh
      
      # run scripts
      

work on this once branch pipeline_parallel is merged.

f-hafner commented 5 months ago

I have tested the logic above on the ossc and it works