Slurm: Nextflow Script and Code Integration with OSMP Stack

[jennyziyi-xu]

I tried to run Hannah's PR, but ran into an error when making a basic gene query. That PR contains a lot of code changes and it's hard to find and fix issues. To break Hannah's PR into smaller steps:

• [x] Send a dummy job from OSMP backend to Slurm. Verify that result can be seen on the working directory in the Slurm node.
• [ ] Send a dummy job from OSMP backend that kickstarts a Nextflow job.
• [ ] Further develop Nextflow script, have a working version of it.
• [ ] When gene size is larger than 0.6 mbps, kickstart the Nextflow job.
• [ ] TBD

[jennyziyi-xu]

Running these commands in the terminal can start the nextflow job successfully:

ssh [user]@slogin.ccm.sickkids.ca
srun --pty bash
module load nextflow/20200707 Singularity/3.7.0_slurm
cd osmp-annotation/jobs  (from my home directory)
nextflow run annotation.nf -profile annotation

On the branch jenny/slurm, the [Slurm API executes the script slurm-script-1.sh] (https://github.com/ccmbioinfo/osmp/blob/bcccb31b2f6b9671f1ae49141a4c9cc23efd9011/server/src/resolvers/getVariantsResolver/getVariantsResolver.ts#L94). slurm-script-1.sh has the following content:

#!/bin/bash
module () {
    eval `/usr/bin/modulecmd bash $*`
}
module load nextflow Singularity
cd osmp-annotation/jobs
nextflow run annotation_test.nf -profile annotation
echo "Done executing"

• Calling this REST API will however result in a permission error related to NextFlow.
• Kevin Lu has suggested sourcing .bashrc before executing the script: link