Job.write sbatch script error when using modules

[surh]

I have a script that creates a job with

midas_job = fyrd.Job(midas_command,runpath = os.getcwd(),outpath = args.logdir,
                      scriptpath = args.submissions_dir, clean_files = False,
                      clean_outputs = False, mem = args.memory, name = job_name,
                      outfile = "midas.snps." + sample + ".log",
                      errfile = "midas.snps." + sample + ".err",
                      partition = args.queue,
                      nodes = 1, cores = 8, time = args.time,
                      modules = "MIDAS/1.3.0")

and then writes the submission scripts with:

midas_job.write(overwrite = True)

However inspecting the sbatch scripts that are generated shows that the module load and the sbatch time option are in the same line and without any space in between

#!/bin/bash
#SBATCH --ntasks 1
#SBATCH --cpus-per-task 8
#SBATCH --workdir=/scratch/users/surh/micropopgen/exp/2017/today4
#SBATCH --mem=10000
#SBATCH -o /scratch/users/surh/micropopgen/exp/2017/today4/logs/midas.snps.SRS051941.log
#SBATCH -e /scratch/users/surh/micropopgen/exp/2017/today4/logs/midas.snps.SRS051941.err
#SBATCH -p owners
#SBATCH --time=10:00:00module load MIDAS/1.3.0

The last line causes an error if I try to submit with sbatch. The following minimal example recreates the error.

#!/usr/bin/env python
import fyrd

command = 'blastp -h'

job = fyrd.Job(command, modules = "ncbi-blast+",
               partition = 'owners', time = "00:30:00",
               mem = "100MB")
job.submit()

And here is the result of running that script

$ module_fail.py 
20170817 15:51:28.145 | WARNING --> Command sbatch /scratch/users/surh/micropopgen/exp/2017/today4/small/blastp.0.28a13d85.cluster.sbatch failed with code 1, retrying.
20170817 15:51:29.163 | WARNING --> Command sbatch /scratch/users/surh/micropopgen/exp/2017/today4/small/blastp.0.28a13d85.cluster.sbatch failed with code 1, retrying.
20170817 15:51:30.187 | WARNING --> Command sbatch /scratch/users/surh/micropopgen/exp/2017/today4/small/blastp.0.28a13d85.cluster.sbatch failed with code 1, retrying.
20170817 15:51:31.204 | WARNING --> Command sbatch /scratch/users/surh/micropopgen/exp/2017/today4/small/blastp.0.28a13d85.cluster.sbatch failed with code 1, retrying.
20170817 15:51:32.247 | CRITICAL --> sbatch failed with code 1
-----------------------------------> stdout: 
-----------------------------------> stderr: sbatch: error: Invalid argument: load

Traceback (most recent call last):
  File "/home/surh/src/micropopgen/module_fail.py", line 9, in <module>
    job.submit()
  File "/share/PI/hbfraser/conda_envs/sur-python/lib/python3.6/site-packages/fyrd/job.py", line 566, in submit
    raise _CalledProcessError(code, args, stdout, stderr)
subprocess.CalledProcessError: Command '['sbatch', '/scratch/users/surh/micropopgen/exp/2017/today4/small/blastp.0.28a13d85.cluster.sbatch']' returned non-zero exit status 1.

Maybe the easiest thing is to remove the module load from the ".sbatch" script since it is already called correctly in the ".script" script

#!/bin/bash
module load MIDAS/1.3.0

I am using the following commit on the dev branch:

commit 674c8729523088af7e564d42b5f483e92fa47de5

[MikeDacre]

Note, there are currently a few other bugs in dev, fixing now.

[MikeDacre]

Should be fixed in dev now, let me know if you still have issues @surh