search

EricR86 / segway-issues-proxy

0 stars 0 forks source link

Add Slurm cluster system #27

Open EricR86 opened 10 years ago

EricR86 commented 10 years ago

From cconn...@fhcrc.org on May 01, 2013 12:51:33

Hi, Would it be possible to add the ability to use SLURM with Segway? SLURM has a DRMAA implementation. I am willing to help.

Sincerely,

Chuck Connolly

Original issue: http://code.google.com/p/segway-genome/issues/detail?id=27

EricR86 commented 10 years ago

From hoffman...@gmail.com on May 05, 2013 19:57:43

Sure! There are basically a couple of things we need to do.

First, what is the value of Session.drmsInfo on SLURM? (On LSF it begins with "Platform LSF", on Grid Engine it is "GE" or "SGE" or "UGE".)

We can switch on that value in segway/cluster/init.py to use a different driver file (segway/cluster/slurm.py) to manage the cluster (current ones in segway are segway/cluster/sge.py, segway/cluster/lsf.py, and segway/cluster/pbs.py). At a minimum, this should contain a JobTemplateFactory class, which is a subclass of segway.cluster.common._JobTemplateFactory. This should have make_res_req() and make_native_spec() methods.

segway.slurm.JobTemplateFactory.make_res_req(self, mem_usage, tmp_usage) should take two arguments, specifying the amount of memory and temp space required for a task respectively in bytes. It should set self.res_req with some value that will later be used by segway.slurm.JobTemplateFactory.make_native_spec().

segway.slurm.JobTemplateFactory.make_native_spec(self) should take no non-self arguments and use the self.res_req to return a string that will go into the DRMAA job template's nativeSpecification field.

Do you think you would be able to make a patch for this? I am a little short on cycles right now to actually do this myself (mainly because I don't have access to a SLURM system at the moment to test on).

Summary: Add Slurm cluster system (was: add ability to use slurm)
Status: Accepted

EricR86 commented 10 years ago

From cconn...@fhcrc.org on May 07, 2013 10:10:27

Hi, The Session.drmsInfo value I get is 'SLURM 2.5.4'. I can manage the patch.

Chuck

EricR86 commented 10 years ago

From hoffman...@gmail.com on May 07, 2013 11:42:16

The next release will have a hook for drms_info.startswith("SLURM") in segway.cluster.get_driver_name()

elif drms_info.startswith("SLURM"):
    return "slurm"

So this should work well with a new slurm.py. The attached pbs.py from the next release is the simplest driver and might make a good starting point.

Attachment: pbs.py

EricR86 commented 10 years ago

From cconn...@fhcrc.org on May 24, 2013 13:06:44

I've attached a slurm.py driver file. This file works to process the test.genomedata on a system running slurm 2.5.4. Please let me know if you need modify it.

Attachment: slurm.py

EricR86 commented 10 years ago

From hoffman...@gmail.com on May 24, 2013 13:11:30

Thanks so much! I'm glad you were able to make this work. I will add integrating this patch to my to-do list.