pygridtools / gridmap

Easily map Python functions onto a cluster using a DRMAA-compatible grid engine like Sun Grid Engine (SGE).
GNU General Public License v3.0
83 stars 34 forks source link

specifying resources? #88

Closed nick-youngblut closed 4 years ago

nick-youngblut commented 4 years ago

The gridmap docs seem to have essentially nothing about specifying job resources (eg., time and memory), although I see there is an unmerged PR for specifying a gpu resource. All I se in the code is:

        if self.mem_free and USE_MEM_FREE:
            ret += " -l mem_free={}".format(self.mem_free)
        if self.num_slots and self.num_slots > 1:
            ret += " -pe {} {}".format(self.par_env, self.num_slots)
        if self.white_list:
            ret += " -l h={}".format('|'.join(self.white_list))
        if self.queue:
            ret += " -q {}".format(self.queue)

How is the user supposed to set -l h_rt or -l h_vmem? We also have special resource settings that our cluster admin setup, so how can the user specify non-standard resources (e.g., -l tmpfs)?

mulhod commented 4 years ago

You make a great point. It is only possible to pass in very few resource requests. The GPU resource-related PR that you mentioned was actually just merged. I might suggest making a PR based on that one that satisfies at least the first two resources you mentioned, h_rt and h_vmem. As for the last, we obviously would need to do some refactoring to allow users to specify other resources that are either not currently supported or that are non-standard, like the one you mentioned. With that in mind, I created #89. I'm not sure when we will get to this, but of course we are open to contributions. We will try to make an effort to expedite it.

nick-youngblut commented 4 years ago

Thanks for merging that GPU PR and the general feedback! I already forked the repo and made some changes. I make a PR after testing the code

mulhod commented 4 years ago

Addressed by #93