No way to pass resources parameter to SLURM?

[alexvorobiev]

I am trying to migrate my BatchJobs-based code to batchjobs_slurm. The code passes a bunch of paramters to the SLURM template using resources argument:

submitJobs(registry, 
           resources = list('ntasks'   = 10,
                            'ncpus'    = 4,
                            'walltime' = 5))

I could not find anywhere in the documentation how to pass this argument to plan(batchjobs_slurm, pathname = "my_slurm_template").

[HenrikBengtsson]

This will be available in next release. The reason why I've held back (*) on this is because I wanted to identify a stable and generic way of doing this. For instance, should these be specific to each scheduler (as in your example) or could we possibly identify a set of common resource parameters that could be shared across types of schedulers and thereby also share common names? However, before solving the latter I think future.BatchJobs should provide a way to pass whatever down, so ...

In the develop branch, which you can install as:

source('http://callr.org/install#HenrikBengtsson/future.BatchJobs@develop')

you can now pass anything down to the BatchJobs template via argument args (a named list or an environment), e.g.

resources <- list(ntasks = 10, ncpus = 4, walltime = 5)
plan(batchjobs_slurm, pathname = "./.BatchJobs.slurm.brew", args = list(resources=resources))

FYI, I'm using this approach myself in our large-scale analysis on a TORQUE scheduler.

Let me know if this helps.

PS. (*) It was actually available in the past, then I hid it because I anticipated a change in API, which happened.

[alexvorobiev]

Yes, that helps, thanks. When are you planning to do the next release?

[HenrikBengtsson]

I could probably submit it to CRAN quite soon if needed, but would like for it to get some mileage first. I also do not wanna submit to CRAN to frequently to lower their burden, which is quite high already.

So, give it some real testing and let me know if you think it works as expected etc etc. Then I can start planning a new submission. Since there aren't many reverse dependencies, testing a new CRAN submission should be fairly quick.

(Closing this issue, but feel free to report back here later if you run into issues or just wanna say it works perfectly.)

[HenrikBengtsson]

Just some more thoughts / comments:

I'm trying to decide whether to use (generic) args or (specific) resources. In the past I used resources and then I recently generalized it to args. However, I just (re-)realized why I used resources in the past. It's because BatchJobs::submitJobs() used internally only supports passing variable name resources to the BatchJobs template.

It's still args, but I just pushed (to develop) a version that early on asserts that args only contains the element resources (because that's what BatchJobs only supports). But of course, it's rather silly to have to write:

plan(batchjobs_slurm, args = list(resources = list(...))`

each time. This would argue for going back to:

plan(batchjobs_slurm, resources = list(...))`

The latter is also more familiar to existing BatchJobs users.

So, I'm strongly leaning toward going back to an argument named `resources'.

(I'm reopening this in case someone else sees it and wish to pitch in to the discussion).

[alexvorobiev]

I have very limited experience with using BatchJobs with Slurm. If the resources name is hard-coded anywhere in BatchJobs (aside from the sample templates which are easy for the users to edit), then I agree it should be reflected somewhere in future.BatchJobs. If BatchJobs allows to pass something other than the resources to the Slurm scripts then future.BatchJobs ought to allow that as well and the args should stay. Otherwise there seems to be no need to keep it.

[HenrikBengtsson]

Next release will use argument resources (deprecating args).

[HenrikBengtsson]

FYI @alexvorobiev, future.BatchJobs 0.13.0 supporting resources argument just hit CRAN.

[alexvorobiev]

Thanks! Does args still work? Or do I have to change my code prior to updating from CRAN?

[HenrikBengtsson]

You need to update scripts first, otherwise tweak(batchjobs_slurm, args=...) will give an error.

On Aug 3, 2016 23:41, "Alex Vorobiev" notifications@github.com wrote:

Thanks! Does args still work? Or do I have to change my code prior to updating from CRAN?

— You are receiving this because you modified the open/close state.

Reply to this email directly, view it on GitHub https://github.com/HenrikBengtsson/future.BatchJobs/issues/64#issuecomment-237383348, or mute the thread https://github.com/notifications/unsubscribe-auth/ABir0tGbidqh9wYQxghSoVLKzoAkdIVYks5qcQrZgaJpZM4JLzXL .