rimorob
commented
3 years ago Figured this out: SLURM doesn't accept memory option on ParallelCluster. This is shocking, but there is a workaround, which is quite messy, since it requires manual configuration - a bad thing on parallelcluster. I suppose post-install script is an option. More here: https://github.com/aws/aws-parallelcluster/issues/1517
Are slurm stubs supposed to be logged as doFuture launches jobs? I can't find any in my .future folder. I have a reason to believe that my resources setttings (specifically, memory usage and wall time) are being ignored when launching jobs. I did notice that Henrik refers to the memory argument as "mem" here: https://github.com/HenrikBengtsson/future/issues/95 but future.batchtools calls it "vmem" here: https://cran.r-project.org/web/packages/future.batchtools/vignettes/future.batchtools.html.
Further, there's a difference in usage - some of the manuals recommend using strings ("5gb") and others - using numeric designation - "5000". I can't find consistent and, especially, detailed documentation on any of the above.
For reference, I try to configure doFuture like this:
future::plan(batchtools_slurm, template=tmplPath, resources = list(nodes = self$nCores, vmem = self$memReq, walltime = self$wallTime, ncpus = 1, # # of cpus per single job
conda.env = self$condaEnv, conda.path=self$condaPath))