search

HenrikBengtsson / future.BatchJobs

:rocket: R package: future.BatchJobs: A Future API for Parallel and Distributed Processing using BatchJobs [Intentionally archived on CRAN on 2021-01-08]
https://cran.r-project.org/package=future.BatchJobs
8 stars 0 forks source link

job.delay supported on torque cluster? #70

Closed sorhawell closed 7 years ago

sorhawell commented 7 years ago

Hi we're a couple of people using BatchJobs on our universitys torque cluster, thx alot for the package. Now the support department would like us to reduce the speed of job submissions to like only one every 2 seconds or so. In submitJobs(reg,ids,job.delay), I can change the job.delay function, and it works well in an interactive session. However when the cluster function is torque, all jobs a submitted at a constant speed of 10 jobs per second, the job.delay function seems to be ignored completely.

How can I make BatchJobs delay the job submission?

printing from R linux session on torque cluster, job.delays is ignored and jobs are submitted really fast

> library(BatchJobs)
Loading required package: BBmisc                                                                                                                       Sour             cing configuration file: '/zhome/c7/0/66069/R/x86_64-pc-linux-gnu-library/3.2/BatchJobs/etc/BatchJobs_global_config.R'
Sourcing configuration file: '/zhome/c7/0/66069/.BatchJobs.R'
BatchJobs configuration:
  cluster functions: Torque
  mail.from:
  mail.to:
  mail.start: none
  mail.done: none
  mail.error: none
  default.resources:
  debug: FALSE
  raise.warnings: FALSE
  staged.queries: TRUE
  max.concurrent.jobs: Inf
  fs.timeout: NA

> reg = makeRegistry("blop")
Creating dir: /zhome/c7/0/66069/tmp/JnWaL2CbhKJ2/blop-files
Saving registry: /zhome/c7/0/66069/tmp/JnWaL2CbhKJ2/blop-files/registry.RData
> ids = batchMap(reg,function(x) x+1, 1:24)
Adding 24 jobs to DB.
> submitJobs(reg,1:24,job.delay = function(n,i) 2)
Saving conf: /zhome/c7/0/66069/tmp/JnWaL2CbhKJ2/blop-files/conf.RData
Submitting 24 chunks / 24 jobs.
Cluster functions: Torque.
Auto-mailer settings: start=none, done=none, error=none.
Writing 24 R scripts...
SubmitJobs |+++++++++++++++++++++++++++++++++++++++++++++++++| 100% (00:00:00)
Sending 24 submit messages...
Might take some time, do not interrupt this!

printing from a interactive session (windows PC), there is a delay of of 2 seconds before each submission(and execution)

> install.packages("BatchJobs")
also installing the dependency ‘RSQLite’

trying URL 'https://cran.rstudio.com/bin/windows/contrib/3.3/RSQLite_1.1.zip'
Content type 'application/zip' length 1966046 bytes (1.9 MB)
downloaded 1.9 MB

trying URL 'https://cran.rstudio.com/bin/windows/contrib/3.3/BatchJobs_1.6.zip'
Content type 'application/zip' length 416942 bytes (407 KB)
downloaded 407 KB

package ‘RSQLite’ successfully unpacked and MD5 sums checked
package ‘BatchJobs’ successfully unpacked and MD5 sums checked

The downloaded binary packages are in
    C:\Users\sowe\AppData\Local\Temp\RtmpsPkzxB\downloaded_packages
> library(BatchJobs)
Loading required package: BBmisc
Sourcing configuration file: 'C:/R/R-3.3.1/library/BatchJobs/etc/BatchJobs_global_config.R'
BatchJobs configuration:
  cluster functions: Interactive
  mail.from: 
  mail.to: 
  mail.start: none
  mail.done: none
  mail.error: none
  default.resources: 
  debug: FALSE
  raise.warnings: FALSE
  staged.queries: TRUE
  max.concurrent.jobs: Inf
  fs.timeout: NA

Warning message:
package ‘BatchJobs’ was built under R version 3.3.2 
> reg = makeRegistry("blop")
Loading registry: C:/Users/sowe/Documents/GitHub/fastRditijuu/blop-files/registry.RData
> ids = batchMap(reg,function(x) x+1, 1:24)
Error in batchMap(reg, function(x) x + 1, 1:24) : Registry is not empty!
> library(BatchJobs)
> reg = makeRegistry("blop")
Creating dir: C:/Users/sowe/Documents/GitHub/fastRditijuu/blop-files
Saving registry: C:/Users/sowe/Documents/GitHub/fastRditijuu/blop-files/registry.RData
> ids = batchMap(reg,function(x) x+1, 1:24)
Adding 24 jobs to DB.
Warning messages:
1: RSQLite::dbGetPreparedQuery() is deprecated, please switch to DBI::dbGetQuery(params = bind.data). 
2: Named parameters not used in query: fun_id, pars, jobname 
3: Named parameters not used in query: job_def_id, seed 
> submitJobs(reg,1:24,job.delay = function(n,i) 3)
Saving conf: C:/Users/sowe/Documents/GitHub/fastRditijuu/blop-files/conf.RData
Submitting 24 chunks / 24 jobs.
Cluster functions: Interactive.
Auto-mailer settings: start=none, done=none, error=none.
Writing 24 R scripts...
SubmitJobs |+                                                                     |   0% (00:00:00)SubmitJobs |+                                                                     |   0% (00:00:00)SubmitJobs |+++                                                                   |   4% (00:01:32)SubmitJobs |++++++                                                                |   8% (00:01:28)SubmitJobs |+++++++++                                                             |  12% (00:01:24)SubmitJobs |++++++++++++                                                          |  17% (00:01:20)SubmitJobs |+++++++++++++++                                                       |  21% (00:01:16)SubmitJobs |++++++++++++++++++                                                    |  25% (00:01:12)SubmitJobs |++++++++++++++++++++                                                  |  29% (00:01:08)SubmitJobs |+++++++++++++++++++++++                                               |  33% (00:01:04)SubmitJobs |++++++++++++++++++++++++++                                            |  38% (00:01:00)SubmitJobs |+++++++++++++++++++++++++++++                                         |  42% (00:00:56)SubmitJobs |++++++++++++++++++++++++++++++++                                      |  46% (00:00:52)SubmitJobs |+++++++++++++++++++++++++++++++++++                                   |  50% (00:00:48)SubmitJobs |++++++++++++++++++++++++++++++++++++++                                |  54% (00:00:43)SubmitJobs |+++++++++++++++++++++++++++++++++++++++++                             |  58% (00:00:39)SubmitJobs |++++++++++++++++++++++++++++++++++++++++++++                          |  62% (00:00:35)SubmitJobs |+++++++++++++++++++++++++++++++++++++++++++++++                       |  67% (00:00:31)SubmitJobs |++++++++++++++++++++++++++++++++++++++++++++++++++                    |  71% (00:00:27)SubmitJobs |++++++++++++++++++++++++++++++++++++++++++++++++++++                  |  75% (00:00:24)SubmitJobs |+++++++++++++++++++++++++++++++++++++++++++++++++++++++               |  79% (00:00:20)SubmitJobs |++++++++++++++++++++++++++++++++++++++++++++++++++++++++++            |  83% (00:00:16)SubmitJobs |+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++         |  88% (00:00:12)SubmitJobs |++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++      |  92% (00:00:07)SubmitJobs |+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++   |  96% (00:00:03)SubmitJobs |++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++| 100% (00:00:00)
Sending 24 submit messages...
Might take some time, do not interrupt this!
HenrikBengtsson commented 7 years ago

Was this question meant to be addressed to the maintainers of the BatchJobs package? Please note that this is the future.BatchJobs package (that builds on top of the former).

sorhawell commented 7 years ago

oh sorry, I'll repost here instead: https://github.com/tudo-r/BatchJobs/issues