Closed ja-thomas closed 6 years ago
Should work now. Re-open if you still have problems.
Hm, I think we get a bit further, but still have the same error.
EDIT: No, I think it's still the same problem, i.e. w$script is empty.
Here's the traceback
Error in submitJobs(1) :
Assertion on 'sys.cmd' failed: Must have length 1, but has length 0.
> traceback()
18: stop(simpleError(sprintf(msg, ...), sys.call(1L)))
17: mstop("Assertion on '%s' failed: %s.", var.name, res)
16: makeAssertion(x, res, .var.name, add)
15: assertCharacter(sys.cmd, any.missing = FALSE, len = 1L)
14: runOSCommand(self$script, args, nodename = self$nodename)
13: private$filter_output(runOSCommand(self$script, args, nodename = self$nodename))
12: private$run(c("list-jobs", reg$file.dir))
11: stri_trim_both(private$run(c("list-jobs", reg$file.dir))$output)
10: stri_join(self$nodename, "#", stri_trim_both(private$run(c("list-jobs",
reg$file.dir))$output))
9: w$list(reg)
8: FUN(X[[i]], ...)
7: lapply(workers, function(w) w$list(reg))
6: unlist(lapply(workers, function(w) w$list(reg)), use.names = FALSE)
5: cf$listJobsRunning(reg)
4: unique(cf$listJobsRunning(reg))
3: getBatchIds(reg, status = status)
2: .findOnSystem(reg = reg, cols = c("job.id", "batch.id"))
1: submitJobs(1)
More quoting issues... :cry: -> 02f83b167fcb6aa38a89b7149d1c4258e5e1638b
I hope https://github.com/ropensci/ssh will be released soon ...
Can you try again?
Hm, now I can't spawn the worker anymore
> Worker$new("10.155.47.223", 1)
Error in stop(simpleError(sprintf(...), call = sys.call(sys.parent()))) :
bad error message
traceback
5: stop(simpleError(sprintf(...), call = sys.call(sys.parent())))
4: stopf("runOSCommand failed: Expected BOF+EOF markers for '%s %s', but got:\n %s",
res$sys.cmd, stri_flatten(res$sys.args, " "), stri_flatten(res$output,
"\n"))
3: private$filter_output(res)
2: .subset2(public_bind_env, "initialize")(...)
1: Worker$new("10.155.47.223", 1)
Worker$new("localhost", 1)
works, though.
Have you re-installed batchtools on all nodes?
So after reinstalling on all node we can submit jobs, but they are expiring. Log file does not exist.
Here is the debugme output:
Sys.setenv(DEBUGME = "batchtools")
library(batchtools)
unlink("asdf", recursive = TRUE)
reg = makeRegistry("asdf", conf.file = ".batchtools.conf.R")
batchMap(function(x) {
return(x)
}, x = 1:10)
submitJobs(1)
> reg = makeRegistry("asdf", conf.file = ".batchtools.conf.R")
Sourcing configuration file '.batchtools.conf.R' ...
batchtools [runOSCommand]: cmd:
batchtools [runOSCommand]: cmd: +33ms
batchtools [runOSCommand]: cmd: +2ms
batchtools [runOSCommand]: cmd: +1ms
batchtools [runOSCommand]: cmd: +1ms
batchtools [runOSCommand]: cmd: +1ms
batchtools [runOSCommand]: cmd: +1ms
batchtools [runOSCommand]: cmd: +1ms
batchtools [runOSCommand]: cmd: ssh -q ubuntu@10.155.47.232 "Rscript -e 'message(\"[bt] --BOF--\\n\", \"[b
t] \", system.file(\"bin/linux-helper\", package = \"batchtools\"), \"\\n[bt] --EOF--\\n\")'" +1ms
batchtools +-[runOSCommand]: OS result (stdin ' +623ms
batchtools +-[runOSCommand]: OS result (stdin ' +7ms
batchtools +-[runOSCommand]: OS result (stdin ' +3ms
batchtools +-[runOSCommand]: OS result (stdin ' +3ms
batchtools +-[runOSCommand]: OS result (stdin ' +2ms
batchtools +-[runOSCommand]: OS result (stdin ' +2ms
batchtools +-[runOSCommand]: OS result (stdin ' +2ms
batchtools +-[runOSCommand]: OS result (stdin ' +2ms
batchtools [runOSCommand]: OS result (stdin '', exit code 0): +2ms
batchtools +-[runOSCommand]: +2ms
batchtools +-[runOSCommand]: +2ms
batchtools +-[runOSCommand]: +2ms
batchtools +-[runOSCommand]: +2ms
batchtools +-[runOSCommand]: +2ms
batchtools +-[runOSCommand]: +2ms
batchtools +-[runOSCommand]: +2ms
batchtools +-[runOSCommand]: +2ms
batchtools [runOSCommand]: During startup - Warning messages:
+2ms batchtools [runOSCommand]: 1: Setting LC_TIME failed, using "C"
+2ms batchtools [runOSCommand]: 2: Setting LC_MONETARY failed, using "C"
+2ms batchtools [runOSCommand]: 3: Setting LC_PAPER failed, using "C"
+2ms batchtools [runOSCommand]: 4: Setting LC_MEASUREMENT failed, using "C"
+2ms batchtools [runOSCommand]: [bt] --BOF--
+2ms batchtools [runOSCommand]: [bt] /usr/local/lib/R/site-library/batchtools/bin/linux-helper
+2ms batchtools [runOSCommand]: [bt] --EOF--
+2ms batchtools [runOSCommand]:
+2ms
batchtools [makeRegistry]: Creating directories in ' +27ms
batchtools [makeRegistry]: Creating directories in ' +2ms
batchtools [makeRegistry]: Creating directories in ' +2ms
batchtools [makeRegistry]: Creating directories in ' +2ms
batchtools [makeRegistry]: Creating directories in ' +2ms
batchtools [makeRegistry]: Creating directories in ' +2ms
batchtools [makeRegistry]: Creating directories in ' +2ms
batchtools [makeRegistry]: Creating directories in ' +2ms
batchtools [makeRegistry]: Creating directories in 'asdf' +2ms
batchtools +-[loadRegistryDependencies]: Starting ... +8ms
batchtools +-[loadRegistryDependencies]: Starting ... +2ms
batchtools +-[loadRegistryDependencies]: Starting ... +2ms
batchtools +-[loadRegistryDependencies]: Starting ... +2ms
batchtools +-[loadRegistryDependencies]: Starting ... +2ms
batchtools +-[loadRegistryDependencies]: Starting ... +2ms
batchtools +-[loadRegistryDependencies]: Starting ... +2ms
batchtools +-[loadRegistryDependencies]: Starting ... +2ms
batchtools +-[loadRegistryDependencies]: Starting ... +2ms
batchtools +-[saveRegistry]: Saving Registry +3ms
batchtools +-[saveRegistry]: Saving Registry +2ms
batchtools +-[saveRegistry]: Saving Registry +2ms
batchtools +-[saveRegistry]: Saving Registry +2ms
batchtools +-[saveRegistry]: Saving Registry +2ms
batchtools +-[saveRegistry]: Saving Registry +2ms
batchtools +-[saveRegistry]: Saving Registry +2ms
batchtools +-[saveRegistry]: Saving Registry +4ms
batchtools +-[saveRegistry]: Saving Registry +2ms
Created registry in '/home/janek/test_batchtools/asdf' using cluster functions 'SSH'
>
>
> batchMap(function(x) {
+ return(x)
+ }, x = 1:10)
Adding 10 jobs ...
batchtools [saveRegistry]: Saving Registry +2381ms
batchtools [saveRegistry]: Saving Registry +2ms
batchtools [saveRegistry]: Saving Registry +2ms
batchtools [saveRegistry]: Saving Registry +2ms
batchtools [saveRegistry]: Saving Registry +2ms
batchtools [saveRegistry]: Saving Registry +2ms
batchtools [saveRegistry]: Saving Registry +2ms
batchtools [saveRegistry]: Saving Registry +2ms
batchtools [saveRegistry]: Saving Registry +2ms
>
> submitJobs(1)
batchtools [syncRegistry]: Triggered syncRegistry +1280ms
batchtools [syncRegistry]: Triggered syncRegistry +2ms
batchtools [syncRegistry]: Triggered syncRegistry +1ms
batchtools [syncRegistry]: Triggered syncRegistry +1ms
batchtools [syncRegistry]: Triggered syncRegistry +1ms
batchtools [syncRegistry]: Triggered syncRegistry +2ms
batchtools [syncRegistry]: Triggered syncRegistry +1ms
batchtools [syncRegistry]: Triggered syncRegistry +1ms
batchtools [syncRegistry]: Triggered syncRegistry +1ms
batchtools [castIds]: Casting ids from vector to data.table +2ms
batchtools [castIds]: Casting ids from vector to data.table +1ms
batchtools [castIds]: Casting ids from vector to data.table +1ms
batchtools [castIds]: Casting ids from vector to data.table +19ms
batchtools [castIds]: Casting ids from vector to data.table +1ms
batchtools [castIds]: Casting ids from vector to data.table +1ms
batchtools [castIds]: Casting ids from vector to data.table +1ms
batchtools [castIds]: Casting ids from vector to data.table +1ms
batchtools [castIds]: Casting ids from vector to data.table +1ms
batchtools [getBatchIds]: Getting running Jobs +7ms
batchtools [getBatchIds]: Getting running Jobs +2ms
batchtools [getBatchIds]: Getting running Jobs +1ms
batchtools [getBatchIds]: Getting running Jobs +1ms
batchtools [getBatchIds]: Getting running Jobs +1ms
batchtools [getBatchIds]: Getting running Jobs +1ms
batchtools [getBatchIds]: Getting running Jobs +2ms
batchtools [getBatchIds]: Getting running Jobs +2ms
batchtools [getBatchIds]: Getting running Jobs +2ms
batchtools +-[runOSCommand]: cmd: +34ms
batchtools +-[runOSCommand]: cmd: +2ms
batchtools +-[runOSCommand]: cmd: +2ms
batchtools +-[runOSCommand]: cmd: +2ms
batchtools +-[runOSCommand]: cmd: +2ms
batchtools +-[runOSCommand]: cmd: +2ms
batchtools +-[runOSCommand]: cmd: +2ms
batchtools +-[runOSCommand]: cmd: +2ms
batchtools +-[runOSCommand]: cmd: ssh -q ubuntu@10.155.47.232 '/usr/local/lib/R/site-library/batchtools/bi
n/linux-helper list-jobs /home/janek/test_batchtools/asdf' +2ms
batchtools +-[runOSCommand]: OS result (stdin ' +373ms
batchtools +-[runOSCommand]: OS result (stdin ' +6ms
batchtools +-[runOSCommand]: OS result (stdin ' +4ms
batchtools +-[runOSCommand]: OS result (stdin ' +3ms
batchtools +-[runOSCommand]: OS result (stdin ' +3ms
batchtools +-[runOSCommand]: OS result (stdin ' +2ms
batchtools +-[runOSCommand]: OS result (stdin ' +2ms
batchtools +-[runOSCommand]: OS result (stdin ' +2ms
batchtools +-[runOSCommand]: OS result (stdin '', exit code 0): +2ms
batchtools +-[runOSCommand]: +2ms
batchtools +-[runOSCommand]: +2ms
batchtools +-[runOSCommand]: +2ms
batchtools +-[runOSCommand]: +2ms
batchtools +-[runOSCommand]: +2ms
batchtools +-[runOSCommand]: +2ms
batchtools +-[runOSCommand]: +2ms
batchtools +-[runOSCommand]: +2ms
batchtools +-[runOSCommand]: [bt] --BOF--
+2ms batchtools +-[runOSCommand]: [bt] --EOF--
+2ms
Submitting 1 jobs in 1 chunks using cluster functions 'SSH' ...
batchtools [Worker]: Updating Worker ' +82ms
batchtools [Worker]: Updating Worker ' +2ms
batchtools [Worker]: Updating Worker ' +2ms
batchtools [Worker]: Updating Worker ' +1ms
batchtools [Worker]: Updating Worker ' +1ms
batchtools [Worker]: Updating Worker ' +1ms
batchtools [Worker]: Updating Worker ' +2ms
batchtools [Worker]: Updating Worker 'ubuntu@10.155.47.232' +1ms
batchtools +-[runOSCommand]: cmd: +2ms
batchtools +-[runOSCommand]: cmd: +2ms
batchtools +-[runOSCommand]: cmd: +2ms
batchtools +-[runOSCommand]: cmd: +2ms
batchtools +-[runOSCommand]: cmd: +2ms
batchtools +-[runOSCommand]: cmd: +2ms
batchtools +-[runOSCommand]: cmd: +2ms
batchtools +-[runOSCommand]: cmd: +1ms
batchtools +-[runOSCommand]: cmd: ssh -q ubuntu@10.155.47.232 '/usr/local/lib/R/site-library/batchtools/bi
n/linux-helper status /home/janek/test_batchtools/asdf' +1ms
batchtools +-[runOSCommand]: OS result (stdin ' +418ms
batchtools +-[runOSCommand]: OS result (stdin ' +5ms
batchtools +-[runOSCommand]: OS result (stdin ' +2ms
batchtools +-[runOSCommand]: OS result (stdin ' +2ms
batchtools +-[runOSCommand]: OS result (stdin ' +3ms
batchtools +-[runOSCommand]: OS result (stdin ' +3ms
batchtools +-[runOSCommand]: OS result (stdin ' +3ms
batchtools +-[runOSCommand]: OS result (stdin ' +2ms
batchtools +-[runOSCommand]: OS result (stdin '', exit code 0): +3ms
batchtools +-[runOSCommand]: +3ms
batchtools +-[runOSCommand]: +2ms
batchtools +-[runOSCommand]: +2ms
batchtools +-[runOSCommand]: +5ms
batchtools +-[runOSCommand]: +3ms
batchtools +-[runOSCommand]: +3ms
batchtools +-[runOSCommand]: +2ms
batchtools +-[runOSCommand]: +3ms
batchtools +-[runOSCommand]: [bt] --BOF--
+2ms batchtools +-[runOSCommand]: [bt] 0.01 0 0 0
+2ms batchtools +-[runOSCommand]: [bt] --EOF--
+2ms
batchtools [runOSCommand]: cmd: +7ms
batchtools [runOSCommand]: cmd: +2ms
batchtools [runOSCommand]: cmd: +2ms
batchtools [runOSCommand]: cmd: +2ms
batchtools [runOSCommand]: cmd: +2ms
batchtools [runOSCommand]: cmd: +2ms
batchtools [runOSCommand]: cmd: +2ms
batchtools [runOSCommand]: cmd: +2ms
batchtools [runOSCommand]: cmd: ssh -q ubuntu@10.155.47.232 '/usr/local/lib/R/site-library/batchtools/bin/
linux-helper start-job /home/janek/test_batchtools/asdf/jobs/job1b087a98bff4077d9033a78a4e84ed51.rds /home
/janek/test_batchtools/asdf/logs/job1b087a98bff4077d9033a78a4e84ed51.log' +3ms
batchtools +-[runOSCommand]: OS result (stdin ' +399ms
batchtools +-[runOSCommand]: OS result (stdin ' +7ms
batchtools +-[runOSCommand]: OS result (stdin ' +3ms
batchtools +-[runOSCommand]: OS result (stdin ' +3ms
batchtools +-[runOSCommand]: OS result (stdin ' +3ms
batchtools +-[runOSCommand]: OS result (stdin ' +3ms
batchtools +-[runOSCommand]: OS result (stdin ' +2ms
batchtools +-[runOSCommand]: OS result (stdin ' +2ms
batchtools [runOSCommand]: OS result (stdin '', exit code 0): +2ms
batchtools +-[runOSCommand]: +2ms
batchtools +-[runOSCommand]: +2ms
batchtools +-[runOSCommand]: +2ms
batchtools +-[runOSCommand]: +2ms
batchtools +-[runOSCommand]: +2ms
batchtools +-[runOSCommand]: +2ms
batchtools +-[runOSCommand]: +2ms
batchtools +-[runOSCommand]: +2ms
batchtools [runOSCommand]: [bt] --BOF--
+2ms batchtools [runOSCommand]: [bt] 7004
+2ms batchtools [runOSCommand]: [bt] --EOF--
+2ms batchtools [runOSCommand]: /usr/local/lib/R/site-library/batchtools/bin/linux-helper: line 53: /home
/janek/test_batchtools/asdf/logs/job1b087a98bff4077d9033a78a4e84ed51.log: No such file or directory
+2ms
batchtools [makeSubmitJobResult]: Result for batch.id ' +3ms
batchtools [makeSubmitJobResult]: Result for batch.id ' +4ms
batchtools [makeSubmitJobResult]: Result for batch.id ' +2ms
batchtools [makeSubmitJobResult]: Result for batch.id ' +2ms
batchtools [makeSubmitJobResult]: Result for batch.id ' +2ms
batchtools [makeSubmitJobResult]: Result for batch.id ' +2ms
batchtools [makeSubmitJobResult]: Result for batch.id ' +2ms
batchtools [makeSubmitJobResult]: Result for batch.id ' +2ms
batchtools [makeSubmitJobResult]: Result for batch.id 'ubuntu@10.155.47.232#7004,': 0 (OK) +2ms
batchtools [saveRegistry]: Saving Registry +4ms
batchtools [saveRegistry]: Saving Registry +2ms
batchtools [saveRegistry]: Saving Registry +2ms
batchtools [saveRegistry]: Saving Registry +2ms
batchtools [saveRegistry]: Saving Registry +2ms
batchtools [saveRegistry]: Saving Registry +2ms
batchtools [saveRegistry]: Saving Registry +2ms
batchtools [saveRegistry]: Saving Registry +2ms
batchtools [saveRegistry]: Saving Registry +2ms
You need the same (relative) file system layout on your machine and the node. You are logging in as user ubuntu
but try to access /home/janek
.
Try creating the registry with a file.dir
provided not relative to your current work dir. ~
will not be resolved to a complete path (or at least it should not be resolved, this is hard to test) and can be used in your case:
reg = makeRegistry("~/asdf", conf.file = "~/.batchtools.conf.R")
Did this work for you?
NB: if you install the devel version of debugme
, the output is not cluttered.
Sorry for the late reply.
It works now, thanks for your help!
Hi,
Maybe I'm just too stupid to get it to work, but when I try to use a makeshift SSH cluster with remote systems I get the following error:
the problem is that the Worker w
w$script
is empty. When I set the worker to localhost there is a path to/batchtools/bin/linux-helper
and it works.Sorry I can't really give a reproducible example since the systems I'm using are only available in our vpn. And I don't have another server to test
The code looks like this: config: