Closed liutiming closed 3 years ago
Can you reproduce the error with just clustermq
?
options(clustermq.scheduler = "lsf", clustermq.template = "your_template.tmpl")
library(clustermq)
f <- function(x) x * 2
Q(f, x = seq_len(2), n_jobs = 1)
thanks a lot @wlandau! i think the issue might be clustermq config then?
Submitting 1 worker jobs (ID: cmq8734) ...
Returning output by mail is not supported on this cluster.
Please use the -o option to write output to disk.
Request aborted by esub. Job not submitted.
[1] "BSUB-J cmq8734[1-1] # name of the job / array jobs\nBSUB-n 1 # number of cores to use per job\nBSUB-o ~/clustermq_log/1.log # stdout + stderr; %I for array index\nBSUB-M 4096 # Memory requirements in Mbytes\nBSUB-R \"span[hosts=1] select[mem>4096] rusage[mem=4096]\" # Memory requirements in Mbytes\nBSUB-q normal # name of the queue (uncomment)\n##BSUB-W 6:00 # walltime (uncomment)\nBSUB-G team281\n\nulimit -v $(( 1024 * 4096 ))\nCMQ_AUTH=nswxv R --no-save --no-restore -e 'clustermq:::worker(\"tcp://farm5-head2:8734\")'\n"
Error in (function (n_jobs, ..., log_worker = FALSE, verbose = TRUE) :
Job submission failed with error code 255
I did specify BSUB-o {{ log_file | ~/clustermq_log/ }}
in the tmpl file, as explained above so I am not too sure why
These are the only four lines that produce the error above:
options(clustermq.scheduler = "lsf", clustermq.template = "sge.tmpl")
library(clustermq)
f <- function(x) x * 2
Q(f, x = seq_len(2), n_jobs = 1)
I think your
BSUB-o {{ log_file | ~/clustermq_log/ }}
Is not valid because you're (1) referencing a directory instead of a file, and (2) I'm not sure if LSF expands ~
correctly
And aren't there #
missing at the beginning of the line?
Thank you @mschubert ! by adding back the #
and changing the log path to an absolute path, the job was sent beautifully.
#BSUB-J {{ job_name }}[1-{{ n_jobs }}] # name of the job / array jobs
#BSUB-n {{ cores | 1 }} # number of cores to use per job
#BSUB-o {{ log_file | /nfs/users/nfs_t/tl11/analysis/w3/report/target_multicore/targets-minimal_null}} # stdout + stderr; %I for array index
#BSUB-M {{ memory | 4096 }} # Memory requirements in Mbytes
#BSUB-R "span[hosts=1] select[mem>4096] rusage[mem=4096]" # Memory requirements in Mbytes
#BSUB-q normal # name of the queue (uncomment)
##BSUB-W {{ walltime | 6:00 }} # walltime (uncomment)
#BSUB-G team281
ulimit -v $(( 1024 * {{ memory | 4096 }} ))
CMQ_AUTH={{ auth }} R --no-save --no-restore -e 'clustermq:::worker("{{ master }}")'
Could you give a bit more guidance on how to modify the template? I do not quite understand how the syntax with curly brackets works so if there are documentations on that (could not find in the pkgdown page), it will be great!
@liutiming The wildcard words inside the double curly-brackets will get "filled in" when clustermq
submits the jobs using this template. The values coming after the vertical pipe (|
) are fall-back default values.
Let's take #BSUB-M {{ memory | 4096 }}
as an example, if you do not pass a memory
value when calling clustermq
(either directly or indirectly via targets
), then it will use 4096MB as the memory for your jobs.
Here is how you would control memory via targets
:
tar_option_set(
resources = list(memory = 1000) # this will replace {{ memory }}
)
tar_make_clustermq()
With the issue you were having: BSUB-o {{ log_file | ~/clustermq_log/ }}
, you can either specify the path to a file for logging using log_file
, otherwise it'll will use the default. However, your default pointed to a directory, not a file.
Lastly, the convention for specifying LSF directives using #BSUB-o
rather than just BSUB-o
is a matter of how the submission script is parsed to find the job configuration options.
Hope this is helpful.
@matthewstrasiotto immensely helpful. thank you!
I am not sure where to put this error message @wlandau @mschubert but I put it here first since I am running this in Sanger Institute's LSF Farm and perhaps @mschubert will be more familiar with LSF there?
Steps to reproduce:
options(clustermq.scheduler = "lsf", clustermq.template = "sge.tmpl")
after line 12tar_make_clustermq()