mschubert / clustermq

R package to send function calls as jobs on LSF, SGE, Slurm, PBS/Torque, or each via SSH
https://mschubert.github.io/clustermq/
Apache License 2.0
145 stars 26 forks source link

Nonstandard R path via SSH #281

Closed mschubert closed 2 years ago

mschubert commented 2 years ago

Not sure where to place this...but I had the issue of having to log into my cluster via ssh where R is only available upon loading a specific module to make the R version available. It took me almost a day to figure out what to do:

The key step was to come up with this modified SSH template:

ssh -o "ExitOnForwardFailure yes" -f \
    -R {{ ctl_port }}:localhost:{{ local_port }} \
    -R {{ job_port }}:localhost:{{ fwd_port }} \
    {{ ssh_host }} \
    "source /etc/profile; module load R-bundle; (/absolute/path/to/bin/R --no-save --no-restore -e \
        'clustermq:::ssh_proxy(ctl={{ ctl_port }}, job={{ job_port }})' \
        > {{ ssh_log | /dev/null }} 2>&1 )"

Note the sourcing of /etc/profile and use of the absolute path to R. Maybe this is worth including in the doc? Sorry if this is off topic here, but this issue got me on track (or at least I came across it).

Originally posted by @wds15 in https://github.com/mschubert/clustermq/issues/193#issuecomment-1006023449

mschubert commented 2 years ago

Thanks @wds15, I'll track this here.

wds15 commented 2 years ago

It's more about having to use "modules" to get the R version one needs to use on the target machine rather than non-standard path's, I'd say. At least for me it was some trouble to figure out all the details.

Aariq commented 2 years ago

This also took me a while to figure out when I needed to (#275)