mschubert / clustermq

R package to send function calls as jobs on LSF, SGE, Slurm, PBS/Torque, or each via SSH
https://mschubert.github.io/clustermq/
Apache License 2.0
146 stars 27 forks source link

Add an option to get an interface's IP address #172

Closed valtandor closed 4 years ago

valtandor commented 5 years ago

@mschubert I don't know about portability, but this should work on most Macs and Linux machines, based on availability of ifconfig.

valtandor commented 5 years ago

@mschubert I'm also aware that there isn't much in the way of good error handling. But I haven't had any more than a cursory look at the code yet, so I don't know how you typically handle errors, e.g. (in this case) the interface name not existing on the host.

kendonB commented 5 years ago

I am testing this out now and will report back if it worked either later tonight or in the morning.

kendonB commented 5 years ago

Yip it works!

library(clustermq)
options(clustermq.scheduler = "slurm", 
        clustermq.template = "slurm_clustermq.tmpl",
        clustermq.network.interface = "ib0")

clustermq::Q(function(x) Sys.sleep(x), x=rep(90*60,2), n_jobs=2,
             template = list(
               log_file = "make.log", 
               memory = 500,
               walltime = 250
             ))
#> Submitting 2 worker jobs (ID: 7340) ...
#> Running 2 calculations (0 objs/0 Mb common; 1 calls/chunk) ...
#> Master: [5466.6s 0.0% CPU]; Worker: [avg 0.0% CPU, max 247.7 Mb]
#> [[1]]
#> NULL
#> 
#> [[2]]
#> NULL
kendonB commented 5 years ago

@valtandor do you know why Travis is failing?

valtandor commented 5 years ago

@kendonB No, all I see in the logs to indicate the problem is this:

No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.
Check the details on how to adjust your build configuration on: https://docs.travis-ci.com/user/common-build-problems/#build-times-out-because-no-output-was-received

I was hoping @mschubert might be able to comment.

mschubert commented 4 years ago

Thank you for your effort @valtandor, and my apologies for the (very) late reply.

I'm hesitant to merge this, because it relies on the availability on the external ifconfig utility.

This may not be available on all systems, as is the case on e.g. my own (Archlinux). It will also fail on Windows (which is now not working due to mcparallel, but I want to fix that).

I'll see if I find a better option, and if not I will come back to this.

mschubert commented 4 years ago

I will not depend on ifconfig, for the reasons outlined in #170.

Nevertheless, I would thank you very much @valtandor for looking into this and providing a possible solution (+ @kendonB for reporting).

kendonB commented 4 years ago

@mschubert is there a way to include this as opt in such that it doesn't enforce the dependence on ifconfig but makes it optional?

mschubert commented 4 years ago

You will be able to specify the interface, just the option will work slightly differently and not depend on ifconfig.

So I see no reason to?

In any case, I will ask you to test this out before I make a new release to make sure it works for your use case.