Closed mhesselbarth closed 3 years ago
This looks less like a library issue, more like a network (SSH) forwarding issue.
Can you tell me:
clustermq
are you using?Hey,
Interesting that this might be a SSH issue.
> fx(5)
nodename
"gl-login1.arc-ts.umich.edu
clustermq_0.8.95.1
clustermq
before, but on a different HPC. On the HPC I am using currently I never used clustermq
and I am also not aware somebody else did.Does your code work if you run it on your login node instead of via SSH?
fx(5)
I meant with Q(...)
:smile:
That makes a lot more sense, sorry 😅
Mmh...this doesn't work and Clustermq get stuck during this step:
Submitting 3 worker jobs (ID: clustermq) ...
Running 3 calculations (0 objs/0 Mb common; 1 calls/chunk) ...
Which is the same step where it gets stuck when using SSH.
Ok, that makes it easier because now we know the issue is a connection problem from the workers to the login node, and not related to ssh.
Your login node likely has multiple network interfaces, and if a worker tries to connect to Sys.info()["nodename"]
it resolves to the wrong interface.
You likely need to set options(clustermq.host="<interface that accepts worker connections>"
.
Hello,
I am currently having the problem that jobs are sent to the workers, but it seems they never really start and thus get canceled due to the time limit. The code itself should be okay since it runs without problem using e.g. the
future
package and all I'm doing is to get nodename (fx <- function(x) {Sys.sleep(30); Sys.info()["nodename"]}
.My first guess is that the works cannot communicate because they don't find zeromq. I tried to set the
LD_LIBRARY_PATH
to the installation of zeromq, but this didn't help (setenv ('LD_LIBRARY_PATH', 'home/mhessel/zeromq-4.0.3/')
).Worker log
SSH log
Thank you very much