Ipython Notebooks - SSH tunnel in interactive jobs?

[rafwiewiora]

Running an IPython Notebook server on the head node with access through an SSH tunnel works well.

I was wondering if there is a way to, following starting an interactive job, tunnel into the job node and do the same thing? X-forwarding works in that case, but that's too slow to be useful.

[tatarsky]

Do you know the port your iPython notebook will open on a node?

[tatarsky]

As I have to go what I'm getting at here is if you know what node (which you can ask for by hostname in qsub) and the port you can use the same style tunnel to the head node but connect the local port to the same of the node. Aka this example in which you intend to run the ipython on gpu-1-4 using a qsub which states that and on port 4444.

ssh -L 4444:gpu-1-4:4444 hal.cbio.mskcc.org

That would allow you to talk to that ipython on localhost:4444 on your remote end.

Again, this requires a bit of "placement" which I can elaborate on if this doesn't make sense.

[tatarsky]

Oh, and that assumes you don't need 10G speeds ;) If you do, be sure you forward to the 10G side which in the case above is gpu-1-4.fast.

[rafwiewiora]

Sorry for not replying earlier!

You can choose the port yourself for the notebook, so whichever.

Alright, that makes complete sense! But, it's not working for me sadly - 4444 doesn't actually respond in anyway, simply nothing on localhost:4444, but I tried ports 8010 or 20000 for example, I'm doing:

ssh -L 8010:gpu-1-4:8010 user.name@hal.cbio.mskcc.org

qsub -I -q active -l nodes=gpu-1-4:ppn=1
ipython notebook --no-browser --port=8010

and on trying to connect to localhost:8010 the notebook server gives:

channel 3: open failed: connect failed: Connection refused

[tatarsky]

4444 was just a "example port" ;)

So the question for your example above is if ipython binds to an external interface (which my tunnel example requires) or localhost with just that --port=8010.

If it binds to localhost you will indeed have an issue as you describe. I'm not overly ipython familiar.

Are you running an ipython somewhere now?

[tatarsky]

Several examples out there seem to imply ipython binds to localhost by default. I'm not overly able to test at the moment but they seem to imply feeding it in addition to the --port=XXXX

--ip='*'

You force ipython to bind to all available interfaces. Obviously I would be careful where you do that but in the case of the nodes its an internal network so you are reasonable safe for testing purposes.

[tatarsky]

I would also review some of the password security settings just in case you go down this path. To prevent accidents with perhaps somebody else running ipython notebooks ;)

[rafwiewiora]

Hmm, the --ip='*' gives me the same error actually. I have a notebook running now with that option on --port=8010 on gpu-1-4, job 6186355.mskcc-fe1 - will that help you look into why that's still not ok?

[tatarsky]

So I see your ipython bound to all interfaces

ipython   31180 (removed)   3u  IPv4 1573940351      0t0  TCP *:8010 (LISTEN)

And when I do this from my remote location:

ssh -L 8010:gpu-1-4:8010 hal.cbio.mskcc.org

And then with my browser connect to:

http://localhost:8010/

I get your ipython session (see comment about passwords ;)

[tatarsky]

BTW, where are you trying to connect from (Just making sure you are talking to localhost on your remote system, not hal)

[rafwiewiora]

Ok, works like a charm now! Looks like the problem was I was SSH-ing through a personal VPN, when I get off the VPN and connect from the MSK network now it works. Why is this? I guess this has to do with your question about where I'm connecting from!

So what do you mean about passwords - do I need to protect the notebook server or the SSH tunnel itself?

And an extra question :) : how do I get the names of all the nodes?

[tatarsky]

Ok, sounds like your remote side perhaps has some filtering going on at a level I wouldn't attempt to debug here. (public git)

Passwords would be for your ipython notebook if concerned about another Hal user accidentally connecting to it as you are opening to the internal node network. To stress, that network is exclusive to hal nodes and users but I would be careful as I don't know what you are using the notebook for. Can discuss this more with you Monday if concerned.

The names of the nodes there are a few ways to query them but I tend to do this and dump it to a list:

pbsnodes -a|egrep '^[a-z]'

Its cheesy and there are other ways but they is semi-portable in Torque if the nodes name starts with a letter ;)

[tatarsky]

mdiag -n is another way but parsing left to user ;)

[rafwiewiora]

Alright, thanks so much for all the help! I think it's fine open to anyone on the network for now, nothing that I wouldn't have in a public github repo anyway.

Thanks again!

[tatarsky]

No prob.

[KjongLehmann]

I am not sure whether it is quite that easy. Please note that everybody with access to your ipython notebook will also be able to read and write from all directories accessible to you, that includes potentially restricted data you have access to. I suggest to consider this.

On Oct 24, 2015, at 9:10 PM, Rafal Wiewiora notifications@github.com wrote:

Alright, thanks so much for all the help! I think it's fine open to anyone on the network for now, nothing that I wouldn't have in a public github repo anyway.

Thanks again!

— Reply to this email directly or view it on GitHub.

[rafwiewiora]

Thanks a lot for pointing that out @KjongLehmann, I didn't think about the access to all directories issue. I'm running the server password / SSL protected now.

I guess this is good to close.

[tatarsky]

OK. Closing. Advise if you have further issues.