search

spyder-ide / spyder

Official repository for Spyder - The Scientific Python Development Environment
https://www.spyder-ide.org
MIT License
8.35k stars 1.62k forks source link

Remote kernel connection not working for cluster behind a login node #20170

Open ifariasg opened 1 year ago

ifariasg commented 1 year ago

Problem Description

I'm trying to connect to a remote kernel set up on a HPC cluster using spyder. I followed the guide provided by Spyder and also some recommendations provided by the cluster administrators. The issue that I have is that I get an endless "Connecting to kernel". I'm sure I can connect to the remote node via SSH directly via shell or any SSH client.

Once of hundreds of times I've tried the connection was successful but since I have not been unable to reproduce this.

Is there a way to activate some kind of verbose to see what paramiko is doing under the hood? The cluster needs to be accessed through a proxy jump set up in a config file in the .ssh folder and it is impossible for me to check what paramiko/spyder are doing to connect.

Both the local machine and the server are running the same version of spyder-kernels.

What steps reproduce the problem?

  1. Login to Cluster on a particular node and launch a kernel python -m spyder_kernels.console
  2. Get the Json to my local PC and connect through Spyder
  3. Stuck on and endless connecting to kernel with no errors or tracebacks

Versions

Dependencies

# Mandatory:
atomicwrites >=1.2.0          :  1.4.0 (OK)
chardet >=2.0.0               :  4.0.0 (OK)
cloudpickle >=0.5.0           :  2.0.0 (OK)
cookiecutter >=1.6.0          :  1.7.3 (OK)
diff_match_patch >=20181111   :  20200713 (OK)
intervaltree >=3.0.2          :  3.1.0 (OK)
IPython >=7.31.1;<8.0.0       :  7.32.0 (OK)
jedi >=0.17.2;<0.19.0         :  0.18.1 (OK)
jellyfish >=0.7               :  0.9.0 (OK)
jsonschema >=3.2.0            :  4.4.0 (OK)
keyring >=17.0.0              :  23.4.0 (OK)
nbconvert >=4.0               :  6.4.5 (OK)
numpydoc >=0.6.0              :  1.2.1 (OK)
paramiko >=2.4.0              :  2.10.3 (OK)
parso >=0.7.0;<0.9.0          :  0.8.3 (OK)
pexpect >=4.4.0               :  4.8.0 (OK)
pickleshare >=0.4             :  0.7.5 (OK)
psutil >=5.3                  :  5.9.0 (OK)
pygments >=2.0                :  2.11.2 (OK)
pylint >=2.5.0;<3.0           :  2.13.1 (OK)
pylint_venv >=2.1.1           :  2.3.0 (OK)
pyls_spyder >=0.4.0           :  0.4.0 (OK)
pylsp >=1.6.0;<1.7.0          :  1.6.0 (OK)
pylsp_black >=1.2.0           :  1.2.1 (OK)
qdarkstyle >=3.0.2;<3.1.0     :  3.0.2 (OK)
qstylizer >=0.2.2             :  0.2.2 (OK)
qtawesome >=1.2.1             :  1.2.1 (OK)
qtconsole >=5.4.0;<5.5.0      :  5.4.0 (OK)
qtpy >=2.1.0                  :  2.2.1 (OK)
rtree >=0.9.7                 :  0.9.7 (OK)
setuptools >=49.6.0           :  61.2.0 (OK)
sphinx >=0.6.6                :  3.5.3 (OK)
spyder_kernels >=2.4.0;<2.5.0 :  2.4.0 (OK)
textdistance >=4.2.0          :  4.2.2 (OK)
three_merge >=0.1.1           :  0.1.1 (OK)
watchdog >=0.10.3             :  2.1.7 (OK)
zmq >=22.1.0                  :  24.0.1 (OK)

# Optional:
cython >=0.21                 :  None (NOK)
matplotlib >=3.0.0            :  3.5.1 (OK)
numpy >=1.7                   :  1.21.5 (OK)
pandas >=1.1.1                :  1.4.1 (OK)
scipy >=0.17.0                :  1.8.0 (OK)
sympy >=0.7.3                 :  None (NOK)
ccordoba12 commented 1 year ago

Hey @ifariasg, thanks for reporting. You said

Is there a way to activate some kind of verbose to see what paramiko is doing under the hood?

Yes, you need to start Spyder in a terminal with

spyder --debug-output terminal

to activate logging of Spyder and all its dependent libraries. If paramiko generates some kind of logging, it should be shown in the terminal.

The cluster needs to be accessed through a proxy jump set up in a config file in the .ssh folder and it is impossible for me to check what paramiko/spyder are doing to connect.

What we do through paramiko is open a SSH tunnel to the remote machine using your credentials. I really don't know how this could work in your case since the connection requires an additional step. Perhaps paramiko allows for a script to be executed after the connection.

Let us know what you find.

ifariasg commented 1 year ago

Yep, there is a problem with the SSH tunnel.

All I get is and endless stream of Secsh channel X open FAILED: Connection refused: Connect failed

ccordoba12 commented 1 year ago

You could try to create the tunnel manually, as described here:

https://github.com/ipython/ipython/wiki/Cookbook:-Connecting-to-a-remote-kernel-via-ssh

Once you have that working, I think you should be able to connect to that kernel from Spyder as you would for a local one.

ifariasg commented 1 year ago

Made it work! It required a couple of extra steps and its quite cumbersome but at least it is now possible.

So to give more context, the cluster I'm trying to connect requires a proxy jump of this style via the .ssh/config file:

Host server
  Port 22
  HostName <ip>

Host node*
  HostName %h
  ProxyJump server

Main issue here was that the spyder context menu for remote kernel connection points to a server adress via the format user@hostname:port. With this structure the config file is never read and thus the proxyjump never happens, particularly because Spyder (or paramiko?) adds the port to the SSH command always.

What I didn't know is that you can get the needed behavior by leaving the port and username blank and use as hostname user@nodeXX

This is not all though, this solves the proxy jump issue but still the SSH tunnel is not succesful via paramiko. To solve this I had to reformat the config file to use LocalForward ending with a structure like the following:

Host server
  Port 22
  HostName <ip>

Host node*
  HostName %h
  ProxyJump server
  LocalForward port1 127.0.0.1:port1
  LocalForward port2 127.0.0.1:port2
  LocalForward port3 127.0.0.1:port3
  LocalForward port4 127.0.0.1:port4
  LocalForward port5 127.0.0.1:port5

I then have to manually SSH to nodeXX and afterwards I can connect to the remote kernel with user@nodeXX as hostname.

So everytime I start a kernel I will have to update the config file. But hey, at least it works now!

ifariasg commented 1 year ago

Found another issue that stems from this, because the tunneling is no longer being handled by paramiko/Spyder, the variable explorer tunneling is also not being done. If I manually add the port I get with get_ipython().kernel.frontend_comm.comm_port to the config file the variable explorer works again. Related to #11538?

It would be nice to know what other ports Spyder is using in a remote kernel connection for any other features, as this behavior might apply to those.

ccordoba12 commented 1 year ago

Related to https://github.com/spyder-ide/spyder/issues/11538?

Actuallo of issue #14894, which is already fixed as you can see. The fix will be available in Spyder 6, to be released next year.

It would be nice to know what other ports Spyder is using in a remote kernel connection for any other features, as this behavior might apply to those.

There are no more ports.