Tools to connect to and interact with the Mila cluster
MIT License
60
stars
11
forks
source link
[v0.1.2] Intermittent connection errors ("ssh-copy-id appears to have failed", "An error occured while trying to establish a connection with mila") #102
Intermittent errors during mila init and other commands:
'ssh-copy-id mila' appears to have failed!
ERROR: An error happened while trying to establish a connection with mila
-The cluster might be under maintenance
Check #mila-cluster for updates on the state of the cluster
-Check the status of your connection to the cluster by ssh'ing onto it.
-Retry connecting with mila
-Try to exclude the node with -x mila parameter
For example:
$ mila -vvv init
(...)
Checking connection to compute nodes
(...)
[02/14/24 14:19:21] ERROR 2024-02-14 14:19:21,979 - ERROR - Exception (client): Error reading SSH protocol banner transport.py:1893
ERROR 2024-02-14 14:19:21,985 - ERROR - Traceback (most recent call last): transport.py:1891
ERROR 2024-02-14 14:19:21,988 - ERROR - File transport.py:1891
"/home/fabrice/miniconda3/envs/milatools/lib/python3.11/site-packages/paramiko/transport.py", line 2292, in _check_banner
ERROR 2024-02-14 14:19:21,991 - ERROR - buf = self.packetizer.readline(timeout) transport.py:1891
ERROR 2024-02-14 14:19:21,993 - ERROR - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ transport.py:1891
ERROR 2024-02-14 14:19:21,994 - ERROR - File transport.py:1891
"/home/fabrice/miniconda3/envs/milatools/lib/python3.11/site-packages/paramiko/packet.py", line 374, in readline
ERROR 2024-02-14 14:19:21,996 - ERROR - buf += self._read_timeout(timeout) transport.py:1891
ERROR 2024-02-14 14:19:21,998 - ERROR - ^^^^^^^^^^^^^^^^^^^^^^^^^^^ transport.py:1891
ERROR 2024-02-14 14:19:21,999 - ERROR - File transport.py:1891
"/home/fabrice/miniconda3/envs/milatools/lib/python3.11/site-packages/paramiko/packet.py", line 611, in _read_timeout
[02/14/24 14:19:22] ERROR 2024-02-14 14:19:22,001 - ERROR - raise socket.timeout() transport.py:1891
ERROR 2024-02-14 14:19:22,002 - ERROR - TimeoutError transport.py:1891
ERROR 2024-02-14 14:19:22,004 - ERROR - transport.py:1891
ERROR 2024-02-14 14:19:22,005 - ERROR - During handling of the above exception, another exception occurred: transport.py:1891
ERROR 2024-02-14 14:19:22,007 - ERROR - transport.py:1891
ERROR 2024-02-14 14:19:22,008 - ERROR - Traceback (most recent call last): transport.py:1891
ERROR 2024-02-14 14:19:22,009 - ERROR - File transport.py:1891
"/home/fabrice/miniconda3/envs/milatools/lib/python3.11/site-packages/paramiko/transport.py", line 2113, in run
ERROR 2024-02-14 14:19:22,010 - ERROR - self._check_banner() transport.py:1891
ERROR 2024-02-14 14:19:22,011 - ERROR - File transport.py:1891
"/home/fabrice/miniconda3/envs/milatools/lib/python3.11/site-packages/paramiko/transport.py", line 2296, in _check_banner
ERROR 2024-02-14 14:19:22,012 - ERROR - raise SSHException( transport.py:1891
ERROR 2024-02-14 14:19:22,013 - ERROR - paramiko.ssh_exception.SSHException: Error reading SSH protocol banner transport.py:1891
ERROR 2024-02-14 14:19:22,014 - ERROR - transport.py:1891
ERROR: An error happened while trying to establish a connection with mila
-The cluster might be under maintenance
Check #mila-cluster for updates on the state of the cluster
-Check the status of your connection to the cluster by ssh'ing onto it.
-Retry connecting with mila
-Try to exclude the node with -x mila parameter
These all seem to be caused by this "Error reading SSH protocol banner" error.
Intermittent errors during
mila init
and other commands:For example:
These all seem to be caused by this "Error reading SSH protocol banner" error.