AsyncSSH Connection to Cisco Router with Jump Host

wz88 commented 5 months ago

Describe the bug When using tunnel option under the AsyncScrapli instantiation, as a key under transport_options -> asyncssh specifically, connectivity is established to the jump host but not the router.

A bit of context; I am trying to get netbox-config-diff plugin of NetBox's to work for devices sitting behind a jump host. It is currently configured to access devices directly with this config.

To Reproduce Steps to reproduce the behavior: Trying to write a simple script to run from my laptop (MacOS Sonoma 14.5) by connecting to a jump host, authenticating using credentials, then extending the SSH connection to a router behind. The script is as follows

import asyncssh
import asyncio
from scrapli import AsyncScrapli

class SSHTunnel:
    """
    Class with a tailored `create_connection` method handling SSH connection via jump host
    """
    async def create_connection(self, protocol_factory,
                                _remote_host, _remote_port):
        """Open an SSH connection"""
        conn = await asyncssh.connect(<jump_host_ip>, port=<jump_host_port>, username="user", password="pass")
        return await conn.create_session(protocol_factory, encoding=None)

# kwargs to use for instantiating `AsyncScrapli` class
config = {
    "host": <cisco_router_ip>,
    "auth_username": "cisco",
    "auth_password": "cisco",
    "auth_secondary": "cisco",
    "platform": "cisco_iosxe",
    "ssh_known_hosts_file": <path_to_my_known_hosts>,
    "transport": "asyncssh",
    "transport_options": {
        "asyncssh": {
            "tunnel": SSHTunnel(),  # added the tunnel object here
            "kex_algs": [
                "curve25519-sha256",
                "curve25519-sha256@libssh.org",
                "curve448-sha512",
                "ecdh-sha2-nistp521",
                "ecdh-sha2-nistp384",
                "ecdh-sha2-nistp256",
                "ecdh-sha2-1.3.132.0.10",
                "diffie-hellman-group-exchange-sha256",
                "diffie-hellman-group14-sha256",
                "diffie-hellman-group15-sha512",
                "diffie-hellman-group16-sha512",
                "diffie-hellman-group17-sha512",
                "diffie-hellman-group18-sha512",
                "diffie-hellman-group14-sha256@ssh.com",
                "diffie-hellman-group14-sha1",
                "rsa2048-sha256",
                "diffie-hellman-group1-sha1",
                "diffie-hellman-group-exchange-sha1",
                "diffie-hellman-group-exchange-sha256",
            ],
            "encryption_algs": [
                "aes256-cbc",
                "aes192-cbc",
                "aes128-cbc",
                "3des-cbc",
                "aes256-ctr",
                "aes192-ctr",
                "aes128-ctr",
                "aes128-gcm@openssh.com",
                "chacha20-poly1305@openssh.com",
            ],
        },
    },
}

# function to instantiate `AsyncScrapli` and send commands over SSH
async def run(cmd):
  async with AsyncScrapli(**config) as conn:
      result = await conn.send_command(cmd)
      return result.result

loop = asyncio.get_event_loop()
print(loop.run_until_complete(run("show run")))  # fetching device configuration

Expected behavior SSHing to the router should work normally, especially that using ssh -J jump_host_user@jump_host_ip:jump_host_port cisco_router_username@cisco_router_ip works fine. SSH connection is established successfully to the jump host, but fails to connect to the router

Stack Trace Logs and debugging showing the behaviour here. Please search by <<< to find comments within the logs/traces

...
>>> print(loop.run_until_complete(run("show run")))
DEBUG:scrapli:AsyncScrapli factory initialized
INFO:scrapli:Driver '<class 'scrapli.driver.core.cisco_iosxe.async_driver.AsyncIOSXEDriver'>' selected from scrapli core drivers
DEBUG:scrapli.driver:attempting to resolve 'ssh_known_hosts file'
DEBUG:scrapli.driver:using 'path_to_known_hosts' as resolved 'ssh_known_hosts' file'
DEBUG:scrapli.driver:load core transport requested
DEBUG:scrapli.driver:core transport 'asyncssh' loaded successfully
DEBUG:scrapli.driver:generating combined network comms prompt pattern
DEBUG:scrapli.driver:setting 'comms_prompt_pattern' value to '(^[\w.\-@/:]{1,63}>$)|(^[\w.\-@/:]{1,63}#$)|(^[\w.\-@/:]{1,63}\([\w.\-@/:+]{0,32}\)#$)|(^([\w.\-@/+>:]+\(tcl\)[>#]|\+>)$)'
INFO:scrapli.driver:opening connection to 'cisco_router_ip' on port '22'
DEBUG:scrapli.transport:opening transport connection to 'cisco_router_ip' on port '22'
DEBUG:scrapli.transport:Attempting to validate cisco_router_ip public key is in known hosts
INFO:asyncssh:Opening SSH connection to cisco_router_ip, port 22 via SSH tunnel
DEBUG:asyncssh:Reading config from "path_to_ssh_config"
INFO:asyncssh:Opening SSH connection to jump_host_ip, port jump_host_port
INFO:asyncssh:[conn=0] Connected to SSH server at jump_host_ip, port jump_host_port
INFO:asyncssh:[conn=0]   Local address: my_local_ip, port 52510
INFO:asyncssh:[conn=0]   Peer address: jump_host_ip, port jump_host_port
DEBUG:asyncssh:[conn=0] Sending version SSH-2.0-AsyncSSH_2.14.2
DEBUG:asyncssh:[conn=0] Received version SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5
DEBUG:asyncssh:[conn=0] Requesting key exchange
DEBUG:asyncssh:[conn=0]   Key exchange algs: curve25519-sha256,curve25519-sha256@libssh.org,curve448-sha512,ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,ecdh-sha2-1.3.132.0.10,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group15-sha512,diffie-hellman-group16-sha512,diffie-hellman-group17-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256@ssh.com,diffie-hellman-group14-sha1,rsa2048-sha256,ext-info-c,kex-strict-c-v00@openssh.com
DEBUG:asyncssh:[conn=0]   Host key algs: ssh-ed25519,ecdsa-sha2-nistp256
DEBUG:asyncssh:[conn=0]   Encryption algs: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
DEBUG:asyncssh:[conn=0]   MAC algs: hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-sha256-2@ssh.com,hmac-sha224@ssh.com,hmac-sha256@ssh.com,hmac-sha384@ssh.com,hmac-sha512@ssh.com
DEBUG:asyncssh:[conn=0]   Compression algs: zlib@openssh.com,none
DEBUG:asyncssh:[conn=0] Received key exchange request
DEBUG:asyncssh:[conn=0]   Key exchange algs: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256
DEBUG:asyncssh:[conn=0]   Host key algs: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
DEBUG:asyncssh:[conn=0]   Client to server:
DEBUG:asyncssh:[conn=0]     Encryption algs: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
DEBUG:asyncssh:[conn=0]     MAC algs: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
DEBUG:asyncssh:[conn=0]     Compression algs: none,zlib@openssh.com
DEBUG:asyncssh:[conn=0]   Server to client:
DEBUG:asyncssh:[conn=0]     Encryption algs: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
DEBUG:asyncssh:[conn=0]     MAC algs: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
DEBUG:asyncssh:[conn=0]     Compression algs: none,zlib@openssh.com
DEBUG:asyncssh:[conn=0] Beginning key exchange
DEBUG:asyncssh:[conn=0]   Key exchange alg: curve25519-sha256
DEBUG:asyncssh:[conn=0]   Client to server:
DEBUG:asyncssh:[conn=0]     Encryption alg: chacha20-poly1305@openssh.com
DEBUG:asyncssh:[conn=0]     MAC alg: chacha20-poly1305@openssh.com
DEBUG:asyncssh:[conn=0]     Compression alg: zlib@openssh.com
DEBUG:asyncssh:[conn=0]   Server to client:
DEBUG:asyncssh:[conn=0]     Encryption alg: chacha20-poly1305@openssh.com
DEBUG:asyncssh:[conn=0]     MAC alg: chacha20-poly1305@openssh.com
DEBUG:asyncssh:[conn=0]     Compression alg: zlib@openssh.com
DEBUG:asyncssh:[conn=0] Requesting service ssh-userauth
DEBUG:asyncssh:[conn=0] Completed key exchange
DEBUG:asyncssh:[conn=0] Received extension info
DEBUG:asyncssh:[conn=0]   server-sig-algs: ssh-ed25519,sk-ssh-ed25519@openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com
DEBUG:asyncssh:[conn=0] Request for service ssh-userauth accepted
INFO:asyncssh:[conn=0] Beginning auth for user my_username
DEBUG:asyncssh:[conn=0] Remaining auth methods: publickey,password
DEBUG:asyncssh:[conn=0] Preferred auth methods: gssapi-keyex,gssapi-with-mic,hostbased,publickey,keyboard-interactive,password
DEBUG:asyncssh:[conn=0] Trying public key auth with rsa-sha2-256 key
DEBUG:asyncssh:[conn=0] Remaining auth methods: publickey,password
DEBUG:asyncssh:[conn=0] Preferred auth methods: gssapi-keyex,gssapi-with-mic,hostbased,publickey,keyboard-interactive,password
DEBUG:asyncssh:[conn=0] Trying password auth
INFO:asyncssh:[conn=0] Auth for user my_username succeeded                   <<< auth to jump host succeeded here
DEBUG:asyncssh:[conn=0, chan=0] Set write buffer limits: low-water=16384, high-water=65536
INFO:asyncssh:[conn=0, chan=0] Requesting new SSH session                    <<< SSH connection to the router commences
DEBUG:asyncssh:[conn=0, chan=0]   Initial recv window 2097152, packet size 32768
DEBUG:asyncssh:[conn=0] Received unknown global request: hostkeys-00@openssh.com
DEBUG:asyncssh:[conn=0, chan=0]   Initial send window 0, packet size 32768
INFO:asyncssh:[conn=1] Connected to SSH server at cisco_router_ip, port 22
INFO:asyncssh:[conn=1]   Local address: my_local_ip, port 52510
INFO:asyncssh:[conn=1]   Peer address: jump_host_ip, port jump_host_port
DEBUG:asyncssh:[conn=1] Sending version SSH-2.0-AsyncSSH_2.14.2
DEBUG:asyncssh:[conn=0, chan=0] Sending 25 data bytes
INFO:asyncssh:[conn=0, chan=0]   Interactive shell requested                 <<< creating a channel resorts to default interactive shell
DEBUG:asyncssh:[conn=0, chan=0] Received window adjust of 2097152 bytes, new window 2097152
DEBUG:asyncssh:[conn=0, chan=0] Received 1165 data bytes
DEBUG:asyncssh:[conn=0, chan=0] Reading from channel started
DEBUG:asyncssh:[conn=0, chan=0] Received 44 data bytes from STDERR           <<< getting an error
INFO:asyncssh:[conn=1] Aborting connection
INFO:asyncssh:[conn=0, chan=0] Aborting channel
INFO:asyncssh:[conn=1] Connection closed
CRITICAL:scrapli.transport:timed out opening connection to device
Traceback (most recent call last):
  File "/Users/my_username/miniconda3/lib/python3.10/asyncio/tasks.py", line 650, in _wrap_awaitable
    return (yield from awaitable.__await__())
  File "/Users/my_username/.local/lib/python3.10/site-packages/asyncssh/connection.py", line 8269, in connect
    return await asyncio.wait_for(
  File "/Users/my_username/miniconda3/lib/python3.10/asyncio/tasks.py", line 408, in wait_for
    return await fut
  File "/Users/my_username/.local/lib/python3.10/site-packages/asyncssh/connection.py", line 436, in _connect
    await options.waiter
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/my_username/miniconda3/lib/python3.10/asyncio/tasks.py", line 456, in wait_for
    return fut.result()
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/my_username/.local/lib/python3.10/site-packages/scrapli/transport/plugins/asyncssh/transport.py", line 194, in open
    self.session = await asyncio.wait_for(
  File "/Users/my_username/miniconda3/lib/python3.10/asyncio/tasks.py", line 458, in wait_for
    raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/my_username/miniconda3/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "<stdin>", line 2, in run
  File "/Users/my_username/.local/lib/python3.10/site-packages/scrapli/driver/base/async_driver.py", line 45, in __aenter__
    await self.open()
  File "/Users/my_username/.local/lib/python3.10/site-packages/scrapli/driver/base/async_driver.py", line 87, in open
    await self.transport.open()
  File "/Users/my_username/.local/lib/python3.10/site-packages/scrapli/transport/plugins/asyncssh/transport.py", line 205, in open
    raise ScrapliAuthenticationFailed(msg) from exc
scrapli.exceptions.ScrapliAuthenticationFailed: timed out opening connection to device

OS (please complete the following information):

OS: MacOS Sonoma 14.5
asyncssh version: 2.14.2
scrapli version: 2023.7.30

Additional context

The sequence is - scrapli executes create_session method from the first asyncssh connection to the jump host under my tailored SSHTunnel. That in turn calls create_session that attempts to create a channel. My guess is it fails to create the channel, but I do not know why!

Just to point out; using the same config dictionary to instantiate the AsyncCrapli class without tunnel key under transport_options -> asyncssh and running the script from an ubuntu machine sitting within the same LAN of the router works just fine as shown below


>>> import asyncssh
>>> import asyncio
>>> from scrapli import AsyncScrapli
>>> config = {
...     "host": "cisco_router_ip",
...     "auth_username": "cisco",
...     "auth_password": "cisco",
...     "auth_secondary": "cisco",
...     "platform": "cisco_iosxe",
...     "transport": "asyncssh",
...     "transport_options": {
...         "asyncssh": {
...             "kex_algs": [
...                 "curve25519-sha256",
...                 "curve25519-sha256@libssh.org",
...                 "curve448-sha512",
...                 "ecdh-sha2-nistp521",
...                 "ecdh-sha2-nistp384",
...                 "ecdh-sha2-nistp256",
...                 "ecdh-sha2-1.3.132.0.10",
...                 "diffie-hellman-group-exchange-sha256",
...                 "diffie-hellman-group14-sha256",
...                 "diffie-hellman-group15-sha512",
...                 "diffie-hellman-group16-sha512",
...                 "diffie-hellman-group17-sha512",
...                 "diffie-hellman-group18-sha512",
...                 "diffie-hellman-group14-sha256@ssh.com",
...                 "diffie-hellman-group14-sha1",
...                 "rsa2048-sha256",
...                 "diffie-hellman-group1-sha1",
...                 "diffie-hellman-group-exchange-sha1",
...                 "diffie-hellman-group-exchange-sha256",
...             ],
...             "encryption_algs": [
...                 "aes256-cbc",
...                 "aes192-cbc",
...                 "aes128-cbc",
...                 "3des-cbc",
...                 "aes256-ctr",
...                 "aes192-ctr",
...                 "aes128-ctr",
...                 "aes128-gcm@openssh.com",
...                 "chacha20-poly1305@openssh.com",
...             ],
...         },
...     },
... }
>>> async def run(cmd):
...   async with AsyncScrapli(**config) as conn:
...       result = await conn.send_command(cmd)
...       return result.result
... 
>>> loop = asyncio.get_event_loop()
>>> print(loop.run_until_complete(run("show run")))
Building configuration...

Current configuration : 6495 bytes ! ! Last configuration change at 23:35:18 UTC Mon May 27 2024 ! ...

Doing some tedious `pdb` tracing, I found the failure to be [here](https://github.com/carlmontanari/scrapli/blob/main/scrapli/transport/plugins/asyncssh/transport.py#L195-L198) where the objects being used are as follow

""" connect: <asyncssh.misc._ACMWrapper object at 0x104826bf0>

conn_args: dictionary with the following values - noting that 'tunnel' key has the instance of my 'SSHTunnel' as value {'client_keys': '', 'password': 'cisco', 'preferred_auth': ('publickey', 'keyboard-interactive', 'password'), 'host': 'cisco_router_ip', 'port': 22, 'username': 'cisco', 'known_hosts': None, 'agent_path': None, 'config': '', 'tunnel': <main.SSHTunnel object at 0x104078370>, 'kex_algs': ['curve25519-sha256', 'curve25519-sha256@libssh.org', 'curve448-sha512', 'ecdh-sha2-nistp521', 'ecdh-sha2-nistp384', 'ecdh-sha2-nistp256', 'ecdh-sha2-1.3.132.0.10', 'diffie-hellman-group-exchange-sha256', 'diffie-hellman-group14-sha256', 'diffie-hellman-group15-sha512', 'diffie-hellman-group16-sha512', 'diffie-hellman-group17-sha512', 'diffie-hellman-group18-sha512', 'diffie-hellman-group14-sha256@ssh.com', 'diffie-hellman-group14-sha1', 'rsa2048-sha256', 'diffie-hellman-group1-sha1', 'diffie-hellman-group-exchange-sha1', 'diffie-hellman-group-exchange-sha256'], 'encryption_algs': ['aes256-cbc', 'aes192-cbc', 'aes128-cbc', '3des-cbc', 'aes256-ctr', 'aes192-ctr', 'aes128-ctr', 'aes128-gcm@openssh.com', 'chacha20-poly1305@openssh.com']}

self._base_transport_args.timeout_socket: 15.0

self._base_transport_args: BaseTransportArgs instance with the following values BaseTransportArgs(transport_options={'asyncssh': {'tunnel': <main.SSHTunnel object at 0x104078370>, 'kex_algs': ['curve25519-sha256', 'curve25519-sha256@libssh.org', 'curve448-sha512', 'ecdh-sha2-nistp521', 'ecdh-sha2-nistp384', 'ecdh-sha2-nistp256', 'ecdh-sha2-1.3.132.0.10', 'diffie-hellman-group-exchange-sha256', 'diffie-hellman-group14-sha256', 'diffie-hellman-group15-sha512', 'diffie-hellman-group16-sha512', 'diffie-hellman-group17-sha512', 'diffie-hellman-group18-sha512', 'diffie-hellman-group14-sha256@ssh.com', 'diffie-hellman-group14-sha1', 'rsa2048-sha256', 'diffie-hellman-group1-sha1', 'diffie-hellman-group-exchange-sha1', 'diffie-hellman-group-exchange-sha256'], 'encryption_algs': ['aes256-cbc', 'aes192-cbc', 'aes128-cbc', '3des-cbc', 'aes256-ctr', 'aes192-ctr', 'aes128-ctr', 'aes128-gcm@openssh.com', 'chacha20-poly1305@openssh.com']}}, host='cisco_router_ip', port=22, timeout_socket=15.0, timeout_transport=30.0, logging_uid='')

BaseTransportArgs: dataclass - https://github.com/carlmontanari/scrapli/blob/a9cadea73e492cb7362033b09dc8821bc7e0d0e7/scrapli/transport/base/base_transport.py#L11-L17 """ self.session = await asyncio.wait_for( connect(**conn_args), timeout=self._base_transport_args.timeout_socket, )

I can confirm that using pure `asyncssh` and passing the first runtime context as a `tunnel` parameter to the second instead of reusing it with `.create_session` method works as shown below.

import asyncssh import asyncio

async def tunneled_ssh(): ... async with asyncssh.connect("jump_host_ip", port=jump_host_port, username="jumphost_username", password="jumphost_password") as tunnel: ... async with asyncssh.connect("cisco_router_ip", tunnel=tunnel, username="cisco", password="cisco") as conn: ... return await conn.run("show clock") ...

loop = asyncio.get_event_loop()
:1: DeprecationWarning: There is no current event loop print(loop.run_until_complete(tunneled_ssh()).stdout)

*02:53:34.969 UTC Mon May 27 2024

ronf commented 5 months ago

Is there a reason you aren't using the native AsyncSSH support for jump hosts? It could be as simple as passing in a string as a tunnel argument, but if you need to specify usernames and passwords, you could do the initial connect to the jump host yourself and then pass the resulting SSHConnection in as the value of the tunnel argument, in place of your custom SSHTunnel() class.

wz88 commented 5 months ago

thanks for the prompt reply. Are we talking about proxy_command for natively supporting jump hosts? If so, I will need a way to pass the jump host's creds explicitly. With the double SSH connection solution, I gave it a go and it seems to be working up until authenticating to the router, but then the connection is aborted for some reason without any details on why...

Here is the new SSHTunnel

class SSHTunnel:
    async def create_connection(self, protocol_factory,
                                _remote_host, _remote_port):
        """Open an SSH connection over an interactive SSH channel"""
        tunn = await asyncssh.connect("jump_host_ip", port=jump_host_port, username="jump_host_username", password="jump_host_password")
        # using parameters passed to the method
        sess = await asyncssh.connect(_remote_host, tunnel=tunn, username=protocol_factory()._username, password=protocol_factory()._password)
        # returning a tuple with the SSHClientConnection
        return [None, sess]

It connects to the jump host, then to the router but dies there... please search by <<< to find comments in the logs

INFO:asyncssh:Opening SSH connection to cisco_router_ip, port 22 via SSH tunnel
DEBUG:asyncssh:Reading config from "/Users/my_username/.ssh/config"
INFO:asyncssh:Opening SSH connection to jump_host_ip, port jump_host_port
INFO:asyncssh:[conn=52] Connected to SSH server at jump_host_ip, port jump_host_port
INFO:asyncssh:[conn=52]   Local address: my_local_ip, port 57750
INFO:asyncssh:[conn=52]   Peer address: jump_host_ip, port jump_host_port
DEBUG:asyncssh:[conn=52] Sending version SSH-2.0-AsyncSSH_2.14.2
DEBUG:asyncssh:[conn=52] Received version SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5
DEBUG:asyncssh:[conn=52] Requesting key exchange
DEBUG:asyncssh:[conn=52]   Key exchange algs: curve25519-sha256,curve25519-sha256@libssh.org,curve448-sha512,ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,ecdh-sha2-1.3.132.0.10,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group15-sha512,diffie-hellman-group16-sha512,diffie-hellman-group17-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256@ssh.com,diffie-hellman-group14-sha1,rsa2048-sha256,ext-info-c,kex-strict-c-v00@openssh.com
DEBUG:asyncssh:[conn=52]   Host key algs: ssh-ed25519,ecdsa-sha2-nistp256
DEBUG:asyncssh:[conn=52]   Encryption algs: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
DEBUG:asyncssh:[conn=52]   MAC algs: hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-sha256-2@ssh.com,hmac-sha224@ssh.com,hmac-sha256@ssh.com,hmac-sha384@ssh.com,hmac-sha512@ssh.com
DEBUG:asyncssh:[conn=52]   Compression algs: zlib@openssh.com,none
DEBUG:asyncssh:[conn=52] Received key exchange request
DEBUG:asyncssh:[conn=52]   Key exchange algs: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256
DEBUG:asyncssh:[conn=52]   Host key algs: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
DEBUG:asyncssh:[conn=52]   Client to server:
DEBUG:asyncssh:[conn=52]     Encryption algs: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
DEBUG:asyncssh:[conn=52]     MAC algs: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
DEBUG:asyncssh:[conn=52]     Compression algs: none,zlib@openssh.com
DEBUG:asyncssh:[conn=52]   Server to client:
DEBUG:asyncssh:[conn=52]     Encryption algs: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
DEBUG:asyncssh:[conn=52]     MAC algs: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
DEBUG:asyncssh:[conn=52]     Compression algs: none,zlib@openssh.com
DEBUG:asyncssh:[conn=52] Beginning key exchange
DEBUG:asyncssh:[conn=52]   Key exchange alg: curve25519-sha256
DEBUG:asyncssh:[conn=52]   Client to server:
DEBUG:asyncssh:[conn=52]     Encryption alg: chacha20-poly1305@openssh.com
DEBUG:asyncssh:[conn=52]     MAC alg: chacha20-poly1305@openssh.com
DEBUG:asyncssh:[conn=52]     Compression alg: zlib@openssh.com
DEBUG:asyncssh:[conn=52]   Server to client:
DEBUG:asyncssh:[conn=52]     Encryption alg: chacha20-poly1305@openssh.com
DEBUG:asyncssh:[conn=52]     MAC alg: chacha20-poly1305@openssh.com
DEBUG:asyncssh:[conn=52]     Compression alg: zlib@openssh.com
DEBUG:asyncssh:[conn=52] Requesting service ssh-userauth
DEBUG:asyncssh:[conn=52] Completed key exchange
DEBUG:asyncssh:[conn=52] Received extension info
DEBUG:asyncssh:[conn=52]   server-sig-algs: ssh-ed25519,sk-ssh-ed25519@openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com
DEBUG:asyncssh:[conn=52] Request for service ssh-userauth accepted
INFO:asyncssh:[conn=52] Beginning auth for user my_username
DEBUG:asyncssh:[conn=52] Remaining auth methods: publickey,password
DEBUG:asyncssh:[conn=52] Preferred auth methods: gssapi-keyex,gssapi-with-mic,hostbased,publickey,keyboard-interactive,password
DEBUG:asyncssh:[conn=52] Trying public key auth with rsa-sha2-256 key
DEBUG:asyncssh:[conn=52] Remaining auth methods: publickey,password
DEBUG:asyncssh:[conn=52] Preferred auth methods: gssapi-keyex,gssapi-with-mic,hostbased,publickey,keyboard-interactive,password
DEBUG:asyncssh:[conn=52] Trying password auth
INFO:asyncssh:[conn=52] Auth for user my_username succeeded
DEBUG:asyncssh:Reading config from "/Users/my_username/.ssh/config"
DEBUG:asyncssh:[conn=52] Received unknown global request: hostkeys-00@openssh.com
INFO:asyncssh:[conn=52] Opening SSH connection to cisco_router_ip, port 22 via SSH tunnel
INFO:asyncssh:[conn=52] Opening direct TCP connection to cisco_router_ip, port 22
INFO:asyncssh:[conn=52]   Client address: dynamic port
DEBUG:asyncssh:[conn=52, chan=0] Set write buffer limits: low-water=16384, high-water=65536
DEBUG:asyncssh:[conn=52, chan=0]   Initial recv window 2097152, packet size 32768
DEBUG:asyncssh:[conn=52, chan=0]   Initial send window 2097152, packet size 32768
INFO:asyncssh:[conn=55] Connected to SSH server at cisco_router_ip, port 22
INFO:asyncssh:[conn=55]   Local address: my_local_ip, port 57750
INFO:asyncssh:[conn=55]   Peer address: dynamic port
DEBUG:asyncssh:[conn=55] Sending version SSH-2.0-AsyncSSH_2.14.2
DEBUG:asyncssh:[conn=52, chan=0] Sending 25 data bytes
DEBUG:asyncssh:[conn=52, chan=0] Reading from channel started
DEBUG:asyncssh:[conn=52, chan=0] Received 19 data bytes
DEBUG:asyncssh:[conn=55] Received version SSH-2.0-Cisco-1.25
DEBUG:asyncssh:[conn=55] Requesting key exchange
DEBUG:asyncssh:[conn=55]   Key exchange algs: curve25519-sha256,curve25519-sha256@libssh.org,curve448-sha512,ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,ecdh-sha2-1.3.132.0.10,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group15-sha512,diffie-hellman-group16-sha512,diffie-hellman-group17-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256@ssh.com,diffie-hellman-group14-sha1,rsa2048-sha256,ext-info-c,kex-strict-c-v00@openssh.com
DEBUG:asyncssh:[conn=55]   Host key algs: rsa-sha2-256,rsa-sha2-512,ssh-rsa-sha224@ssh.com,ssh-rsa-sha256@ssh.com,ssh-rsa-sha384@ssh.com,ssh-rsa-sha512@ssh.com,ssh-rsa
DEBUG:asyncssh:[conn=55]   Encryption algs: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
DEBUG:asyncssh:[conn=55]   MAC algs: hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-sha256-2@ssh.com,hmac-sha224@ssh.com,hmac-sha256@ssh.com,hmac-sha384@ssh.com,hmac-sha512@ssh.com
DEBUG:asyncssh:[conn=55]   Compression algs: zlib@openssh.com,none
DEBUG:asyncssh:[conn=52, chan=0] Sending 1360 data bytes
DEBUG:asyncssh:[conn=52, chan=0] Received 504 data bytes
DEBUG:asyncssh:[conn=55] Received key exchange request
DEBUG:asyncssh:[conn=55]   Key exchange algs: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha1
DEBUG:asyncssh:[conn=55]   Host key algs: rsa-sha2-512,rsa-sha2-256,ssh-rsa
DEBUG:asyncssh:[conn=55]   Client to server:
DEBUG:asyncssh:[conn=55]     Encryption algs: aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
DEBUG:asyncssh:[conn=55]     MAC algs: hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
DEBUG:asyncssh:[conn=55]     Compression algs: none
DEBUG:asyncssh:[conn=55]   Server to client:
DEBUG:asyncssh:[conn=55]     Encryption algs: aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
DEBUG:asyncssh:[conn=55]     MAC algs: hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
DEBUG:asyncssh:[conn=55]     Compression algs: none
DEBUG:asyncssh:[conn=55] Beginning key exchange
DEBUG:asyncssh:[conn=55]   Key exchange alg: ecdh-sha2-nistp521
DEBUG:asyncssh:[conn=52, chan=0] Sending 152 data bytes
DEBUG:asyncssh:[conn=52, chan=0] Received 560 data bytes
DEBUG:asyncssh:[conn=52, chan=0] Received 152 data bytes
DEBUG:asyncssh:[conn=55]   Client to server:
DEBUG:asyncssh:[conn=55]     Encryption alg: aes256-ctr
DEBUG:asyncssh:[conn=55]     MAC alg: hmac-sha2-256-etm@openssh.com
DEBUG:asyncssh:[conn=55]     Compression alg: none
DEBUG:asyncssh:[conn=55]   Server to client:
DEBUG:asyncssh:[conn=55]     Encryption alg: aes256-ctr
DEBUG:asyncssh:[conn=55]     MAC alg: hmac-sha2-256-etm@openssh.com
DEBUG:asyncssh:[conn=55]     Compression alg: none
DEBUG:asyncssh:[conn=52, chan=0] Sending 16 data bytes
DEBUG:asyncssh:[conn=55] Requesting service ssh-userauth
DEBUG:asyncssh:[conn=52, chan=0] Sending 52 data bytes
DEBUG:asyncssh:[conn=52, chan=0] Sending 68 data bytes
DEBUG:asyncssh:[conn=52, chan=0] Received 16 data bytes
DEBUG:asyncssh:[conn=55] Completed key exchange
DEBUG:asyncssh:[conn=52, chan=0] Received 68 data bytes
DEBUG:asyncssh:[conn=55] Request for service ssh-userauth accepted
INFO:asyncssh:[conn=55] Beginning auth for user cisco
DEBUG:asyncssh:[conn=52, chan=0] Sending 52 data bytes
DEBUG:asyncssh:[conn=52, chan=0] Sending 84 data bytes
DEBUG:asyncssh:[conn=52, chan=0] Received 100 data bytes
DEBUG:asyncssh:[conn=55] Remaining auth methods: publickey,keyboard-interactive,password
DEBUG:asyncssh:[conn=55] Preferred auth methods: gssapi-keyex,gssapi-with-mic,hostbased,publickey,keyboard-interactive,password
DEBUG:asyncssh:[conn=55] Trying public key auth with ssh-rsa key
DEBUG:asyncssh:[conn=52, chan=0] Sending 52 data bytes
DEBUG:asyncssh:[conn=52, chan=0] Sending 644 data bytes
DEBUG:asyncssh:[conn=52, chan=0] Received 100 data bytes
DEBUG:asyncssh:[conn=55] Remaining auth methods: publickey,keyboard-interactive,password
DEBUG:asyncssh:[conn=55] Preferred auth methods: gssapi-keyex,gssapi-with-mic,hostbased,publickey,keyboard-interactive,password
DEBUG:asyncssh:[conn=55] Trying keyboard-interactive auth
DEBUG:asyncssh:[conn=52, chan=0] Sending 52 data bytes
DEBUG:asyncssh:[conn=52, chan=0] Sending 116 data bytes
DEBUG:asyncssh:[conn=52, chan=0] Received 84 data bytes
DEBUG:asyncssh:[conn=52, chan=0] Sending 52 data bytes
DEBUG:asyncssh:[conn=52, chan=0] Sending 68 data bytes
DEBUG:asyncssh:[conn=52, chan=0] Received 52 data bytes
INFO:asyncssh:[conn=55] Auth for user cisco succeeded                <<< authentication succeeded
INFO:asyncssh:[conn=55] Aborting connection                          <<< then aborted the connection after around 10 seconds
INFO:asyncssh:[conn=52, chan=0] Aborting channel
INFO:asyncssh:[conn=55] Connection closed
CRITICAL:scrapli.transport:timed out opening connection to device
Traceback (most recent call last):
  File "/Users/my_username/miniconda3/lib/python3.10/asyncio/tasks.py", line 650, in _wrap_awaitable
    return (yield from awaitable.__await__())
  File "/Users/my_username/.local/lib/python3.10/site-packages/asyncssh/connection.py", line 8269, in connect
    return await asyncio.wait_for(
  File "/Users/my_username/miniconda3/lib/python3.10/asyncio/tasks.py", line 408, in wait_for
    return await fut
  File "/Users/my_username/.local/lib/python3.10/site-packages/asyncssh/connection.py", line 436, in _connect
    await options.waiter
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/my_username/miniconda3/lib/python3.10/asyncio/tasks.py", line 456, in wait_for
    return fut.result()
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/my_username/.local/lib/python3.10/site-packages/scrapli/transport/plugins/asyncssh/transport.py", line 194, in open
    self.session = await asyncio.wait_for(
  File "/Users/my_username/miniconda3/lib/python3.10/asyncio/tasks.py", line 458, in wait_for
    raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/my_username/miniconda3/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "<stdin>", line 3, in run
  File "/Users/my_username/.local/lib/python3.10/site-packages/scrapli/driver/base/async_driver.py", line 45, in __aenter__
    await self.open()
  File "/Users/my_username/.local/lib/python3.10/site-packages/scrapli/driver/base/async_driver.py", line 88, in open
    await self.transport.open()
  File "/Users/my_username/.local/lib/python3.10/site-packages/scrapli/transport/plugins/asyncssh/transport.py", line 205, in open
    raise ScrapliAuthenticationFailed(msg) from exc
scrapli.exceptions.ScrapliAuthenticationFailed: timed out opening connection to device

ronf commented 5 months ago

thanks for the prompt reply. Are we talking about proxy_command for natively supporting jump hosts? If so, I will need a way to pass the jump host's creds explicitly. With the double SSH connection solution, I gave it a go and it seems to be working up until authenticating to the router, but then the connection is aborted for some reason without any details on why...

No - AsyncSSH can support jump hosts directly, without the need for you to create your own tunnel class. A custom tunnel class would only be needed if you wanted to use some non-SSH protocol to tunnel over, like SOCKS or HTTP CONNECT.

The first example you showed where it worked with the double connect is an example of the native support. What I was suggesting is that you only do the first of these connect() calls (to the jump host) and then pass the SSHClientConnection you get back as the tunnel value in your Scrapli config:


tunn = await asyncssh.connect("jump_host_ip", port=jump_host_port, username="jump_host_username", password="jump_host_password")

config = {
    "host": <cisco_router_ip>,
    "auth_username": "cisco",
    "auth_password": "cisco",
    "auth_secondary": "cisco",
    "platform": "cisco_iosxe",
    "ssh_known_hosts_file": <path_to_my_known_hosts>,
    "transport": "asyncssh",
    "transport_options": {
        "asyncssh": {
            "tunnel": tunn,
             # Other transport options here
        },
    },
}

You might even be able to use the same jump host connection to connect to multiple target routers, if the jump host supports multiple simultaneous sessions. There's usually a limit on the number of those you can open, though, so if you are connecting to lots of hosts you might need to create multiple jump host connections to get around that. You also need to be careful not to open connections too quickly. Many SSH servers will limit the number of simultaneous connections they will accept before authentication completes.

I can't immediately explain why your custom SSHTunnel class is failing, but I'd prefer to see if you can get the proper version of this working before digging into that. Creating your own class with a create_connection() method for this purpose has always been a hack, and not officially supported.

Actually, one thing I noticed is that you are opening an interactive shell session on the jump host. That's not generally the way jump hosts work. You need to open a direct tcp-ip connection. Otherwise, you'll be trying to send an SSH handshake when one end is still at a shell prompt on the jump host, rather than talking to sshd on the target host.

wz88 commented 5 months ago

Thanks a tonne, @ronf, for the help! I didn't fully understand how the setup of scrapli and asyncssh come together. Your suggestion worked; I am leaving a copy of the script here for reference/to help anybody looking for this in the future

import asyncssh
import asyncio
from scrapli import AsyncScrapli

async def config():
    return {
        "host": "cisco_router_ip",
        "auth_username": "cisco",
        "auth_password": "cisco",
        "auth_secondary": "cisco",
        "platform": "cisco_iosxe",
        "auth_strict_key": False,
        "ssh_known_hosts_file": "",
        "transport": "asyncssh",
        "transport_options": {
            "asyncssh": {
                "tunnel": await asyncssh.connect("jump_host_ip", port=jump_host_port, username="jump_host_username", password="jump_host_password", known_hosts=None),
                "kex_algs": [
                    "curve25519-sha256",
                    "curve25519-sha256@libssh.org",
                    "curve448-sha512",
                    "ecdh-sha2-nistp521",
                    "ecdh-sha2-nistp384",
                    "ecdh-sha2-nistp256",
                    "ecdh-sha2-1.3.132.0.10",
                    "diffie-hellman-group-exchange-sha256",
                    "diffie-hellman-group14-sha256",
                    "diffie-hellman-group15-sha512",
                    "diffie-hellman-group16-sha512",
                    "diffie-hellman-group17-sha512",
                    "diffie-hellman-group18-sha512",
                    "diffie-hellman-group14-sha256@ssh.com",
                    "diffie-hellman-group14-sha1",
                    "rsa2048-sha256",
                    "diffie-hellman-group1-sha1",
                    "diffie-hellman-group-exchange-sha1",
                    "diffie-hellman-group-exchange-sha256",
                ],
                "encryption_algs": [
                    "aes256-cbc",
                    "aes192-cbc",
                    "aes128-cbc",
                    "3des-cbc",
                    "aes256-ctr",
                    "aes192-ctr",
                    "aes128-ctr",
                    "aes128-gcm@openssh.com",
                    "chacha20-poly1305@openssh.com",
                ],
            },
        },
    }

async def run(cmd):
    local_config = await config()
    async with AsyncScrapli(**local_config) as conn:
        result = await conn.send_command(cmd)
        return result.result

loop = asyncio.get_event_loop()
print(loop.run_until_complete(run("show run")))

I am happy to close this issue if you are ok.

ronf commented 5 months ago

Sounds good - glad to hear you got this to work! I'll go ahead and close it.

Feel free to open another issue if you run into any other problems.

wz88 commented 5 months ago

Thanks a lot!

ronf / asyncssh

AsyncSSH Connection to Cisco Router with Jump Host #659