search

ronf / asyncssh

AsyncSSH is a Python package which provides an asynchronous client and server implementation of the SSHv2 protocol on top of the Python asyncio framework.
Eclipse Public License 2.0
1.55k stars 151 forks source link

Connecting to Cisco IOS devices raises "ProtocolError: Incomplete Packet" #561

Closed JostLuebbe closed 1 year ago

JostLuebbe commented 1 year ago

Version/OS Information

$ python --version
Python 3.11.3
$ pip freeze | grep asyncssh
asyncssh==2.13.1
$ pip freeze | grep scrapli
scrapli==2023.1.30
$ uname -a
Linux 5.4.0-109-generic #123~18.04.1-Ubuntu SMP Fri Apr 8 09:48:52 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Problem Description

We're currently using the scrapli library to automate network device config changes. Scrapli uses the asyncssh library as a transport when connecting to devices in an asynchronous manner. This works really well for the vast majority of the devices in our fleet, but we're having issues with a small subset raising a "ProtocolError: Incomplete Packet" exception when Scrapli tries to enable on them while using the asyncssh transport. This issue does not occur if we use a different transport, such as connecting to the device in a synchronous manner, which is why we think the issue may be with asyncssh, and not Scrapli. We're hoping you might be able to give us more ideas on what we could do to fix this.

Additional Information

We've looked pretty extensively for others who might have had this issue, but the closest we could find is this issue. However, they were not even able to complete a connection to their device, whereas we're able to connect and authenticate using SSH username/password credentials, but when the enable string is sent, then the exception arises. We have tried sending minimal kex_algs/encryption_algs/mac_algs, but that hasn't fixed our issue.

Possible Root Cause

We've noticed that these MSG_IGNOREs of 5 bytes are always sent right before the incomplete packet happens.

2023-04-17 11:01:17,884 [DEBUG] [conn=0, pktid=21] Received MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 05                                   .....
2023-04-17 11:01:17,884 [INFO] [conn=0] Sending disconnect: Incomplete packet (2)

Example Code

async def _main(conn) -> None:
    async with conn as conn:
        result = await conn.send_command("show ver")

        print(result.result)

def main() -> None:
    device_name = "<DEVICE_NAME>"

    async_conn = AsyncScrapli(
        platform="cisco_iosxe",
        transport="asyncssh",
        host=device_name,
        port=22,
        auth_username=AD_USERNAME,
        auth_password=AD_PASSWORD,
        auth_secondary=ENABLE_SECRET,
        auth_strict_key=False,
        ssh_config_file="/etc/ssh/ssh_config",
    )

    print(async_conn)
    print(type(async_conn))
    print(async_conn.transport)

    asyncio.run(_main(async_conn))

Debug Output

2023-04-17 11:01:13,687 [DEBUG] AsyncScrapli factory initialized
2023-04-17 11:01:13,688 [INFO] Driver '<class 'scrapli.driver.core.cisco_iosxe.async_driver.AsyncIOSXEDriver'>' selected from scrapli core drivers
2023-04-17 11:01:13,688 [DEBUG] attempting to resolve 'ssh_config_file' file
2023-04-17 11:01:13,688 [DEBUG] using '/etc/ssh/ssh_config' as resolved 'ssh_config_file' file'
2023-04-17 11:01:13,689 [DEBUG] load core transport requested
2023-04-17 11:01:13,689 [DEBUG] core transport 'asyncssh' loaded successfully
2023-04-17 11:01:13,689 [DEBUG] generating combined network comms prompt pattern
2023-04-17 11:01:13,689 [DEBUG] setting 'comms_prompt_pattern' value to '(^[\w.\-@/:]{1,63}>$)|(^[\w.\-@/:]{1,63}#$)|(^[\w.\-@/:]{1,63}\([\w.\-@/:+]{0,32}\)#$)|(^([\w.\-@/+>:]+\(tcl\)[>#]|\+>)$)'
2023-04-17 11:01:13,689 [DEBUG] Scrapli factory initialized
2023-04-17 11:01:13,689 [INFO] Driver '<class 'scrapli.driver.core.cisco_iosxe.sync_driver.IOSXEDriver'>' selected from scrapli core drivers
2023-04-17 11:01:13,689 [DEBUG] attempting to resolve 'ssh_config_file' file
2023-04-17 11:01:13,689 [DEBUG] using '/etc/ssh/ssh_config' as resolved 'ssh_config_file' file'
2023-04-17 11:01:13,689 [DEBUG] load core transport requested
2023-04-17 11:01:13,690 [DEBUG] core transport 'system' loaded successfully
2023-04-17 11:01:13,690 [DEBUG] generating combined network comms prompt pattern
2023-04-17 11:01:13,690 [DEBUG] setting 'comms_prompt_pattern' value to '(^[\w.\-@/:]{1,63}>$)|(^[\w.\-@/:]{1,63}#$)|(^[\w.\-@/:]{1,63}\([\w.\-@/:+]{0,32}\)#$)|(^([\w.\-@/+>:]+\(tcl\)[>#]|\+>)$)'
Scrapli Driver <DEVICE_NAME>:22
<class 'scrapli.driver.core.cisco_iosxe.async_driver.AsyncIOSXEDriver'>
<scrapli.transport.plugins.asyncssh.transport.AsyncsshTransport object at 0x7f8548a45150>
2023-04-17 11:01:13,723 [DEBUG] Using selector: EpollSelector
2023-04-17 11:01:13,723 [INFO] opening connection to '<DEVICE_NAME>' on port '22'
2023-04-17 11:01:13,723 [DEBUG] opening transport connection to '<DEVICE_NAME>' on port '22'
2023-04-17 11:01:13,724 [DEBUG] Reading config from "/etc/ssh/ssh_config"
2023-04-17 11:01:13,727 [INFO] Opening SSH connection to <DEVICE_NAME>, port 22
2023-04-17 11:01:13,959 [INFO] [conn=0] Connected to SSH server at <DEVICE_NAME>, port 22
2023-04-17 11:01:13,959 [INFO] [conn=0]   Local address: <LOCAL_ADDRESS>, port 55554
2023-04-17 11:01:13,959 [INFO] [conn=0]   Peer address: <PEER_ADDRESS>, port 22
2023-04-17 11:01:13,959 [DEBUG] [conn=0] Sending version SSH-2.0-AsyncSSH_2.13.1
2023-04-17 11:01:14,188 [DEBUG] [conn=0] Received version SSH-2.0-Cisco-1.25
2023-04-17 11:01:14,188 [DEBUG] [conn=0] Requesting key exchange
2023-04-17 11:01:14,188 [DEBUG] [conn=0]   Key exchange algs: curve25519-sha256,curve25519-sha256@libssh.org,curve448-sha512,ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,ecdh-sha2-1.3.132.0.10,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group15-sha512,diffie-hellman-group16-sha512,diffie-hellman-group17-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256@ssh.com,diffie-hellman-group14-sha1,rsa2048-sha256,diffie-hellman-group1-sha1,ext-info-c
2023-04-17 11:01:14,188 [DEBUG] [conn=0]   Host key algs: sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ssh-ed448-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-1.3.132.0.10-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,ssh-ed25519,ssh-ed448,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256,ecdsa-sha2-1.3.132.0.10,rsa-sha2-256,rsa-sha2-512,ssh-rsa-sha224@ssh.com,ssh-rsa-sha256@ssh.com,ssh-rsa-sha384@ssh.com,ssh-rsa-sha512@ssh.com,ssh-rsa
2023-04-17 11:01:14,188 [DEBUG] [conn=0]   Encryption algs: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr,aes128-cbc,3des-cbc,aes192-cbc,aes256-cbc
2023-04-17 11:01:14,189 [DEBUG] [conn=0]   MAC algs: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-sha256-2@ssh.com,hmac-sha224@ssh.com,hmac-sha256@ssh.com,hmac-sha384@ssh.com,hmac-sha512@ssh.com
2023-04-17 11:01:14,189 [DEBUG] [conn=0]   Compression algs: zlib@openssh.com,none
2023-04-17 11:01:14,191 [DEBUG] [conn=0, pktid=0] Sent MSG_KEXINIT (20), 2184 bytes
  00000000: 14 69 bc f7 2c 23 6c 09 3b 29 7a 6a 9e 7a ea 4e  .i..,#l.;)zj.z.N
  00000010: 98 00 00 01 c0 63 75 72 76 65 32 35 35 31 39 2d  .....curve25519-
  00000020: 73 68 61 32 35 36 2c 63 75 72 76 65 32 35 35 31  sha256,curve2551
  00000030: 39 2d 73 68 61 32 35 36 40 6c 69 62 73 73 68 2e  9-sha256@libssh.
  00000040: 6f 72 67 2c 63 75 72 76 65 34 34 38 2d 73 68 61  org,curve448-sha
  00000050: 35 31 32 2c 65 63 64 68 2d 73 68 61 32 2d 6e 69  512,ecdh-sha2-ni
  00000060: 73 74 70 35 32 31 2c 65 63 64 68 2d 73 68 61 32  stp521,ecdh-sha2
  00000070: 2d 6e 69 73 74 70 33 38 34 2c 65 63 64 68 2d 73  -nistp384,ecdh-s
  00000080: 68 61 32 2d 6e 69 73 74 70 32 35 36 2c 65 63 64  ha2-nistp256,ecd
  00000090: 68 2d 73 68 61 32 2d 31 2e 33 2e 31 33 32 2e 30  h-sha2-1.3.132.0
  000000a0: 2e 31 30 2c 64 69 66 66 69 65 2d 68 65 6c 6c 6d  .10,diffie-hellm
  000000b0: 61 6e 2d 67 72 6f 75 70 2d 65 78 63 68 61 6e 67  an-group-exchang
  000000c0: 65 2d 73 68 61 32 35 36 2c 64 69 66 66 69 65 2d  e-sha256,diffie-
  000000d0: 68 65 6c 6c 6d 61 6e 2d 67 72 6f 75 70 31 34 2d  hellman-group14-
  000000e0: 73 68 61 32 35 36 2c 64 69 66 66 69 65 2d 68 65  sha256,diffie-he
  000000f0: 6c 6c 6d 61 6e 2d 67 72 6f 75 70 31 35 2d 73 68  llman-group15-sh
  00000100: 61 35 31 32 2c 64 69 66 66 69 65 2d 68 65 6c 6c  a512,diffie-hell
  00000110: 6d 61 6e 2d 67 72 6f 75 70 31 36 2d 73 68 61 35  man-group16-sha5
  00000120: 31 32 2c 64 69 66 66 69 65 2d 68 65 6c 6c 6d 61  12,diffie-hellma
  00000130: 6e 2d 67 72 6f 75 70 31 37 2d 73 68 61 35 31 32  n-group17-sha512
  00000140: 2c 64 69 66 66 69 65 2d 68 65 6c 6c 6d 61 6e 2d  ,diffie-hellman-
  00000150: 67 72 6f 75 70 31 38 2d 73 68 61 35 31 32 2c 64  group18-sha512,d
  00000160: 69 66 66 69 65 2d 68 65 6c 6c 6d 61 6e 2d 67 72  iffie-hellman-gr
  00000170: 6f 75 70 31 34 2d 73 68 61 32 35 36 40 73 73 68  oup14-sha256@ssh
  00000180: 2e 63 6f 6d 2c 64 69 66 66 69 65 2d 68 65 6c 6c  .com,diffie-hell
  00000190: 6d 61 6e 2d 67 72 6f 75 70 31 34 2d 73 68 61 31  man-group14-sha1
  000001a0: 2c 72 73 61 32 30 34 38 2d 73 68 61 32 35 36 2c  ,rsa2048-sha256,
  000001b0: 64 69 66 66 69 65 2d 68 65 6c 6c 6d 61 6e 2d 67  diffie-hellman-g
  000001c0: 72 6f 75 70 31 2d 73 68 61 31 2c 65 78 74 2d 69  roup1-sha1,ext-i
  000001d0: 6e 66 6f 2d 63 00 00 02 be 73 6b 2d 73 73 68 2d  nfo-c....sk-ssh-
  000001e0: 65 64 32 35 35 31 39 2d 63 65 72 74 2d 76 30 31  ed25519-cert-v01
  000001f0: 40 6f 70 65 6e 73 73 68 2e 63 6f 6d 2c 73 6b 2d  @openssh.com,sk-
  00000200: 65 63 64 73 61 2d 73 68 61 32 2d 6e 69 73 74 70  ecdsa-sha2-nistp
  00000210: 32 35 36 2d 63 65 72 74 2d 76 30 31 40 6f 70 65  256-cert-v01@ope
  00000220: 6e 73 73 68 2e 63 6f 6d 2c 73 73 68 2d 65 64 32  nssh.com,ssh-ed2
  00000230: 35 35 31 39 2d 63 65 72 74 2d 76 30 31 40 6f 70  5519-cert-v01@op
  00000240: 65 6e 73 73 68 2e 63 6f 6d 2c 73 73 68 2d 65 64  enssh.com,ssh-ed
  00000250: 34 34 38 2d 63 65 72 74 2d 76 30 31 40 6f 70 65  448-cert-v01@ope
  00000260: 6e 73 73 68 2e 63 6f 6d 2c 65 63 64 73 61 2d 73  nssh.com,ecdsa-s
  00000270: 68 61 32 2d 6e 69 73 74 70 35 32 31 2d 63 65 72  ha2-nistp521-cer
  00000280: 74 2d 76 30 31 40 6f 70 65 6e 73 73 68 2e 63 6f  t-v01@openssh.co
  00000290: 6d 2c 65 63 64 73 61 2d 73 68 61 32 2d 6e 69 73  m,ecdsa-sha2-nis
  000002a0: 74 70 33 38 34 2d 63 65 72 74 2d 76 30 31 40 6f  tp384-cert-v01@o
  000002b0: 70 65 6e 73 73 68 2e 63 6f 6d 2c 65 63 64 73 61  penssh.com,ecdsa
  000002c0: 2d 73 68 61 32 2d 6e 69 73 74 70 32 35 36 2d 63  -sha2-nistp256-c
  000002d0: 65 72 74 2d 76 30 31 40 6f 70 65 6e 73 73 68 2e  ert-v01@openssh.
  000002e0: 63 6f 6d 2c 65 63 64 73 61 2d 73 68 61 32 2d 31  com,ecdsa-sha2-1
  000002f0: 2e 33 2e 31 33 32 2e 30 2e 31 30 2d 63 65 72 74  .3.132.0.10-cert
  00000300: 2d 76 30 31 40 6f 70 65 6e 73 73 68 2e 63 6f 6d  -v01@openssh.com
  00000310: 2c 72 73 61 2d 73 68 61 32 2d 32 35 36 2d 63 65  ,rsa-sha2-256-ce
  00000320: 72 74 2d 76 30 31 40 6f 70 65 6e 73 73 68 2e 63  rt-v01@openssh.c
  00000330: 6f 6d 2c 72 73 61 2d 73 68 61 32 2d 35 31 32 2d  om,rsa-sha2-512-
  00000340: 63 65 72 74 2d 76 30 31 40 6f 70 65 6e 73 73 68  cert-v01@openssh
  00000350: 2e 63 6f 6d 2c 73 73 68 2d 72 73 61 2d 63 65 72  .com,ssh-rsa-cer
  00000360: 74 2d 76 30 31 40 6f 70 65 6e 73 73 68 2e 63 6f  t-v01@openssh.co
  00000370: 6d 2c 73 6b 2d 73 73 68 2d 65 64 32 35 35 31 39  m,sk-ssh-ed25519
  00000380: 40 6f 70 65 6e 73 73 68 2e 63 6f 6d 2c 73 6b 2d  @openssh.com,sk-
  00000390: 65 63 64 73 61 2d 73 68 61 32 2d 6e 69 73 74 70  ecdsa-sha2-nistp
  000003a0: 32 35 36 40 6f 70 65 6e 73 73 68 2e 63 6f 6d 2c  256@openssh.com,
  000003b0: 73 73 68 2d 65 64 32 35 35 31 39 2c 73 73 68 2d  ssh-ed25519,ssh-
  000003c0: 65 64 34 34 38 2c 65 63 64 73 61 2d 73 68 61 32  ed448,ecdsa-sha2
  000003d0: 2d 6e 69 73 74 70 35 32 31 2c 65 63 64 73 61 2d  -nistp521,ecdsa-
  000003e0: 73 68 61 32 2d 6e 69 73 74 70 33 38 34 2c 65 63  sha2-nistp384,ec
  000003f0: 64 73 61 2d 73 68 61 32 2d 6e 69 73 74 70 32 35  dsa-sha2-nistp25
  00000400: 36 2c 65 63 64 73 61 2d 73 68 61 32 2d 31 2e 33  6,ecdsa-sha2-1.3
  00000410: 2e 31 33 32 2e 30 2e 31 30 2c 72 73 61 2d 73 68  .132.0.10,rsa-sh
  00000420: 61 32 2d 32 35 36 2c 72 73 61 2d 73 68 61 32 2d  a2-256,rsa-sha2-
  00000430: 35 31 32 2c 73 73 68 2d 72 73 61 2d 73 68 61 32  512,ssh-rsa-sha2
  00000440: 32 34 40 73 73 68 2e 63 6f 6d 2c 73 73 68 2d 72  24@ssh.com,ssh-r
  00000450: 73 61 2d 73 68 61 32 35 36 40 73 73 68 2e 63 6f  sa-sha256@ssh.co
  00000460: 6d 2c 73 73 68 2d 72 73 61 2d 73 68 61 33 38 34  m,ssh-rsa-sha384
  00000470: 40 73 73 68 2e 63 6f 6d 2c 73 73 68 2d 72 73 61  @ssh.com,ssh-rsa
  00000480: 2d 73 68 61 35 31 32 40 73 73 68 2e 63 6f 6d 2c  -sha512@ssh.com,
  00000490: 73 73 68 2d 72 73 61 00 00 00 96 63 68 61 63 68  ssh-rsa....chach
  000004a0: 61 32 30 2d 70 6f 6c 79 31 33 30 35 40 6f 70 65  a20-poly1305@ope
  000004b0: 6e 73 73 68 2e 63 6f 6d 2c 61 65 73 32 35 36 2d  nssh.com,aes256-
  000004c0: 67 63 6d 40 6f 70 65 6e 73 73 68 2e 63 6f 6d 2c  gcm@openssh.com,
  000004d0: 61 65 73 31 32 38 2d 67 63 6d 40 6f 70 65 6e 73  aes128-gcm@opens
  000004e0: 73 68 2e 63 6f 6d 2c 61 65 73 32 35 36 2d 63 74  sh.com,aes256-ct
  000004f0: 72 2c 61 65 73 31 39 32 2d 63 74 72 2c 61 65 73  r,aes192-ctr,aes
  00000500: 31 32 38 2d 63 74 72 2c 61 65 73 31 32 38 2d 63  128-ctr,aes128-c
  00000510: 62 63 2c 33 64 65 73 2d 63 62 63 2c 61 65 73 31  bc,3des-cbc,aes1
  00000520: 39 32 2d 63 62 63 2c 61 65 73 32 35 36 2d 63 62  92-cbc,aes256-cb
  00000530: 63 00 00 00 96 63 68 61 63 68 61 32 30 2d 70 6f  c....chacha20-po
  00000540: 6c 79 31 33 30 35 40 6f 70 65 6e 73 73 68 2e 63  ly1305@openssh.c
  00000550: 6f 6d 2c 61 65 73 32 35 36 2d 67 63 6d 40 6f 70  om,aes256-gcm@op
  00000560: 65 6e 73 73 68 2e 63 6f 6d 2c 61 65 73 31 32 38  enssh.com,aes128
  00000570: 2d 67 63 6d 40 6f 70 65 6e 73 73 68 2e 63 6f 6d  -gcm@openssh.com
  00000580: 2c 61 65 73 32 35 36 2d 63 74 72 2c 61 65 73 31  ,aes256-ctr,aes1
  00000590: 39 32 2d 63 74 72 2c 61 65 73 31 32 38 2d 63 74  92-ctr,aes128-ct
  000005a0: 72 2c 61 65 73 31 32 38 2d 63 62 63 2c 33 64 65  r,aes128-cbc,3de
  000005b0: 73 2d 63 62 63 2c 61 65 73 31 39 32 2d 63 62 63  s-cbc,aes192-cbc
  000005c0: 2c 61 65 73 32 35 36 2d 63 62 63 00 00 01 3b 75  ,aes256-cbc...;u
  000005d0: 6d 61 63 2d 36 34 2d 65 74 6d 40 6f 70 65 6e 73  mac-64-etm@opens
  000005e0: 73 68 2e 63 6f 6d 2c 75 6d 61 63 2d 31 32 38 2d  sh.com,umac-128-
  000005f0: 65 74 6d 40 6f 70 65 6e 73 73 68 2e 63 6f 6d 2c  etm@openssh.com,
  00000600: 68 6d 61 63 2d 73 68 61 32 2d 32 35 36 2d 65 74  hmac-sha2-256-et
  00000610: 6d 40 6f 70 65 6e 73 73 68 2e 63 6f 6d 2c 68 6d  m@openssh.com,hm
  00000620: 61 63 2d 73 68 61 32 2d 35 31 32 2d 65 74 6d 40  ac-sha2-512-etm@
  00000630: 6f 70 65 6e 73 73 68 2e 63 6f 6d 2c 68 6d 61 63  openssh.com,hmac
  00000640: 2d 73 68 61 31 2d 65 74 6d 40 6f 70 65 6e 73 73  -sha1-etm@openss
  00000650: 68 2e 63 6f 6d 2c 75 6d 61 63 2d 36 34 40 6f 70  h.com,umac-64@op
  00000660: 65 6e 73 73 68 2e 63 6f 6d 2c 75 6d 61 63 2d 31  enssh.com,umac-1
  00000670: 32 38 40 6f 70 65 6e 73 73 68 2e 63 6f 6d 2c 68  28@openssh.com,h
  00000680: 6d 61 63 2d 73 68 61 32 2d 32 35 36 2c 68 6d 61  mac-sha2-256,hma
  00000690: 63 2d 73 68 61 32 2d 35 31 32 2c 68 6d 61 63 2d  c-sha2-512,hmac-
  000006a0: 73 68 61 31 2c 68 6d 61 63 2d 73 68 61 32 35 36  sha1,hmac-sha256
  000006b0: 2d 32 40 73 73 68 2e 63 6f 6d 2c 68 6d 61 63 2d  -2@ssh.com,hmac-
  000006c0: 73 68 61 32 32 34 40 73 73 68 2e 63 6f 6d 2c 68  sha224@ssh.com,h
  000006d0: 6d 61 63 2d 73 68 61 32 35 36 40 73 73 68 2e 63  mac-sha256@ssh.c
  000006e0: 6f 6d 2c 68 6d 61 63 2d 73 68 61 33 38 34 40 73  om,hmac-sha384@s
  000006f0: 73 68 2e 63 6f 6d 2c 68 6d 61 63 2d 73 68 61 35  sh.com,hmac-sha5
  00000700: 31 32 40 73 73 68 2e 63 6f 6d 00 00 01 3b 75 6d  12@ssh.com...;um
  00000710: 61 63 2d 36 34 2d 65 74 6d 40 6f 70 65 6e 73 73  ac-64-etm@openss
  00000720: 68 2e 63 6f 6d 2c 75 6d 61 63 2d 31 32 38 2d 65  h.com,umac-128-e
  00000730: 74 6d 40 6f 70 65 6e 73 73 68 2e 63 6f 6d 2c 68  tm@openssh.com,h
  00000740: 6d 61 63 2d 73 68 61 32 2d 32 35 36 2d 65 74 6d  mac-sha2-256-etm
  00000750: 40 6f 70 65 6e 73 73 68 2e 63 6f 6d 2c 68 6d 61  @openssh.com,hma
  00000760: 63 2d 73 68 61 32 2d 35 31 32 2d 65 74 6d 40 6f  c-sha2-512-etm@o
  00000770: 70 65 6e 73 73 68 2e 63 6f 6d 2c 68 6d 61 63 2d  penssh.com,hmac-
  00000780: 73 68 61 31 2d 65 74 6d 40 6f 70 65 6e 73 73 68  sha1-etm@openssh
  00000790: 2e 63 6f 6d 2c 75 6d 61 63 2d 36 34 40 6f 70 65  .com,umac-64@ope
  000007a0: 6e 73 73 68 2e 63 6f 6d 2c 75 6d 61 63 2d 31 32  nssh.com,umac-12
  000007b0: 38 40 6f 70 65 6e 73 73 68 2e 63 6f 6d 2c 68 6d  8@openssh.com,hm
  000007c0: 61 63 2d 73 68 61 32 2d 32 35 36 2c 68 6d 61 63  ac-sha2-256,hmac
  000007d0: 2d 73 68 61 32 2d 35 31 32 2c 68 6d 61 63 2d 73  -sha2-512,hmac-s
  000007e0: 68 61 31 2c 68 6d 61 63 2d 73 68 61 32 35 36 2d  ha1,hmac-sha256-
  000007f0: 32 40 73 73 68 2e 63 6f 6d 2c 68 6d 61 63 2d 73  2@ssh.com,hmac-s
  00000800: 68 61 32 32 34 40 73 73 68 2e 63 6f 6d 2c 68 6d  ha224@ssh.com,hm
  00000810: 61 63 2d 73 68 61 32 35 36 40 73 73 68 2e 63 6f  ac-sha256@ssh.co
  00000820: 6d 2c 68 6d 61 63 2d 73 68 61 33 38 34 40 73 73  m,hmac-sha384@ss
  00000830: 68 2e 63 6f 6d 2c 68 6d 61 63 2d 73 68 61 35 31  h.com,hmac-sha51
  00000840: 32 40 73 73 68 2e 63 6f 6d 00 00 00 15 7a 6c 69  2@ssh.com....zli
  00000850: 62 40 6f 70 65 6e 73 73 68 2e 63 6f 6d 2c 6e 6f  b@openssh.com,no
  00000860: 6e 65 00 00 00 15 7a 6c 69 62 40 6f 70 65 6e 73  ne....zlib@opens
  00000870: 73 68 2e 63 6f 6d 2c 6e 6f 6e 65 00 00 00 00 00  sh.com,none.....
  00000880: 00 00 00 00 00 00 00 00                          ........
2023-04-17 11:01:14,192 [DEBUG] [conn=0, pktid=0] Received MSG_KEXINIT (20), 271 bytes
  00000000: 14 df 3f 03 30 5a 29 56 c7 d7 de 2e c0 52 83 6f  ..?.0Z)V.....R.o
  00000010: 90 00 00 00 1a 64 69 66 66 69 65 2d 68 65 6c 6c  .....diffie-hell
  00000020: 6d 61 6e 2d 67 72 6f 75 70 31 2d 73 68 61 31 00  man-group1-sha1.
  00000030: 00 00 07 73 73 68 2d 72 73 61 00 00 00 29 61 65  ...ssh-rsa...)ae
  00000040: 73 31 32 38 2d 63 62 63 2c 33 64 65 73 2d 63 62  s128-cbc,3des-cb
  00000050: 63 2c 61 65 73 31 39 32 2d 63 62 63 2c 61 65 73  c,aes192-cbc,aes
  00000060: 32 35 36 2d 63 62 63 00 00 00 29 61 65 73 31 32  256-cbc...)aes12
  00000070: 38 2d 63 62 63 2c 33 64 65 73 2d 63 62 63 2c 61  8-cbc,3des-cbc,a
  00000080: 65 73 31 39 32 2d 63 62 63 2c 61 65 73 32 35 36  es192-cbc,aes256
  00000090: 2d 63 62 63 00 00 00 2b 68 6d 61 63 2d 73 68 61  -cbc...+hmac-sha
  000000a0: 31 2c 68 6d 61 63 2d 73 68 61 31 2d 39 36 2c 68  1,hmac-sha1-96,h
  000000b0: 6d 61 63 2d 6d 64 35 2c 68 6d 61 63 2d 6d 64 35  mac-md5,hmac-md5
  000000c0: 2d 39 36 00 00 00 2b 68 6d 61 63 2d 73 68 61 31  -96...+hmac-sha1
  000000d0: 2c 68 6d 61 63 2d 73 68 61 31 2d 39 36 2c 68 6d  ,hmac-sha1-96,hm
  000000e0: 61 63 2d 6d 64 35 2c 68 6d 61 63 2d 6d 64 35 2d  ac-md5,hmac-md5-
  000000f0: 39 36 00 00 00 04 6e 6f 6e 65 00 00 00 04 6e 6f  96....none....no
  00000100: 6e 65 00 00 00 00 00 00 00 00 00 00 00 00 00     ne.............
2023-04-17 11:01:14,192 [DEBUG] [conn=0] Received key exchange request
2023-04-17 11:01:14,192 [DEBUG] [conn=0]   Key exchange algs: diffie-hellman-group1-sha1
2023-04-17 11:01:14,192 [DEBUG] [conn=0]   Host key algs: ssh-rsa
2023-04-17 11:01:14,192 [DEBUG] [conn=0]   Client to server:
2023-04-17 11:01:14,192 [DEBUG] [conn=0]     Encryption algs: aes128-cbc,3des-cbc,aes192-cbc,aes256-cbc
2023-04-17 11:01:14,192 [DEBUG] [conn=0]     MAC algs: hmac-sha1,hmac-sha1-96,hmac-md5,hmac-md5-96
2023-04-17 11:01:14,192 [DEBUG] [conn=0]     Compression algs: none
2023-04-17 11:01:14,192 [DEBUG] [conn=0]   Server to client:
2023-04-17 11:01:14,192 [DEBUG] [conn=0]     Encryption algs: aes128-cbc,3des-cbc,aes192-cbc,aes256-cbc
2023-04-17 11:01:14,192 [DEBUG] [conn=0]     MAC algs: hmac-sha1,hmac-sha1-96,hmac-md5,hmac-md5-96
2023-04-17 11:01:14,192 [DEBUG] [conn=0]     Compression algs: none
2023-04-17 11:01:14,192 [DEBUG] [conn=0] Beginning key exchange
2023-04-17 11:01:14,193 [DEBUG] [conn=0]   Key exchange alg: diffie-hellman-group1-sha1
2023-04-17 11:01:14,197 [DEBUG] [conn=0, pktid=1] Sent MSG_KEXDH_INIT (30), 134 bytes
  00000000: 1e 00 00 00 81 00 cd 8b 86 c5 b4 76 a5 68 aa 4c  ...........v.h.L
  00000010: a0 46 51 1e 3b 1e 10 1f ec 8b 56 b2 ee 82 43 a0  .FQ.;.....V...C.
  00000020: c0 19 47 94 8c c7 62 ac a0 25 e0 21 85 ea 10 d1  ..G...b..%.!....
  00000030: ad e9 c8 4b 8c a9 dd 08 80 b5 22 90 cb 00 42 c6  ...K......"...B.
  00000040: d9 36 a9 2f fa 18 e2 db 0d 24 81 66 d3 a3 18 c7  .6./.....$.f....
  00000050: d2 67 15 59 13 46 8c c4 e6 cd 71 b3 2c e0 b5 9e  .g.Y.F....q.,...
  00000060: a4 ae 59 37 98 24 69 df 97 ea 7f ff a5 84 d8 3c  ..Y7.$i.......<
  00000070: ab 10 53 b8 91 8e e6 74 f0 11 17 9e 06 11 e0 08  ..S....t........
  00000080: a8 b9 8b ea 28 3e                                ....(>
2023-04-17 11:01:14,663 [DEBUG] [conn=0, pktid=1] Received MSG_KEXDH_REPLY (31), 435 bytes
  <HEX_HEXDH_REPLY>
2023-04-17 11:01:14,666 [DEBUG] [conn=0]   Client to server:
2023-04-17 11:01:14,666 [DEBUG] [conn=0]     Encryption alg: aes128-cbc
2023-04-17 11:01:14,666 [DEBUG] [conn=0]     MAC alg: hmac-sha1
2023-04-17 11:01:14,666 [DEBUG] [conn=0]     Compression alg: none
2023-04-17 11:01:14,666 [DEBUG] [conn=0]   Server to client:
2023-04-17 11:01:14,666 [DEBUG] [conn=0]     Encryption alg: aes128-cbc
2023-04-17 11:01:14,666 [DEBUG] [conn=0]     MAC alg: hmac-sha1
2023-04-17 11:01:14,666 [DEBUG] [conn=0]     Compression alg: none
2023-04-17 11:01:14,667 [DEBUG] [conn=0, pktid=2] Sent MSG_NEWKEYS (21), 1 byte
  00000000: 15                                               .
2023-04-17 11:01:14,667 [DEBUG] [conn=0] Requesting service ssh-userauth
2023-04-17 11:01:14,667 [DEBUG] [conn=0, pktid=3] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
2023-04-17 11:01:14,668 [DEBUG] [conn=0, pktid=4] Sent MSG_SERVICE_REQUEST (5), 17 bytes
  00000000: 05 00 00 00 0c 73 73 68 2d 75 73 65 72 61 75 74  .....ssh-useraut
  00000010: 68                                               h
2023-04-17 11:01:14,668 [DEBUG] [conn=0, pktid=2] Received MSG_NEWKEYS (21), 1 byte
  00000000: 15                                               .
2023-04-17 11:01:14,668 [DEBUG] [conn=0] Completed key exchange
2023-04-17 11:01:14,898 [DEBUG] [conn=0, pktid=3] Received MSG_SERVICE_ACCEPT (6), 17 bytes
  00000000: 06 00 00 00 0c 73 73 68 2d 75 73 65 72 61 75 74  .....ssh-useraut
  00000010: 68                                               h
2023-04-17 11:01:14,898 [DEBUG] [conn=0] Request for service ssh-userauth accepted
2023-04-17 11:01:14,899 [INFO] [conn=0] Beginning auth for user <AD_USERNAME>
2023-04-17 11:01:14,899 [DEBUG] [conn=0, pktid=5] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
2023-04-17 11:01:14,899 [DEBUG] [conn=0, pktid=6] Sent MSG_USERAUTH_REQUEST (50), 39 bytes
  00000000: <HEX_STRING_WITH_AD_USERNAME>  2....<AD_USERNAME>...
  00000010: 0e 73 73 68 2d 63 6f 6e 6e 65 63 74 69 6f 6e 00  .ssh-connection.
  00000020: 00 00 04 6e 6f 6e 65                             ...none
2023-04-17 11:01:15,129 [DEBUG] [conn=0, pktid=4] Received MSG_USERAUTH_BANNER (53), 1064 bytes
  00000000: 35 00 00 04 1f 0d 0a 2d 2d 2d 2d 2d 2d 2d 2d 2d  5......---------
  00000010: 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d  ----------------
  00000020: 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d  ----------------
  00000030: 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d  ----------------
  00000040: 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d  ----------------
  00000050: 2d 2d 2d 0d 0a 2d 2d 2d 20 20 20 55 4e 41 55 54  ---..---   UNAUT
  00000060: 48 4f 52 49 5a 45 44 20 55 53 45 20 4d 41 59 20  HORIZED USE MAY 
  00000070: 52 45 53 55 4c 54 20 49 4e 20 43 49 56 49 4c 20  RESULT IN CIVIL 
  00000080: 41 4e 44 2f 4f 52 20 43 52 49 4d 49 4e 41 4c 20  AND/OR CRIMINAL 
  00000090: 50 52 4f 53 45 43 55 54 49 4f 4e 20 20 20 2d 2d  PROSECUTION   --
  000000a0: 2d 0d 0a 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d  -..-------------
  000000b0: 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d  ----------------
  000000c0: 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d  ----------------
  000000d0: 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d  ----------------
  000000e0: 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 0d  ---------------.
  000000f0: 0a 0d 0a 20 20 20 20 54 68 69 73 20 69 73 20 61  ...    This is a
  00000100: 20 70 72 69 76 61 74 65 20 63 6f 6d 70 75 74 69   private computi
  00000110: 6e 67 20 73 79 73 74 65 6d 20 66 6f 72 20 75 73  ng system for us
  00000120: 65 20 6f 6e 6c 79 20 62 79 20 61 75 74 68 6f 72  e only by author
  00000130: 69 7a 65 64 20 75 73 65 72 73 2e 0d 0a 20 20 20  ized users...   
  00000140: 20 20 20 20 20 20 49 66 20 79 6f 75 20 64 6f 20        If you do 
  00000150: 6e 6f 74 20 68 61 76 65 20 61 75 74 68 6f 72 69  not have authori
  00000160: 7a 61 74 69 6f 6e 2c 20 64 69 73 63 6f 6e 74 69  zation, disconti
  00000170: 6e 75 65 20 61 74 20 6f 6e 63 65 2e 0d 0a 0d 0a  nue at once.....
  00000180: 20 59 6f 75 72 20 75 73 65 20 6f 66 20 74 68 69   Your use of thi
  00000190: 73 20 73 79 73 74 65 6d 2c 20 61 6e 64 20 61 6c  s system, and al
  000001a0: 6c 20 69 6e 66 6f 72 6d 61 74 69 6f 6e 20 6f 72  l information or
  000001b0: 20 63 6f 6d 6d 75 6e 69 63 61 74 69 6f 6e 73 20   communications 
  000001c0: 73 74 6f 72 65 64 20 6f 72 0d 0a 20 74 72 61 6e  stored or.. tran
  000001d0: 73 6d 69 74 74 65 64 20 6f 6e 20 69 74 20 62 79  smitted on it by
  000001e0: 20 79 6f 75 2c 20 77 69 74 68 20 6f 72 20 77 69   you, with or wi
  000001f0: 74 68 6f 75 74 20 61 75 74 68 6f 72 69 74 79 2c  thout authority,
  00000200: 20 6f 72 20 69 6e 20 65 78 63 65 73 73 20 6f 66   or in excess of
  00000210: 20 79 6f 75 72 0d 0a 20 61 75 74 68 6f 72 69 74   your.. authorit
  00000220: 79 2c 20 69 73 20 73 75 62 6a 65 63 74 20 74 6f  y, is subject to
  00000230: 20 72 65 76 69 65 77 2c 20 6d 6f 6e 69 74 6f 72   review, monitor
  00000240: 69 6e 67 2c 20 61 6e 64 20 72 65 63 6f 72 64 69  ing, and recordi
  00000250: 6e 67 20 62 79 20 73 79 73 74 65 6d 0d 0a 20 70  ng by system.. p
  00000260: 65 72 73 6f 6e 6e 65 6c 2c 20 77 69 74 68 6f 75  ersonnel, withou
  00000270: 74 20 6e 6f 74 69 63 65 20 6f 72 20 70 65 72 6d  t notice or perm
  00000280: 69 73 73 69 6f 6e 2e 20 41 6e 79 20 69 6e 66 6f  ission. Any info
  00000290: 72 6d 61 74 69 6f 6e 20 6f 72 20 64 61 74 61 20  rmation or data 
  000002a0: 74 68 75 73 0d 0a 20 6f 62 74 61 69 6e 65 64 20  thus.. obtained 
  000002b0: 62 79 20 73 79 73 74 65 6d 20 70 65 72 73 6f 6e  by system person
  000002c0: 6e 65 6c 20 61 6c 73 6f 20 69 73 20 73 75 62 6a  nel also is subj
  000002d0: 65 63 74 20 74 6f 20 72 65 76 69 65 77 20 62 79  ect to review by
  000002e0: 20 6c 61 77 20 65 6e 66 6f 72 63 65 6d 65 6e 74   law enforcement
  000002f0: 0d 0a 20 61 6e 64 20 6a 75 64 69 63 69 61 6c 20  .. and judicial 
  00000300: 6f 66 66 69 63 69 61 6c 73 20 69 6e 20 63 6f 6e  officials in con
  00000310: 6e 65 63 74 69 6f 6e 20 77 69 74 68 20 74 68 65  nection with the
  00000320: 20 69 6e 76 65 73 74 69 67 61 74 69 6f 6e 20 6f   investigation o
  00000330: 72 20 70 72 6f 73 65 63 75 74 69 6f 6e 0d 0a 20  r prosecution.. 
  00000340: 6f 66 20 70 6f 73 73 69 62 6c 65 20 63 72 69 6d  of possible crim
  00000350: 69 6e 61 6c 20 61 63 74 69 76 69 74 79 20 69 6e  inal activity in
  00000360: 76 6f 6c 76 69 6e 67 20 74 68 65 20 73 79 73 74  volving the syst
  00000370: 65 6d 2e 0d 0a 0d 0a 20 49 66 20 79 6f 75 20 6e  em..... If you n
  00000380: 6f 77 20 63 6f 6e 74 69 6e 75 65 20 75 73 65 20  ow continue use 
  00000390: 6f 66 20 74 68 65 20 73 79 73 74 65 6d 2c 20 79  of the system, y
  000003a0: 6f 75 20 65 78 70 72 65 73 73 6c 79 20 63 6f 6e  ou expressly con
  000003b0: 73 65 6e 74 20 74 6f 20 74 68 65 73 65 0d 0a 20  sent to these.. 
  000003c0: 63 6f 6e 64 69 74 69 6f 6e 73 20 6f 66 20 75 73  conditions of us
  000003d0: 65 2e 0d 0a 0d 0a 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d  e.....----------
  000003e0: 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d  ----------------
  000003f0: 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d  ----------------
  00000400: 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d  ----------------
  00000410: 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d  ----------------
  00000420: 2d 2d 0d 0a 00 00 00 00                          --......
2023-04-17 11:01:15,129 [DEBUG] [conn=0] Received authentication banner
2023-04-17 11:01:15,130 [DEBUG] [conn=0, pktid=5] Received MSG_USERAUTH_FAILURE (51), 14 bytes
  00000000: 33 00 00 00 08 70 61 73 73 77 6f 72 64 00        3....password.
2023-04-17 11:01:15,130 [DEBUG] [conn=0] Remaining auth methods: password
2023-04-17 11:01:15,130 [DEBUG] [conn=0] Preferred auth methods: publickey,keyboard-interactive,password
2023-04-17 11:01:15,130 [DEBUG] [conn=0] Trying password auth
2023-04-17 11:01:15,130 [DEBUG] [conn=0, pktid=7] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
2023-04-17 11:01:15,131 [DEBUG] [conn=0, pktid=8] Sent MSG_USERAUTH_REQUEST (50), 56 bytes
  00000000: <HEX_STRING_WITH_AD_USERNAME>  2....<AD_USERNAME>...
  00000010: 0e 73 73 68 2d 63 6f 6e 6e 65 63 74 69 6f 6e 00  .ssh-connection.
  00000020: 00 00 08 70 61 73 73 77 6f 72 64 00 00 00 00 08  ...password.....
  00000030: <HEX_STRING_WITH_AD_PASSWORD>                          <AD_PASSWORD>
2023-04-17 11:01:16,487 [DEBUG] [conn=0, pktid=6] Received MSG_USERAUTH_SUCCESS (52), 1 byte
  00000000: 34                                               4
2023-04-17 11:01:16,487 [INFO] [conn=0] Auth for user <AD_USERNAME> succeeded
2023-04-17 11:01:16,488 [DEBUG] [conn=0, chan=0] Set write buffer limits: low-water=16384, high-water=65536
2023-04-17 11:01:16,488 [INFO] [conn=0, chan=0] Requesting new SSH session
2023-04-17 11:01:16,489 [DEBUG] [conn=0, chan=0]   Initial recv window 2097152, packet size 32768
2023-04-17 11:01:16,489 [DEBUG] [conn=0, pktid=9] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
2023-04-17 11:01:16,489 [DEBUG] [conn=0, chan=0, pktid=10] Sent MSG_CHANNEL_OPEN (90), 24 bytes
  00000000: 5a 00 00 00 07 73 65 73 73 69 6f 6e 00 00 00 00  Z....session....
  00000010: 00 20 00 00 00 00 80 00                          . ......
2023-04-17 11:01:16,719 [DEBUG] [conn=0, pktid=7] Received MSG_CHANNEL_OPEN_CONFIRMATION (91), 17 bytes
  00000000: 5b 00 00 00 00 00 00 00 03 00 00 04 00 00 00 10  [...............
  00000010: 00                                               .
2023-04-17 11:01:16,719 [DEBUG] [conn=0, chan=0]   Initial send window 1024, packet size 4096
2023-04-17 11:01:16,720 [DEBUG] [conn=0, chan=0]   Env: LANG=en_US.UTF-8
2023-04-17 11:01:16,720 [DEBUG] [conn=0, pktid=11] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
2023-04-17 11:01:16,720 [DEBUG] [conn=0, chan=0, pktid=12] Sent MSG_CHANNEL_REQUEST (98), 36 bytes
  00000000: 62 00 00 00 03 00 00 00 03 65 6e 76 00 00 00 00  b........env....
  00000010: 04 4c 41 4e 47 00 00 00 0b 65 6e 5f 55 53 2e 55  .LANG....en_US.U
  00000020: 54 46 2d 38                                      TF-8
2023-04-17 11:01:16,720 [DEBUG] [conn=0, chan=0]   Terminal type: xterm
2023-04-17 11:01:16,720 [DEBUG] [conn=0, pktid=13] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
2023-04-17 11:01:16,721 [DEBUG] [conn=0, chan=0, pktid=14] Sent MSG_CHANNEL_REQUEST (98), 47 bytes
  00000000: 62 00 00 00 03 00 00 00 07 70 74 79 2d 72 65 71  b........pty-req
  00000010: 01 00 00 00 05 78 74 65 72 6d 00 00 00 00 00 00  .....xterm......
  00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00     ...............
2023-04-17 11:01:16,951 [DEBUG] [conn=0, chan=0, pktid=8] Received MSG_CHANNEL_SUCCESS (99), 5 bytes
  00000000: 63 00 00 00 00                                   c....
2023-04-17 11:01:16,951 [INFO] [conn=0, chan=0]   Interactive shell requested
2023-04-17 11:01:16,952 [DEBUG] [conn=0, pktid=15] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
2023-04-17 11:01:16,952 [DEBUG] [conn=0, chan=0, pktid=16] Sent MSG_CHANNEL_REQUEST (98), 15 bytes
  00000000: 62 00 00 00 03 00 00 00 05 73 68 65 6c 6c 01     b........shell.
2023-04-17 11:01:17,182 [DEBUG] [conn=0, chan=0, pktid=9] Received MSG_CHANNEL_SUCCESS (99), 5 bytes
  00000000: 63 00 00 00 00                                   c....
2023-04-17 11:01:17,182 [DEBUG] transport connection to '<DEVICE_NAME>' on port '22' opened successfully
2023-04-17 11:01:17,182 [DEBUG] [conn=0, chan=0] Reading from channel started
2023-04-17 11:01:17,183 [DEBUG] write: '\n'
2023-04-17 11:01:17,184 [DEBUG] [conn=0, chan=0] Sending 1 data byte
2023-04-17 11:01:17,184 [DEBUG] [conn=0, pktid=17] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
2023-04-17 11:01:17,184 [DEBUG] [conn=0, chan=0, pktid=18] Sent MSG_CHANNEL_DATA (94), 10 bytes
  00000000: 5e 00 00 00 03 00 00 00 01 0a                    ^.........
2023-04-17 11:01:17,184 [DEBUG] [conn=0, chan=0, pktid=10] Received MSG_CHANNEL_DATA (94), 23 bytes
  00000000: <HEX_STRING_WITH_DEVICE_NAME>  ^..........<DEVICE_NAME>
2023-04-17 11:01:17,185 [DEBUG] [conn=0, chan=0] Received 14 data bytes
2023-04-17 11:01:17,185 [DEBUG] read: b'\n<DEVICE_NAME>>'
2023-04-17 11:01:17,185 [INFO] attempting to acquire 'privilege_exec' privilege level
2023-04-17 11:01:17,186 [DEBUG] determined current privilege level is one of '['exec']'
2023-04-17 11:01:17,186 [DEBUG] determined privilege escalation necessary
2023-04-17 11:01:17,186 [INFO] sending interactive input: enable; expecting: ^(?:enable\s){0,1}password:\s?$; hidden_input: False
2023-04-17 11:01:17,186 [DEBUG] write: 'enable'
2023-04-17 11:01:17,186 [DEBUG] [conn=0, chan=0] Sending 6 data bytes
2023-04-17 11:01:17,187 [DEBUG] [conn=0, pktid=19] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
2023-04-17 11:01:17,187 [DEBUG] [conn=0, chan=0, pktid=20] Sent MSG_CHANNEL_DATA (94), 15 bytes
  00000000: 5e 00 00 00 03 00 00 00 06 65 6e 61 62 6c 65     ^........enable
2023-04-17 11:01:17,414 [DEBUG] [conn=0, chan=0, pktid=11] Received MSG_CHANNEL_DATA (94), 11 bytes
  00000000: 5e 00 00 00 00 00 00 00 02 0d 0a                 ^..........
2023-04-17 11:01:17,415 [DEBUG] [conn=0, chan=0] Received 2 data bytes
2023-04-17 11:01:17,415 [DEBUG] [conn=0, chan=0, pktid=12] Received MSG_CHANNEL_DATA (94), 21 bytes
  00000000: <HEX_STRING_WITH_DEVICE_NAME>  ^........<DEVICE_NAME>
2023-04-17 11:01:17,415 [DEBUG] [conn=0, chan=0] Received 12 data bytes
2023-04-17 11:01:17,415 [DEBUG] read: b'\n'
2023-04-17 11:01:17,415 [DEBUG] read: b'<DEVICE_NAME>>'
2023-04-17 11:01:17,417 [DEBUG] [conn=0, chan=0, pktid=13] Received MSG_CHANNEL_DATA (94), 10 bytes
  00000000: 5e 00 00 00 00 00 00 00 01 65                    ^........e
2023-04-17 11:01:17,417 [DEBUG] [conn=0, chan=0] Received 1 data byte
2023-04-17 11:01:17,418 [DEBUG] read: b'e'
2023-04-17 11:01:17,418 [DEBUG] [conn=0, chan=0, pktid=14] Received MSG_CHANNEL_DATA (94), 10 bytes
  00000000: 5e 00 00 00 00 00 00 00 01 6e                    ^........n
2023-04-17 11:01:17,418 [DEBUG] [conn=0, chan=0] Received 1 data byte
2023-04-17 11:01:17,418 [DEBUG] [conn=0, chan=0, pktid=15] Received MSG_CHANNEL_DATA (94), 10 bytes
  00000000: 5e 00 00 00 00 00 00 00 01 61                    ^........a
2023-04-17 11:01:17,419 [DEBUG] [conn=0, chan=0] Received 1 data byte
2023-04-17 11:01:17,419 [DEBUG] read: b'n'
2023-04-17 11:01:17,419 [DEBUG] read: b'a'
2023-04-17 11:01:17,419 [DEBUG] [conn=0, chan=0, pktid=16] Received MSG_CHANNEL_DATA (94), 10 bytes
  00000000: 5e 00 00 00 00 00 00 00 01 62                    ^........b
2023-04-17 11:01:17,419 [DEBUG] [conn=0, chan=0] Received 1 data byte
2023-04-17 11:01:17,419 [DEBUG] read: b'b'
2023-04-17 11:01:17,420 [DEBUG] [conn=0, chan=0, pktid=17] Received MSG_CHANNEL_DATA (94), 10 bytes
  00000000: 5e 00 00 00 00 00 00 00 01 6c                    ^........l
2023-04-17 11:01:17,420 [DEBUG] [conn=0, chan=0] Received 1 data byte
2023-04-17 11:01:17,420 [DEBUG] read: b'l'
2023-04-17 11:01:17,420 [DEBUG] [conn=0, chan=0, pktid=18] Received MSG_CHANNEL_DATA (94), 10 bytes
  00000000: 5e 00 00 00 00 00 00 00 01 65                    ^........e
2023-04-17 11:01:17,421 [DEBUG] [conn=0, chan=0] Received 1 data byte
2023-04-17 11:01:17,421 [DEBUG] read: b'e'
2023-04-17 11:01:17,421 [DEBUG] write: '\n'
2023-04-17 11:01:17,421 [DEBUG] [conn=0, chan=0] Sending 1 data byte
2023-04-17 11:01:17,421 [DEBUG] [conn=0, pktid=21] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
2023-04-17 11:01:17,421 [DEBUG] [conn=0, chan=0, pktid=22] Sent MSG_CHANNEL_DATA (94), 10 bytes
  00000000: 5e 00 00 00 03 00 00 00 01 0a                    ^.........
2023-04-17 11:01:17,652 [DEBUG] [conn=0, chan=0, pktid=19] Received MSG_CHANNEL_DATA (94), 11 bytes
  00000000: 5e 00 00 00 00 00 00 00 02 0d 0a                 ^..........
2023-04-17 11:01:17,652 [DEBUG] [conn=0, chan=0] Received 2 data bytes
2023-04-17 11:01:17,652 [DEBUG] read: b'\n'
2023-04-17 11:01:17,653 [DEBUG] [conn=0, chan=0, pktid=20] Received MSG_CHANNEL_DATA (94), 19 bytes
  00000000: 5e 00 00 00 00 00 00 00 0a 50 61 73 73 77 6f 72  ^........Passwor
  00000010: 64 3a 20                                         d: 
2023-04-17 11:01:17,653 [DEBUG] [conn=0, chan=0] Received 10 data bytes
2023-04-17 11:01:17,653 [DEBUG] read: b'Password: '
2023-04-17 11:01:17,653 [INFO] sending interactive input: REDACTED; expecting: ^[\w.\-@/:]{1,63}#$; hidden_input: True
2023-04-17 11:01:17,653 [DEBUG] write: REDACTED
2023-04-17 11:01:17,653 [DEBUG] [conn=0, chan=0] Sending 8 data bytes
2023-04-17 11:01:17,654 [DEBUG] [conn=0, pktid=23] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
2023-04-17 11:01:17,654 [DEBUG] [conn=0, chan=0, pktid=24] Sent MSG_CHANNEL_DATA (94), 17 bytes
  00000000: <HEX_STRING_WITH_ENABLE_SECRET>  ^........<ENABLE_SECRET>
2023-04-17 11:01:17,654 [DEBUG] write: '\n'
2023-04-17 11:01:17,654 [DEBUG] [conn=0, chan=0] Sending 1 data byte
2023-04-17 11:01:17,654 [DEBUG] [conn=0, pktid=25] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
2023-04-17 11:01:17,654 [DEBUG] [conn=0, chan=0, pktid=26] Sent MSG_CHANNEL_DATA (94), 10 bytes
  00000000: 5e 00 00 00 03 00 00 00 01 0a                    ^.........
2023-04-17 11:01:17,884 [DEBUG] [conn=0, pktid=21] Received MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 05                                   .....
2023-04-17 11:01:17,884 [INFO] [conn=0] Sending disconnect: Incomplete packet (2)
2023-04-17 11:01:17,884 [DEBUG] [conn=0, pktid=27] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
2023-04-17 11:01:17,885 [DEBUG] [conn=0, pktid=28] Sent MSG_DISCONNECT (1), 35 bytes
  00000000: 01 00 00 00 02 00 00 00 11 49 6e 63 6f 6d 70 6c  .........Incompl
  00000010: 65 74 65 20 70 61 63 6b 65 74 00 00 00 05 65 6e  ete packet....en
  00000020: 2d 55 53                                         -US
2023-04-17 11:01:17,885 [INFO] [conn=0] Connection failure: Incomplete packet
2023-04-17 11:01:17,885 [INFO] [conn=0, chan=0] Closing channel due to connection close
2023-04-17 11:01:17,885 [INFO] [conn=0, chan=0] Channel closed: Incomplete packet
Traceback (most recent call last):
  File "/home/a0233979/vscode/qh-colo-cleanup/temp.py", line 222, in <module>
    main()
  File "/home/a0233979/vscode/qh-colo-cleanup/temp.py", line 211, in main
    asyncio.run(_main(async_conn))
  File "/usr/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/a0233979/vscode/qh-colo-cleanup/temp.py", line 118, in _main
    async with conn as conn:
  File "/home/a0233979/vscode/qh-colo-cleanup/.venv/lib/python3.11/site-packages/scrapli/driver/base/async_driver.py", line 42, in __aenter__
    await self.open()
  File "/home/a0233979/vscode/qh-colo-cleanup/.venv/lib/python3.11/site-packages/scrapli/driver/base/async_driver.py", line 100, in open
    await self.on_open(self)
  File "/home/a0233979/vscode/qh-colo-cleanup/.venv/lib/python3.11/site-packages/scrapli/driver/core/cisco_iosxe/async_driver.py", line 25, in iosxe_on_open
    await conn.acquire_priv(desired_priv=conn.default_desired_privilege_level)
  File "/home/a0233979/vscode/qh-colo-cleanup/.venv/lib/python3.11/site-packages/scrapli/driver/network/async_driver.py", line 168, in acquire_priv
    await self._escalate(escalate_priv=target_priv)
  File "/home/a0233979/vscode/qh-colo-cleanup/.venv/lib/python3.11/site-packages/scrapli/driver/network/async_driver.py", line 104, in _escalate
    await super().send_interactive(
  File "/home/a0233979/vscode/qh-colo-cleanup/.venv/lib/python3.11/site-packages/scrapli/decorators.py", line 274, in decorate
    result = await wrapped_func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/a0233979/vscode/qh-colo-cleanup/.venv/lib/python3.11/site-packages/scrapli/driver/generic/async_driver.py", line 437, in send_interactive
    raw_response, processed_response = await self.channel.send_inputs_interact(
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/a0233979/vscode/qh-colo-cleanup/.venv/lib/python3.11/site-packages/scrapli/decorators.py", line 191, in decorate
    return await asyncio.wait_for(wrapped_func(*args, **kwargs), timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/asyncio/tasks.py", line 479, in wait_for
    return fut.result()
           ^^^^^^^^^^^^
  File "/home/a0233979/vscode/qh-colo-cleanup/.venv/lib/python3.11/site-packages/scrapli/channel/async_channel.py", line 654, in send_inputs_interact
    buf += await self._read_until_explicit_prompt(prompts=prompts)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/a0233979/vscode/qh-colo-cleanup/.venv/lib/python3.11/site-packages/scrapli/channel/async_channel.py", line 178, in _read_until_explicit_prompt
    b = await self.read()
        ^^^^^^^^^^^^^^^^^
  File "/home/a0233979/vscode/qh-colo-cleanup/.venv/lib/python3.11/site-packages/scrapli/channel/async_channel.py", line 69, in read
    buf = await self.transport.read()
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/a0233979/vscode/qh-colo-cleanup/.venv/lib/python3.11/site-packages/scrapli/decorators.py", line 191, in decorate
    return await asyncio.wait_for(wrapped_func(*args, **kwargs), timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/asyncio/tasks.py", line 479, in wait_for
    return fut.result()
           ^^^^^^^^^^^^
  File "/home/a0233979/vscode/qh-colo-cleanup/.venv/lib/python3.11/site-packages/scrapli/transport/plugins/asyncssh/transport.py", line 264, in read
    buf: bytes = await self.stdout.read(65535)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/a0233979/vscode/qh-colo-cleanup/.venv/lib/python3.11/site-packages/asyncssh/stream.py", line 157, in read
    return await self._session.read(n, self._datatype, exact=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/a0233979/vscode/qh-colo-cleanup/.venv/lib/python3.11/site-packages/asyncssh/stream.py", line 528, in read
    raise exc
  File "/home/a0233979/vscode/qh-colo-cleanup/.venv/lib/python3.11/site-packages/asyncssh/connection.py", line 1275, in data_received
    while self._inpbuf and self._recv_handler():
                           ^^^^^^^^^^^^^^^^^^^^
  File "/home/a0233979/vscode/qh-colo-cleanup/.venv/lib/python3.11/site-packages/asyncssh/connection.py", line 1524, in _recv_packet
    raise ProtocolError(str(exc)) from None
asyncssh.misc.ProtocolError: Incomplete packet
ronf commented 1 year ago

There's definitely a problem with the SSH implementation on the target system. As you noted, it is sending a MSG_IGNORE packet with a length of 5 in this case, but this ignore packet has no bytes after that length field, triggering the "Incomplete packet" error. Previous MSG_IGNORE packets had a length of 0, and they worked fine. In theory, a length of 5 would also be allowed by the spec, but only if that MSG_IGNORE included an additional 5 bytes of data after this length field.

I noticed that just before this you were sending a secret followed by a newline ('\n') in response to what looks like a password prompt. Could that password be wrong? If so, the target device may be trying to close the connection and it got something wrong in the output it generated when doing that, but perhaps you wouldn't see that if the password was correct. I also wonder if maybe it is expecting a carriage return ('\r') instead of a newline at the end of the password, and perhaps that might be triggering the problem. Could you try changing the '\n' to a '\r' and sees if it makes a difference?

ronf commented 1 year ago

Looking at OpenSSH source, it appears that it might be ignoring any payload data sent in a MSG_IGNORE, not even verifying that the length field in the MSG_IGNORE matches with the number of bytes remaining in the packet. This seems questionable to me, but if you want to see if you can progress any further by ignoring this error, you can try commenting out the following lines in SSHConnection._process_ignore in connection.py:

    def _process_ignore(self, _pkttype: int, _pktid: int,
                        packet: SSHPacket) -> None:
        """Process an ignore message"""

        # pylint: disable=no-self-use

        _ = packet.get_string()     # data            <------- COMMENT THIS OUT
        packet.check_end()                            <------- COMMENT THIS OUT
JostLuebbe commented 1 year ago

Hi @ronf,

Thanks for the quick response! Commenting out those two lines fixed the issue. We went from getting ~250 devices giving us an "Incomplete Packet" exception to none. We were able to authenticate to all of them and run a "show version" command. I don't think the newline or the password were the issue, because the password works when connecting via a normal SSH connection on the same device, and the same newline works on other devices.

You mentioned this is an implementation problem on the target system, should I open a ticket with Cisco to see if they'll release a fix for it? Or is there some other more permanent fix we could implement?

Thanks again for the awesome support, really appreciate it.

ronf commented 1 year ago

Regarding the newline, I wasn't sure that would be an issue, but I only mentioned it because when you SSH by hand, you would be sending a carriage return when you hit "enter" after providing the password, not a newline. It probably accepts both, but I thought maybe the newline was confusing it and getting you into a code path where it sent the broken MSG_IGNORE.

In terms of a more permanent fix, I was thinking of possibly looking at the server version and selectively turning off the validation of the data argument. This might look something like:

    def _process_ignore(self, _pkttype: int, _pktid: int,
                        packet: SSHPacket) -> None:
        """Process an ignore message"""

        # Work around missing payload bytes in an ignore message
        # in some Cisco SSH servers
        if b'Cisco' not in self._server_version:
            _ = packet.get_string()     # data
            packet.check_end()

That way, this checking is still enabled for most SSH servers, but it could allow you to work around the bug when connecting to a Cisco SSH server. Of course, there may be Cisco SSH servers out there which don't have this problem, but without more data it's hard to know how to narrow that down further (by something like the version number it reports).

Could you give this version a try?

If you wanted to open a ticket with Cisco on this, that'd be great. I've managed to get things like this fixed in some other cases and that might help other SSH implementations to interoperate better with it. Of course, getting all the broken versions already deployed out there updated will mean we'll be living with this for a long time, even if it does get fixed in future releases.

When reporting this, you can reference section 11.2 of RFC 4253, which defines the ignore message as:

  byte      SSH_MSG_IGNORE
  string    data

The definition of "string" is in [section 5]() of RFC 4251:

  Arbitrary length binary string.  Strings are allowed to contain
  arbitrary binary data, including null characters and 8-bit
  characters.  They are stored as a uint32 containing its length
  (number of bytes that follow) and zero (= empty string) or more
  bytes that are the value of the string.  Terminating null
  characters are not used.

  Strings are also used to store text.  In that case, US-ASCII is
  used for internal names, and ISO-10646 UTF-8 for text that might
  be displayed to the user.  The terminating null character SHOULD
  NOT normally be stored in the string.  For example: the US-ASCII
  string "testing" is represented as 00 00 00 07 t e s t i n g.  The
  UTF-8 mapping does not alter the encoding of US-ASCII characters.

However, in the case of the broken Cisco MSG_IGNORE, the 4-byte length field of the "data" string in the message is "00 00 00 05" (a length of 5 bytes) but it is not followed by actual data of that length. Perhaps the implementation thought the length was supposed to include the 1-byte message identifier (MSG_IGNORE == 2) and the length field itself, as that would total 5 bytes (though that could also just be a coincidence). In any case, that's not how strings in SSH work. The length only counts bytes immediately after a 4-byte length field, and does not include the length itself or any other data in that count. If the packet ends before that number of bytes are present, it is treated as an incomplete packet.

JostLuebbe commented 1 year ago

Could you give this version a try?

That version of the fix also works for all of our devices.

If you wanted to open a ticket with Cisco on this, that'd be great.

It seems Cisco has already fixed this issue in later versions of their router/switch code and their (admittedly fair) advice to us would be to upgrade our systems. We would still highly appreciate it if you could support these older versions by including that check in the _process_ignore function.

However, if you wanted to make that check a little more specific, in testing in our environment it seems the 12.x versions of Cisco IOS were the main issue. I wish I could be even more specific, but Cisco's versioning is unfortunately not very friendly for these sorts of checks.

ronf commented 1 year ago

The SSH version reported by the target in this case is SSH-2.0-Cisco-1.25, so I'm not entirely sure how to associate that with an IOS version. I'm happy to make this change, but it would probably need to be done for all Cisco devices. Since it only affects packet validation and not anything AsyncSSH sends, though, I think that's probably ok. Were you able to find a reference to a bug report or release notes mentioning the fix that I could add to the comments in the change?

ronf commented 1 year ago

The workaround for this issue is now available in the "develop" branch as commit 16ece51, and will be included in the next release. Thanks again for reporting this and providing such detailed debug information!

JostLuebbe commented 1 year ago

No problem! Thanks for being so receptive and putting that fix in for us. Unfortunately I wasn't able to find a specific release note from Cisco mentioning changing or updating their SSH implementation on versions past 12.x.

ronf commented 1 year ago

The fix for this is now available as part of AsyncSSH 2.13.2.