agent forwarding fails and leave agent in unresponsive state

[mplattner]

As briefly described in https://github.com/dlech/KeeAgent/issues/349#issuecomment-1257996972, when using v0.13.2, the agent hangs when I use agent forwarding, i.e., I try to open a nested ssh session.

Here's how to reproduce it, assuming a terminal opened on myhost:

# connect to host1, works correctly
myhost-prompt> ssh host1

# being connected to host1 from myhost, open a connection to host2
host1-prompt> ssh host2
| <-- stale cursor (*)

# (*) ssh-key-being-used-notification pops up (see below), but then the terminal hangs with no further output

Once I close the hanging terminal on myhost, I can again connect to e.g. host1, but see the same behavior once I try to use agent forwarding (again).

Versions: myhost OS: Windows 10 21H2 myhost SSH client: OpenSSH_for_Windows_8.1p1, LibreSSL 3.0.2 myhost KeePass: 2.51.1 myhost KeeAgent: v0.13.2

host1 and host2 OS: Ubuntu 18.04.5 LTS host1 and host2 SSH client: OpenSSH_7.6p1 Ubuntu-4ubuntu0.7, OpenSSL 1.0.2n 7 Dec 2017

However, I see the same behavior when host1 and host2 are up-to-date Kali Linux machines with OpenSSH_9.0p1 Debian-1+b1, OpenSSL 3.0.4 21 Jun 2022.

---

Below is the output of the command host1-prompt> ssh -vvv host2 (host1/2 being the Ubuntu 18.04.5/SSH 7.6 ones):

debug1: rekey after 134217xxx blocks
debug2: key:  (0x......), agent
debug2: key: /root/.ssh/id_rsa ((nil))
debug2: key: /root/.ssh/id_dsa ((nil))
debug2: key: /root/.ssh/id_ecdsa ((nil))
debug2: key: /root/.ssh/id_ed25519 ((nil))
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: ED25519 SHA256:PGJ............
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: pkalg ssh-ed25519 blen 51
debug2: input_userauth_pk_ok: fp SHA256:PGJ............
debug3: sign_and_send_pubkey: ED25519 SHA256:PGJ............
debug3: send packet: type 50
debug3: receive packet: type 52
debug1: Authentication succeeded (publickey).
Authenticated to myhost2 ([127.0.1.1]:4000).
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug3: send packet: type 90
debug1: Requesting no-more-sessions@openssh.com
debug3: send packet: type 80
debug1: Entering interactive session.
debug1: pledge: network
debug3: receive packet: type 80
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug3: receive packet: type 91

---

Side-note: the pop-up does not correctly show the name of the key. Might be related to the comment not being loaded correctly (#348). The incorrect pop-up is independent of using agent forwarding or not.

Let me know if you need more info. Thanks!

[mplattner]

This might be related to the changes of #341. Also relevant: using KeeAgent v0.12.1, agent forwarding worked correctly with OpenSSH_7.6p1 Ubuntu-4ubuntu0.7, OpenSSL 1.0.2n 7 Dec 2017 on the remote host(s), but not with OpenSSH_9.0p1 Debian-1+b1, OpenSSL 3.0.4 21 Jun 2022. For the latter version I got the result described in #341 (agent refused operation).

[Gnarfoz]

Using the build from https://github.com/dlech/KeeAgent/actions/runs/3198121096, I am experiencing almost the same thing. Except that for me, I can successfully connect from local to host1 and further from host1 to host2. However, once that "inner" connection is established, the agent stops working and I can't connect for example from local to host3. If I log out of host2 (dropping me back to host1), it starts working again.

Clients get stuck waiting between these two debug output lines:

debug1: rekey in after 134217728 blocks
XXX here XXX
debug1: get_agent_identities: bound agent to hostkey

Local is: OpenSSH_for_Windows_8.9p1, LibreSSL 3.4.3 Host1 is: OpenSSH_7.4p1, OpenSSL 1.0.2k-fips Host2 is: OpenSSH_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013

[dlech]

It would be helpful if someone could set up a couple docker containers using docker compose to reproduce the problem.

See https://github.com/dlech/SshAgentLib/tree/master/docker for a couple of existing ones.

[mplattner]

I've had a look at the existing Dockerfiles. I'm not sure about your expectations: you want a docker-compose file that sets up 2 SSH servers that are on the same network so that we can test ssh client --- (ssh agent) --> docker sshd 1 --- (ssh agent (via forwarding)) ---> docker sshd 2, right?

[dlech]

Yes, if I understand the problem correctly, this seems like the minimum needed to reproduce the issue.

[dlech]

Do you have something in ~/.ssh/config to enable agent forwarding since no one seems to be using the ssh -A option?

[dlech]

I was finally able to reproduce some issues with the Windows OpenSSH agent using Tabby (#384 ) configured to use Windows OpenSSH (nothing related to agent forwarding but more along the lines of "after a while it stops working").

[Gnarfoz]

Awesome, thanks @dlech! 🎉