Questions about imported authorized_keys, option key expansion (%r, remote user) and agent forwarding path

[feeloo007]

Hello, thanks for this great job !

I feel like I've done a lot more than some other SSH frameworks in a lot less time and a lot less lines.

However, there are a few things I'm missing:

• Is it possible to get the path to the forwarding socket obtained for the session? The goal is to be able to pass it as an SSH_AUTH_SOCK variable to an ssh command (which uses connection persistence).

• An authorized_keys file was loaded via SSHServerConnection.set_authorized_keys(). For the key, the command is set as follows:

command="ssh -q %r@bastion"

Is there a helper function that does string expansion (replacing %r by the current remote user)?

The goal is to make an SSH server with AsyncSSH that uses openSSH connection persistence to a bastion (routing based on usernames, exemple of "username" user@limbo.apps-qual+SSH-APP-LIMBO-CMP-SSH-NUM-01+user).

The code is totally inspired by https://github.com/ronf/asyncssh/issues/657 (thanks to ronf and xuoguoto!!).

Contents of the file authorized_keys

/var/tmp/authorized_keys

no-user-rc,agent-forwarding,pty,x11-forwarding,command="ssh -q %r@bastion" ssh-ed25519 ...

Contents of the file redirect_server.py

import asyncio, asyncssh, subprocess, sys, logging, os, fcntl, struct, termios

logger = logging.getLogger(__name__)
logging.basicConfig(level=logging.DEBUG)

async def handle_client(process: asyncssh.SSHServerProcess) -> None:

    local_pty, local_tty = os.openpty()

    local_proc = await asyncio.create_subprocess_shell(
#                 EXPAND_TOKENS_HELPER( process.command )
                   process.command
                   ,
                   shell=True
                   ,
                   stdin=local_tty
                   ,
                   stdout=local_tty
                   ,
                   stderr=local_tty
#                   ,
#                   env=dict( os.environ, { "SSH_AUTH_SOCK": GET_CURRENT_AGENT_SOCKET_PATH_HELPER } )
              )

    os.close( local_tty )

    await process.redirect(
            stdin = local_pty
            ,
            stdout = os.dup( local_pty )
          )

    await process.stdout.drain()

    process.exit(0)

    await process.wait_closed()

class MySSHServer(asyncssh.SSHServer):
    def connection_made(self, conn: asyncssh.SSHServerConnection) -> None:
        self._conn = conn

    def begin_auth(self, username: str) -> bool:
        try:
            self._conn.set_authorized_keys('/var/tmp/authorized_keys')
        except OSError:
            pass

        return True

async def start_server() -> None:
    await asyncssh.create_server(MySSHServer, '', 8022,
                                 server_host_keys=['/var/tmp/id_server_ed25519'],
                                 process_factory=handle_client,
                                 agent_forwarding=True,
                                 reuse_address=True,
                                 reuse_port=True,
                                 line_editor=False,
                                )

loop = asyncio.new_event_loop()

try:
    loop.run_until_complete(start_server())
except (OSError, asyncssh.Error) as exc:
    sys.exit('Error starting server: ' + str(exc))

loop.run_forever()

Thanks in advance for your answers

Philippe

[ronf]

Hello - glad to hear you are finding AsyncSSH useful!

Is it possible to get the path to the forwarding socket obtained for the session? The goal is to be able to pass it as an SSH_AUTH_SOCK variable to an ssh command (which uses connection persistence).

Are you talking agent forwarding? If so, there's a method called get_agent_path() which you can call on the SSHServerConnection instance. If agent forwarding is enabled and the client requested it, that should return the local path to a UNIX domain socket that is being forwarded back to the SSH client's agent.

An authorized_keys file was loaded via SSHServerConnection.set_authorized_keys(). For the key, the command is set as follows:

command="ssh -q %r@bastion"

Is there a helper function that does string expansion (replacing %r by the current remote user)?

Sorry, no. There is code to do that kind of percent expansion in the AsyncSSH config module, but it's very specific to the expansions used in config files. So far, I haven't needed anything like that anywhere else.

Is there a reason you are running OpenSSH as a command on the server host instead of using AsyncSSH to make the upstream connection and then using redirects to forward data between the upstream SSH client connection and the server connection which triggered it? You could get at things like the username from the server connection and pass that through (after parsing out the remote host) without the need for a "command" argument in an authorized keys file. You could also pass other things through like terminal type and environment variables if you wanted. AsyncSSH will even pass window size information through, and even pick up window size changes in the middle of a connection and forward them.

You mentioned connection persistence. Do you mean the connection multiplexing feature in OpenSSH? While AsyncSSH doesn't let you share an SSH connection between independent processes the way OpenSSH does, it can open multiple sessions on something like a bastion host over a single upstream SSH connection, as long as all the sessions are created by the same process (and using the same asyncio event loop).

[feeloo007]

Hello ronf,

thanks for your answer!

Yes, I am talking about agent forwarding.

Concerning the acquisition of the path to the agent, in the handle_client function, process._conn.get_agent_path() returns the path to the socket. My pseudo-code to define the environment is not good but I am correcting this to test that it works well.

Concerning the use of openSSH for the final client connection to the bastion, the idea was not to replay the entire authentication on the bastion (quite long in our platform) rather than using multiplexing.

On the front end, I will do tests to replace OpenSSH with asyncssh which could indeed be lighter.

The chain is:

ssh client with forward agent of a known key in subjectAlternativeIdentity on an Active directory

• Bastion that validates the identity by addressing Active Directory

• OpenSSH with public key known to Active Directory by users

• wishlist from charmbracelet to display personalized menus per user, each entry being a different user of the bastion leading to a different application. wishlist does not know how to launch a command, only make a connection.

• asyncssh as a server that launches an ssh command using persistence to the bastion.

• Bastion that validates the identity by addressing Active Directory.

• For example, access to an environment with ansible configured using the forwarding agent to access servers using the starting identity to be used again through the bastion.

In absolute terms, the initial openssh server is not necessary, wishlist should be able to do the job (although the user/key ventilation is less fine than with openssh) but without it, the connection to the bastion "hangs".

Thank you very much for your feedback!

Philippe

[feeloo007]

I got it working but I think loading a client configuration and using ProxyCommand would lead to the same result.

I'll try later.

Thanks ronf!

[ronf]

Glad to hear you were able to get something working...

That said, you probably don't want to use ProxyCommand here. That's pretty costly, as it has to fork & exec a new process for every client connection, similar to the overhead overhead in the current code of calling create_subprocess_shell() to run an OpenSSH client on every new connection. One of the main points of AsyncSSH is to allow multiple SSH sessions to run in a single process, but I'm still not completely clear on what happens between AsyncSSH as a server, the OpenSSH client it invokes to the bastion server, and what happens after that on the bastion server. I'm also not sure if the bastion host you are using would support TCP port forwarding (to allow using the equivalent of ProxyJump through the bastion, but I'd be happy to help you explore that further if you like.

[ronf]

Here's roughly what I had in mind to get rid of the create_subprocess_shell():

If you want to avoid having to authenticate to the bastion host every time, you can reuse a connection for multiple incoming SSHServerProcess instances by keeping a dictionary of open connections (one per username). This might look something like:

connections = {}

async def handle_client(process: asyncssh.SSHServerProcess -> None:
    username = process.get_extra_info('username')
    agent_path = process.channel.get_connection().get_agent_path()

    try:
        conn = connections[username]
    except KeyError:
        conn = await asyncssh.connect('bastion', username=username)
        connections[username] = conn

    await conn.run(
        env = {'SSH_AUTH_SOCK': agent_path},
        term_type=process.term_type,
        stdin=process.stdin, stdout=process.stdout)

If you need to modify the username in some way, you can do that before you pass it to asyncssh.connect().

This will allow you to authenticate once but use that connection for many inbound sessions. You will need to have some error handling I don't show here to deal with what happens if an existing server connection breaks and needs to be restarted, but this should give you a rough idea of how this might look.

Once you have a connection, you can just start an outbound AsyncSSH client session which attaches to the stdin/stdout of the SSHServerProcess and passes through the terminal type. You can also pass other things through for the SSHServerProcess like term_size and term_modes if you want to. I'm assuming here you are setting a terminal type and getting a PTY, so you don't need to redirect stderr.

All the Active Directory stuff would still happen in the bastion host, as would anything related to personalized menus.

[ronf]

Note that with the above, you wouldn't need the "command=" in the authorized keys file, and could potentially even just set authorized_keys when you open the SSH server listener, unless you need a different authorized_keys file per user. If you can share a common authorized_keys on the connection to AsyncSSH, you could eliminate the MySSHServer class as well.

[feeloo007]

Thank you for these explanations, ronf !

The bastion server is a commercial offer (wallix bastion, sorry, the public part of the site is not explicit about its operation).

The main purpose it serves (administrative constraints) is to make complete recordings of sessions.

For this reason, we cannot use ProxyJump, it acts as a man-in-the-middle itself.

It can use many authentication sources, assign user rights to specific targets, etc.

However, all this comes at a cost in terms of processing time.

However, all this has a cost from a processing time point of view (session recording has a lower cost than the primary authentication duration + rights definition).

This is where the persistence control of the OpenSSH client comes into play, do not replay a full authentication for a connection established in the last seconds.

To illustrate, if we use the ansible setup module while accessing a host through the bastion without connection persistence, the processing will take between 5 and 10 seconds, potentially much longer.

With connection persistence, the processing will take 1 or 2 seconds.

The overhead of starting the ssh client (many times for setup module) is negligible compared to the overall performance improvement.

However, the proposition of not using an additional ssh client process is attractive.

During the week, probably Tuesday, I will set up the code you sent.

However, I think that to meet a need equivalent to that provided by a persistent connection, delayed connections closing is necessary.

Does such a feature exist in asyncssh (do not close the socket immediately when the server has requested it but keep the socket open for an indicated period of time to allow the socket to be reused) ?

I indicated this pseudo process but I have no idea how peristence connection is really implemented in openSSH (At the system level, a master process remains, what keeps the socket open, but ssh protocol level, no idea how it works).

Thank you Ronf for all this great work!

Philippe

[ronf]

Thanks for the additional detail on the bastion host - that helps. When you make the initial connection to that host that you are using along with persistence, do you use the same credentials on the bastion host for all users, or does your connection persistence only apply to multiple connections coming from the same user? Since you talked about passing in a different "remote user" name to the bastion, I'm guessing that your connection persistence would only be for connections coming from the same user and each user would have different auth credentials on the bastion, but I just wanted to make sure.

With OpenSSH, each session runs on a separate UNIX process, but OpenSSH does some clever passing around of file descriptors between processes to allow the different sessions to all send their messages over a single multiplexed SSH connection, allowing it to only need to do auth once. Even after the last session is closed on a connection, the multiplexer will keep the connection open for a limited time, allowing future connections to continue to benefit from it. If a connection remains idle for some period of time with no new sessions being started, the multiplexer thread will close the connection and exit, and any future connection will have to re-do the auth (and start a new multiplexer).

With AsyncSSH, all the sessions run in a single process, and you can directly control when you share an SSH connection between sessions and how long you keep those connections open for. You can even have multiple open connections (such as connections authenticated as different users) coexist within a single UNIX process.

Basically, in AsyncSSH you call asyncssh.connect() to create a new SSH connection and do a full set of authentication. This is typically followed up by a call to conn.run() or conn.create_process(), or other functions used to open SSH sessions. Since these are separate steps, you can call the latter functions to create multiple sessions on a conn object to get the equivalent of OpenSSH"s multiplexing, where all the sessions on a connection share the same auth. You can also control how long you keep an SSH connection open for potential use by future sessions. If you want a connection to automatically close after they are idle for some period of time, you can do that, but you'd need to code that yourself -- there's no equivalent right now to the OpenSSH ControlPersist setting in AsyncSSH. The normal AsyncSSH behavior is to keep connections open indefinitely, or until you explicitly close them, though there is a "keepalive" mechanism you can configure to detect and clean up "dead" connections, where something went wrong on the network and a persistent connection may get broken in a way that didn't cause the connection close to be noticed at the TCP level.