can't open tunnel on asyncssh version 2.3.0 - Permission denied

[ProcnerKacper]

Hi, I wanted to connect to devices via reverse SSH, however, I encountered an issue, and as a result, I'm unable to establish the connection. Is there anything that can be done about it because I'm currently unable to connect to multiple devices?

asyncssh==2.3.0

but on the newest version it works

error: File "main.py", line 260, in open self._connection = await asyncssh.connect( File "/Users/kacper/Desktop/test-ssh/conda/lib/python3.8/site-packages/asyncssh/connection.py", line 6344, in connect return await _connect(options.host, options.port, loop, options.tunnel, File "/Users/kacper/Desktop/test-ssh/conda/lib/python3.8/site-packages/asyncssh/connection.py", line 207, in _connect await conn.wait_established() File "/Users/kacper/Desktop/test-ssh/conda/lib/python3.8/site-packages/asyncssh/connection.py", line 2082, in wait_established await self._waiter asyncssh.misc.PermissionDenied: Permission denied

[ronf]

Can you share the code you are attempting to use to connect (removing any private information such as hostname/IP, username, or password)?

The error you're getting suggests that you aren't providing valid auth credentials, but I can't really tell much else without some sample code and preferably also a debug log.

Also, can you confirm that this is working with the latest version of AsyncSSH? Version 2.3.0 you are attempting to use is almost 4 years old, so there are a number of bug fixes since then. Is there a particular reason you need to stick to that old a version?

[ProcnerKacper]

On the newest version, it is working fine without changes in the code. We are using this old version because it was deployed to Rassberries PI and sent to clients and no one takes care of updating it, I'm just maintaining the whole system from the backend site and want to check Raspberry PI to see why it isn't working anymore

def load_private_key(private_key_path: str) -> asyncssh.SSHKey:
    return asyncssh.read_private_key(private_key_path)

class ReverseSSHTunnel(LoggingMixin):
    def __init__(self):
        LoggingMixin.__init__(self)
        self._connection = None
        self._listener = None

    async def open(
        self,
        host: str,
        port: int,
        username: str,
        client_keys: List[asyncssh.SSHKey],
        known_hosts: Optional[str] = None,
        listen_host: str = "",
        listen_port: int = 48822,
        dest_host: str = "localhost",
        dest_port: int = 22,
    ):
        if self._connection:
            self._logger.warning("Tunnel already opened!")
            return

        self._connection = await asyncssh.connect(
            host=host,
            port=port,
            username=username,
            client_keys=client_keys,
            known_hosts=known_hosts,
        )

        self._listener = await self._connection.forward_remote_port(
            listen_host, listen_port, dest_host, dest_port
        )

        self._logger.info("Tunnel opened!")

    async def close(self):
        if self._listener:
            self._listener.close()
            await self._listener.wait_closed()
            self._listener = None

        if self._connection:
            self._connection.close()
            await self._connection.wait_closed()
            self._connection = None

        self._logger.info("Tunnel closed!")

class TunnelManager(LoggingMixin):
    def __init__(self, loop, args: Namespace):
        LoggingMixin.__init__(self)

        config = load_config(args.config_file_path)
        private_key = load_private_key(args.private_key_path)

        logging.debug("Loaded config: %s", config)

        self._loop = loop
        self._schema = MessageSchema()
        self._tunnel = ReverseSSHTunnel()
        self._request_consumer = ConsumerWatchdog(
            consumer=QueueMessageConsumer(
                amqp_url=config.amqp_url,
                queue_name=config.queue_name,
                message_handler=self._schedule_message_processing,
            ),
            check_delay_s=args.watchdog_check_delay,
        )
        self._client_keys = [private_key]

    def run(self):
        self._loop.create_task(self._request_consumer.start())
        self._loop.run_forever()

    async def _open_tunnel(self, msg: dict) -> None:
        await self._tunnel.open(
            host=msg["host"],
            port=22,
            username=msg["user"],
            client_keys=self._client_keys,
            listen_port=msg["l_port"],
            dest_port=msg["port"],
        )

    async def _close_tunnel(self, msg: dict) -> None:
        await self._tunnel.close()

    async def _message_handler(self, body: bytes):
        try:
            msg = self._schema.loads(body)
        except ValidationError as err:
            self._logger.error(err)
            return

        if msg["type"] == "open_tunnel":
            await self._open_tunnel(msg)
        elif msg["type"] == "close_tunnel":
            await self._close_tunnel(msg)

    def _schedule_message_processing(self, body: bytes):
        self._logger.debug("Message processing scheduled! body: '%s", body)
        self._loop.create_task(self._message_handler(body))

def main():
    tunnel_manager = TunnelManager(loop=asyncio.new_event_loop(), args=parse_args())
    tunnel_manager.run()

if __name__ == "__main__":
    main()

[ronf]

Nothing looks unusual to me here.

Can you generate a level 2 debug log? I see you already have logging enabled, so it should just be a matter of setting the log level to 'DEBUG' and running asyncssh.set_debug_level(2) before doing the asyncssh.connect().

In particular, I'd like to look at the types of auth being attempted both in the failed case of the older version and in the successful case with the latest version.

[ProcnerKacper]

version 2.3.0 faild attempt:

EBUG:asyncssh:[conn=0] Requesting key exchange
DEBUG:asyncssh:[conn=0]   Key exchange algs: curve25519-sha256,curve25519-sha256@libssh.org,curve448-sha512,ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,ecdh-sha2-1.3.132.0.10,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group15-sha512,diffie-hellman-group16-sha512,diffie-hellman-group17-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha1,rsa2048-sha256
DEBUG:asyncssh:[conn=0]   Host key algs: sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ssh-ed448-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-1.3.132.0.10-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,ssh-ed25519,ssh-ed448,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256,ecdsa-sha2-1.3.132.0.10,rsa-sha2-256,rsa-sha2-512,ssh-rsa
DEBUG:asyncssh:[conn=0]   Encryption algs: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
DEBUG:asyncssh:[conn=0]   MAC algs: hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
DEBUG:asyncssh:[conn=0]   Compression algs: zlib@openssh.com,none
DEBUG:asyncssh:[conn=0] Received key exchange request
DEBUG:asyncssh:[conn=0]   Key exchange algs: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
DEBUG:asyncssh:[conn=0]   Host key algs: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
DEBUG:asyncssh:[conn=0]   Client to server:
DEBUG:asyncssh:[conn=0]     Encryption algs: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
DEBUG:asyncssh:[conn=0]     MAC algs: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
DEBUG:asyncssh:[conn=0]     Compression algs: none,zlib@openssh.com
DEBUG:asyncssh:[conn=0]   Server to client:
DEBUG:asyncssh:[conn=0]     Encryption algs: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
DEBUG:asyncssh:[conn=0]     MAC algs: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
DEBUG:asyncssh:[conn=0]     Compression algs: none,zlib@openssh.com
DEBUG:asyncssh:[conn=0] Beginning key exchange
DEBUG:asyncssh:[conn=0]   Key exchange alg: curve25519-sha256
DEBUG:asyncssh:[conn=0]   Client to server:
DEBUG:asyncssh:[conn=0]     Encryption alg: chacha20-poly1305@openssh.com
DEBUG:asyncssh:[conn=0]     MAC alg: chacha20-poly1305@openssh.com
DEBUG:asyncssh:[conn=0]     Compression alg: zlib@openssh.com
DEBUG:asyncssh:[conn=0]   Server to client:
DEBUG:asyncssh:[conn=0]     Encryption alg: chacha20-poly1305@openssh.com
DEBUG:asyncssh:[conn=0]     MAC alg: chacha20-poly1305@openssh.com
DEBUG:asyncssh:[conn=0]     Compression alg: zlib@openssh.com
DEBUG:asyncssh:[conn=0] Requesting service ssh-userauth
DEBUG:asyncssh:[conn=0] Completed key exchange
DEBUG:asyncssh:[conn=0] Received extension info
DEBUG:asyncssh:[conn=0]   server-sig-algs: ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
DEBUG:asyncssh:[conn=0] Request for service ssh-userauth accepted
INFO:asyncssh:[conn=0] Beginning auth for user root
DEBUG:asyncssh:[conn=0] Remaining auth methods: publickey
DEBUG:asyncssh:[conn=0] Preferred auth methods: gssapi-keyex,gssapi-with-mic,hostbased,publickey,keyboard-interactive,password
DEBUG:asyncssh:[conn=0] Trying public key auth with rsa-sha2-256 key
DEBUG:asyncssh:[conn=0] Remaining auth methods: publickey
DEBUG:asyncssh:[conn=0] Preferred auth methods: gssapi-keyex,gssapi-with-mic,hostbased,publickey,keyboard-interactive,password
INFO:asyncssh:[conn=0] Auth failed for user root
INFO:asyncssh:[conn=0] Connection failure: Permission denied
INFO:asyncssh:[conn=0] Aborting connection

version 2.14.2 success attempt

DEBUG:asyncssh:[conn=0] Sending version SSH-2.0-AsyncSSH_2.14.2
DEBUG:asyncssh:[conn=0] Received version SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.5
DEBUG:asyncssh:[conn=0] Requesting key exchange
DEBUG:asyncssh:[conn=0]   Key exchange algs: curve25519-sha256,curve25519-sha256@libssh.org,curve448-sha512,ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,ecdh-sha2-1.3.132.0.10,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group15-sha512,diffie-hellman-group16-sha512,diffie-hellman-group17-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256@ssh.com,diffie-hellman-group14-sha1,rsa2048-sha256,ext-info-c,kex-strict-c-v00@openssh.com
DEBUG:asyncssh:[conn=0]   Host key algs: rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ssh-ed448-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-1.3.132.0.10-cert-v01@openssh.com,rsa-sha2-256,rsa-sha2-512,ssh-rsa-sha224@ssh.com,ssh-rsa-sha256@ssh.com,ssh-rsa-sha384@ssh.com,ssh-rsa-sha512@ssh.com,ssh-rsa,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,ssh-ed25519,ssh-ed448,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256,ecdsa-sha2-1.3.132.0.10
DEBUG:asyncssh:[conn=0]   Encryption algs: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
DEBUG:asyncssh:[conn=0]   MAC algs: hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-sha256-2@ssh.com,hmac-sha224@ssh.com,hmac-sha256@ssh.com,hmac-sha384@ssh.com,hmac-sha512@ssh.com
DEBUG:asyncssh:[conn=0]   Compression algs: zlib@openssh.com,none
DEBUG:asyncssh:[conn=0] Received key exchange request
DEBUG:asyncssh:[conn=0]   Key exchange algs: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
DEBUG:asyncssh:[conn=0]   Host key algs: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
DEBUG:asyncssh:[conn=0]   Client to server:
DEBUG:asyncssh:[conn=0]     Encryption algs: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
DEBUG:asyncssh:[conn=0]     MAC algs: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
DEBUG:asyncssh:[conn=0]     Compression algs: none,zlib@openssh.com
DEBUG:asyncssh:[conn=0]   Server to client:
DEBUG:asyncssh:[conn=0]     Encryption algs: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
DEBUG:asyncssh:[conn=0]     MAC algs: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
DEBUG:asyncssh:[conn=0]     Compression algs: none,zlib@openssh.com
DEBUG:asyncssh:[conn=0] Beginning key exchange
DEBUG:asyncssh:[conn=0]   Key exchange alg: curve25519-sha256
DEBUG:asyncssh:[conn=0]   Client to server:
DEBUG:asyncssh:[conn=0]     Encryption alg: chacha20-poly1305@openssh.com
DEBUG:asyncssh:[conn=0]     MAC alg: chacha20-poly1305@openssh.com
DEBUG:asyncssh:[conn=0]     Compression alg: zlib@openssh.com
DEBUG:asyncssh:[conn=0]   Server to client:
DEBUG:asyncssh:[conn=0]     Encryption alg: chacha20-poly1305@openssh.com
DEBUG:asyncssh:[conn=0]     MAC alg: chacha20-poly1305@openssh.com
DEBUG:asyncssh:[conn=0]     Compression alg: zlib@openssh.com
DEBUG:asyncssh:[conn=0] Requesting service ssh-userauth
DEBUG:asyncssh:[conn=0] Completed key exchange
DEBUG:asyncssh:[conn=0] Received extension info
DEBUG:asyncssh:[conn=0]   server-sig-algs: ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
DEBUG:asyncssh:[conn=0] Request for service ssh-userauth accepted
INFO:asyncssh:[conn=0] Beginning auth for user root
DEBUG:asyncssh:[conn=0] Remaining auth methods: publickey
DEBUG:asyncssh:[conn=0] Preferred auth methods: gssapi-keyex,gssapi-with-mic,hostbased,publickey,keyboard-interactive,password
DEBUG:asyncssh:[conn=0] Trying public key auth with rsa-sha2-256 key
DEBUG:asyncssh:[conn=0] Signing request with rsa-sha2-256 key
INFO:asyncssh:[conn=0] Auth for user root succeeded
INFO:asyncssh:[conn=0] Creating remote TCP forwarder from port 49001 to localhost, port 22
INFO:asyncssh:[conn=0] Creating remote TCP listener on port 49001
DEBUG:asyncssh:[conn=0] Received unknown global request: hostkeys-00@openssh.com

[ronf]

Thanks. There are some minor differences in the key exchange which would be expected from the version difference, but I don't think those explain what you're seeing. In particular, the client adds some Tectia SSH (ssh.com) algorithms, but those should be ignored when the client talks to OpenSSH. In addition, the newer version supports SSH ext-info and has the strict kex support to deal with the Terrapin problem.

For host key algs, support was added for SHA-2 signatures in certificates, and there's a bit of reordering of some of the other algorithms, but nothing jumps out at me that should be relevant here.

Assuming that you are using the exact same client key for authentication in both tests, the only thing I can think of is that there was a bug in the signature generation which was fixed somewhere between the two versions. I know there were some bugs in signing related to the use of OpenSSH certificates, but it doesn't look like you are using certs here. Is that right?

If you are using certificates, older versions of OpenSSH (before 7.8) only supported RSA with SHA-1, so that could also be a problem. Later versions of AsyncSSH (2.13.2 and later I think) added a workaround for that, assuming that the OpenSSH server doesn't have SHA-1 disabled (as newer versions do by default).

[ProcnerKacper]

I checked the other version and it seems to be that version 2.5 fixes this issue. on the server OpenSSH_7.6p1 Ubuntu-4ubuntu0.5, OpenSSL 1.0.2n 7 Dec 2017

[ronf]

Do you have an ~/.ssh/config file on the client you are running AsyncSSH on? If so, does that file happen to contain an IdentitiesOnly option? Support for that was added in 2.5.0, and could explain a difference in which keys are being tried. Also, are you use an SSH agent here? There were some other changes related to that.

More generally, it might be worth printing out some information about which specific key is being used to see if there's any difference in the two cases.

Have you confirmed that the fix is in 2.5.0 itself and not the previous version (2.4.2)?

[ProcnerKacper]

Hi, sorry for long not responding, I confirmed that it is not working on 2.4.2 but on 2.5.0 it is there is no ssh/config on the both server and raspberry device what do you mean by use ssh agent?

how can i print this informataion or where i can look for it? More generally, it might be worth printing out some information about which specific key is being used to see if there's any difference in the two cases.

[ronf]

ssh-agent is a process that can hold onto key information for you, rather than having AsyncSSH (or other SSH clients) load the keys themselves. I just wanted to make sure that the keys aren't coming from SSH agent, as there were some issues with RSA keys and SHA-2 signatures that didn't work with older versions of AsyncSSH or of ssh-agent.

Some versions of UNIX will start ssh-agent automatically for you. You can run "echo $SSH_AUTH_SOCK" to see if that's the case in your environment.

Regarding printing key information, you'll probably need to modify AsyncSSH temporarily to do this. For instance, you could go into asyncssh.auth._ClientPublicKeyAuth and modify the debug log message there in the _start() method from:

        self.logger.debug1('Trying public key auth with %s key',
                           self._keypair.algorithm)

to:

        self.logger.debug1('Trying public key auth with %s key, fingerprint %s',
                           self._keypair.algorithm, self._keypair._key.get_fingerprint())

You can then see if the fingerprints of the keys it is trying to use are the same across the different AsyncSSH versions, and compare the values against fingerprints of the keys you think you are using (via ssh-keygen -l -f <key_file>).

[ProcnerKacper]

Hi, I checked and on the rasberry there is no output after echo $SSH_AUTH_SOCK

I tried to log fingerprint but on version 2.5.0 can't print this because on error SSHAgentKeyPair obejct has no attribiute '_key'

[ronf]

If this is an SSHAgentKeyPair, that suggests that you must have an ssh_agent instance running, and that your keys are loaded into that. Are you passing in agent_path to AsyncSSH? If you're not and SSH_AUTH_SOCK is not set in the environment, I'm not sure how it would be contacting the ssh_agent to get such a keypair.

If you explicitly pass in agent_path=None when opening the connection with AsyncSSH on each of the two releases, does that change anything?

[ProcnerKacper]

Hello, so I found out what was wrong..., someone didn't mention using an old server to connect which has a different private key, so it will not work at all. Rigth now it is working fine while I'm connected to this old server, but for sure need to be done updating the packages. Thank you for for time!

[ronf]

Glad you found the problem! Happy to help...