ParallelSSH / parallel-ssh

Asynchronous parallel SSH client library.
https://parallel-ssh.org
GNU Lesser General Public License v2.1
1.2k stars 149 forks source link

Dangling channel reference in TunnelServer #304

Closed maxballenger closed 3 years ago

maxballenger commented 3 years ago

For general questions please use the mail group.

Describe the bug TunnelServer method _wait_send_receive_lets() calls self._client.close_channel(). The client reference that ends up here is a _proxy_client attribute of an SSHClient object connecting to a remote host through a proxy. When disconnect() is called on this SSHClient object, its proxy_client is also disconnected, which eventually triggers the return of the joinall() call inside _wait_send_receive_lets(). By the time this happens, the _proxy_client's session has likely been freed, which also freed the channel, so the call to close_channel() causes a double free.

This can take a while to happen because LocalForwarder only cleans up the servers once every 60 seconds and I think generational garbage collection must also be triggered in order for the segfault to show up.

I believe this can be fixed by moving the call to _proxy_client.disconnect() into _wait_send_receive_lets().

To Reproduce

Steps to reproduce the behavior:

  1. Run the code below
  2. You should see a segfault
import unittest
import time
from pssh.clients.native.single import SSHClient
from pssh.clients.native.tunnel import FORWARDER
import gc

PROXY_IP = '10.1.15.1'
HOST_IP = '172.16.0.101'
PROXY_USER = 'user'
HOST_USER = 'user'
PROXY_KEY = "/path/to/proxy/key"
HOST_KEY = "/path/to/host/key"

GC_DELAY = 1
NUMBER_RECONNECTS = 20

class TestProxiedHost(unittest.TestCase):
    @classmethod
    def setUpClass(cls):
        cls.phost = ProxiedHost(PROXY_IP, HOST_IP, PROXY_USER, HOST_USER, PROXY_KEY, HOST_KEY)

    def test_many_reconnects(self):
        for i in range(NUMBER_RECONNECTS):
            print(f"Running cycle {i+1}/{NUMBER_RECONNECTS}")
            print("Closing ssh connection")
            self.phost.disconnect()
            time.sleep(GC_DELAY)
            gc.collect()    # This may speed up the segfault
            self.phost.connect()
            print("Connection re-established")

class ProxiedHost:
    def __init__(self, ip_proxy, ip_host, user_proxy, user_host, key_proxy, key_host):
        self.ip_proxy = ip_proxy
        self.ip_host = ip_host
        self.user_proxy = user_proxy
        self.user_host = user_host
        self.key_proxy = key_proxy
        self.key_host = key_host

        self.connect()

    def connect(self):
        kwargs = {
            'host': str(self.ip_host),
            'user': self.user_host,
            'pkey': self.key_host,
            'proxy_host': str(self.ip_proxy),
            'proxy_user': self.user_proxy,
            'proxy_pkey': self.key_proxy
        }
        self.ssh = SSHClient(**kwargs)

    def disconnect(self):
        self.ssh.disconnect()
        FORWARDER._cleanup_servers()    # This speeds up the segfault

if __name__ == '__main__':
    unittest.main()

Expected behavior I expect to be able to connect/disconnect from an SSHClient at will without segfaults

Actual behaviour segfault

Additional information Include version of ssh2-python and any other relevant information.

ssh2-python==0.26.0
parallel-ssh==2.5.4