Feature: Support for tunneling IP traffic with TUN device to OpenSSH

[eytanschulman]

Description

OpenSSH provides the ability to connect to an SSH server and map a tun device on both the client and the server over an SSH connection.

This functionality is implemented in OpenSSH and not the SSH RFC, but is a very useful feature for easily tunneling IP traffic.

From my understanding OpenSSH does the encapsulation and decapsulation of data on both sides, but ultimately does so within an existing SSH connection.

Here is an example from OpenSSH in which a channel and tun are being opened on the client end in order to be tied to the server.

In addition OpenSSH describes their usage of the feature on client/server here.

[ronf]

On a positive note, I found the document for the OpenSSH wire protocol changes needed for this in https://cvsweb.openbsd.org/src/usr.bin/ssh/PROTOCOL?annotate=HEAD. However, the challenge will be finding a way to make this code portable across the various OSes that AsyncSSH supports.

In particular, my primary development environment is macOS and it only supports "utun" by default, which only allows for layer 3 tunneling and has a different set of ioctl()'s than the traditional TUN/TAP. It looks to me like OpenSSH doesn't support "utun", so even getting OpenSSH's support for this working on macOS is non-trivial.

There are TUN/TAP implementations for macOS, but the main one has not been supported since 2015 and was never signed, so it's hard to get that to load on modern versions of macOS. I did find TunnelBlick, which seems to include a signed version of TUN/TAP that I'm going to experiment with to see how far I can get, but this feature may not be straightforward for clients to use on stock macOS unless I go in and add my own "utun" support.

As for Windows, it doesn't have traditional TUN/TAP support from what I can see. There's code out there to try and interface with one of the OpenVPN implementations instead, but I really want to minimize the amount of OS-specific code I add in AsyncSSH if I implement this feature and so I'd want to find some external dependency which takes care of all this.

I was thinking of using Scapy's TUN/TAP support and let it deal with the portability issues. It looks like it could potentially work on Linux/BSD, and possibly on macOS with Tunnelblick's TUN/TAP, but from what I can see Scapy doesn't appear to support "utun" on macOS yet, so it would have the same limitations as OpenSSH there. Also, according to https://scapy.readthedocs.io/en/latest/layers/tuntap.html, it doesn't support that TUN/TAP API on Windows.

[eytanschulman]

On a positive note, I found the document for the OpenSSH wire protocol changes needed for this in https://cvsweb.openbsd.org/src/usr.bin/ssh/PROTOCOL?annotate=HEAD. However, the challenge will be finding a way to make this code portable across the various OSes that AsyncSSH supports.

Great find! This definitely seems like it would limit the number of questions when adding support.

but I really want to minimize the amount of OS-specific code I add in AsyncSSH if I implement this feature and so I'd want to find some external dependency which takes care of all this.

Understandable, if the experimentation with TunnelBlick doesn't go as planned, would it be possible/make sense to add support for opening an OpenSSH Tun channel, and providing the user with the responsibility of opening and connecting the tun/tap device?

[ronf]

If the experimentation with TunnelBlick doesn't go as planned, would it be possible/make sense to add support for opening an OpenSSH Tun channel, and providing the user with the responsibility of opening and connecting the tun/tap device?

Yeah - that's a good thought. I was already thinking of adding something like create_tun() and create_tap() methods on the SSHClientConnection that would parallel the existing create_connection(), and/or something similar with open_tun() and open_tap() paralleling open_connection() that would return an SSHReader and SSHWriter. These functions would let you read and write packets to forward to a remote tun/tap device without the need for AsyncSSH to open a local tun/tap.

If I can get the local tun/tap working from Python without too much complexity, I've been thinking I would also add something like forward_tun() and forward_tap() that would parallel forward_local_port(), forwarding all traffic read from a local tun/tap to a remote one, and vice-versa. This would be like OpenSSH's "-w" option.

I got a chance over the weekend to experiment with this a bit, and it's looking promising so far. I managed to get both the macOS 'utun' device to work, as well as both the tun and tcp devices from TunnelBlick, opening these devices and reading and writing packets, with control over which unit number is opened, and it didn't require much code at all. This seemed to work for both IPv4 and IPv6., as well as raw Ethernet frames when using the tap device. I haven't tried connecting any of this into AsyncSSH yet, but I think I have enough to build functions that will provide either a Python socket or file object that should be accessible with asyncio, and that's really all I need.

I need to confirm that I can get something similar working in Linux, but so far it looks like the interface to Linux's tun/tap is very similar to the one provided on macOS by the TunnelBlick tun/tap devices. So, hopefully the same code should cover both of those cases, and possibly other UNIX OSes.

[ronf]

Another update:

I've started to work on adding the necessary hooks to support TUN & TAP in AsyncSSH. As we discussed, I focused first on the cases where AsyncSSH doesn't directly send or receive data from a local TUN/TAP interface, but instead where it can either tunnel packets to/from a TUN/TAP interface on an SSH server or where it acts as a server which can process packets forwarded from a TUN/TAP interface on an SSH client.

So far, I have connect_tun(), connect_tap(), open_tun(), and open_tap() on SSHClientConnection working as well as tun_requested() and tap_requested() on SSHServer to allow an AsyncSSH server to specify which tunnel requests to accept and how to handle the data arriving on those tunnels. Both client and server side support either callback-based handlers or stream-based (SSHReader/SSHWriter) handlers. There are also new classes for SSHTunTapChannel and SSHTunTapSession which act as transport and protocol objects in asyncio for these packets.

Next up is figuring out the best way to get the code which opens local TUN/TAP devices integrated into asyncio, and implementing TUN/TAP listeners in Python for macOS (both utun and the TunnelBlick version) and Linux. It turns out Linux TUN/TAP is different from the OSXTunTap in TunnelBlick, but it shouldn't be hard to support as a separate variant. Also, looking at the OpenSSH code the code to support OSXTunTap may be similar to the TUN/TAP support on BSD, so I'm hoping if I fall back to that version when not on macOS/Linux, it can be used to support BSD platforms. Perhaps in the future a Windows variant could be supported here as well.

The last piece then will be adding forward_tun() and forward_tap() methods to forward packets from a local TUN/TAP interface to an SSH server, and allowing the tun_requested() and tap_requested() to specify when it wants packets arriving over SSH to be automatically forwarded to a local TUN/TAP interface when acting as a server.

[ronf]

More progress:

Today, I got forward_tun() on AsyncSSH to successfully forward traffic from a local utun interface on macOS to a remote tun interface on a Linux system running OpenSSH as the server, and I also got a Linux OpenSSH client to successfully tunnel traffic to a remote utun interface on macOS using AsyncSSH as a server, by returning True in a tun_requested() function.

I tried getting this to work with OSXTunTap, but I've run into a snag there. It appears those devices don't work correctly with Python asyncio. You can read them just fine (in blocking or non-blocking mode), but when you try to set up a callback to fire when data is available to read, the system returns Invalid Argument. I'm guessing the devices don't support kqueue, but even trying other available event loop types didn't help. I'll probably need to do something like spin off a thread to do blocking I/O and make that available via an asyncio Queue to work around this.

I've still got some work to do to allow AsyncSSH on Linux to open TUN/TAP devices, but I should have everything I need for that and that shouldn't take too long.

Other than that, I still have some code cleanup to do and need to write unit tests, but it's getting there!

[ronf]

Sorry for the slow response - I had some other stuff come up last weekend and didn't spend much time on this. However, today I was able to get TunTapOSX working in both TUN and TAP mode, working around the snag I ran into with not being able to use loop.add_reader() on it. Reads are now done in a blocking manner in a separate thread and then the callbacks are triggered via loop.call_soon_threadsafe(), and the thread is started/stopped as needed from within pause_reading() and resume_reading(). I didn't go with the asyncio.Queue approach, as that isn't actually thread-safe.

In addition to the above, I also have a SSHTunTapStreamSession class which preserves packet boundaries when it is used with SSHReader/SSHWriter. It also exposes an async iterator which can be used to loop over packets rather than lines. So, you can do something like:

    async with asyncssh.connect(host, username='root') as conn:
        reader, writer = await conn.open_tun()

        async for packet in reader:
            # Process packets received from the remote host here, using writer to send packets back

You can do something similar on the server side:

async def MySSHTunTapHandler(reader, writer):
    async for packet in reader:
        # Process packets received from the client here, using writer to send packets back

class MySSHServer(asyncssh.SSHServer):
    def tun_requested(self, unit):
        return MySSHTunTapHandler

async def start_server():
    await asyncssh.create_server(MySSHServer, '', 8022,
                                 server_host_keys=['ssh_host_key'],
                                 authorized_client_keys='authorized_keys')

I also have a placeholder for the Linux TUN/TAP support and know what I need to do there. I'll fill that in once I have everything working completely on macOS.

[ronf]

It took a bit longer than I expected, but a first cut of TUN/TAP support is now available in the "develop" branch as commit for e4504e1. As it turned out, one of the hardest parts was figuring out how to stub out the OS drivers for the unit testing, so it could be run without requiring root.

The code may still need some work, particularly around things like cleaning up properly in all cases, but I think there's enough here to let others try it out on an experimental basis. Any feedback and problem reports are welcome!