Using unsafe routes with Windows or macOS exit node

[clarkmcc]

I read and tested this blog post but I'm in a bit of a different situation. I want my exit nodes to be Windows and macOS machines, rather than a Linux machine. The blog post required some special configuration of iptables on the Linux exit node. Is that same configuration required when using a Windows and macOS exit nodes, or is this not possible with nebula?

[brad-defined]

Nebula's unsafe routes feature can get the packets from clients to the unsafe_routes Nebula node. However, it depends on the kernel to perform NAT and get the packets to their final destination - which also ensures return traffic gets back to the unsafe_routes node so that they can be forwarded along to the originating Nebula peer.

Windows has a feature called Internet Connection Sharing that might do what you need, but I'm not sure, and haven't tested it.

I couldn't find anything on MacOS.

The feature was designed with Linux in mind.

If you share more about your goals, we might be able to find another way to resolve your issue. Why do you want to use Windows and MacOS clients as unsafe_routes nodes?

[clarkmcc]

@brad-defined thanks for the response. Our goal is to use Nebula to temporarily expose a route to specific internal non-nebula devices periodically throughout the day so that we can remotely collect data from them using a wide variety of networking protocols (SNMP, HTTP, etc). Our agents are cross-platform and most of them are running on Windows devices. In order for Nebula to help us accomplish this goal, it would need to work on Windows.

Even though there is no current support for this today, are there plans to support it in the future? I know that Tailscale has managed to make this work transparently to the user and the client source code is open-source...

[brad-defined]

The way that Tailscale delivers this is by importing a userland implementation of the TCP/IP stack. They use this to terminate the TCP connection at the Tailscale node, and then establish a local TCP connection from the Tailscale-installed node to the ultimate destination IP:port, and proxying the connection along.

A port forwarder that terminates local connections, establishes upstream connections to the desired IP:port, and proxies the two is basically what Tailscale's built.

One way you can try this today by setting up an SSH server on the Windows host that has network access to the systems you want to collect data from, and configure that SSH server to listen on the Nebula IP of that windows host, and configure it to permit port forwarding. (This would require an SSH client to set up the port forwarding session.)

Other solutions would be to run netcat, or socat, or build a port forwarding proxy into your agent.

Nebula has a built-in SSH server, but it does not support port forwarding today (it's mainly used for status and debug info.) If it did support port forwarding, would that satisfy your use case?

[clarkmcc]

Is there a reason that Nebula does not do something similar? I.e. is that userland implementation is too slow for Nebula's primary goals, or has it just not been implemented yet?

Nebula has a built-in SSH server, but it does not support port forwarding today (it's mainly used for status and debug info.) If it did support port forwarding, would that satisfy your use case?

Possibly? I haven't done much port-forwarding over SSH so I can't say for sure. If it allows our remote data collectors to access a specific non-nebula IP address via a nebula node, over a specific port, then yes (see diagram). Building a forwarding proxy into our agent would be no problem at all. A few questions if we chose to go that route:

1. How would the remote collectors indicate that they want to connect to a non-nebula IP address over a specific port, just use unsafe_routes like normal?
2. How would the nebula peer know to pass those packets to our own forwarding proxy (which would be running on a specific port on the nebula peer)?
3. How would our forwarding proxy extract the destination IP address and port?

[davidflowerday]

@clarkmcc I happened across this issue and the timing is interesting. I started messing around with this the other day for other reasons and I have a basic integration of the Go userspace network stack with Nebula. It supports TCP proxying right now. I have not implemented UDP yet (so SNMP wouldn't work), though I want to look at adding that next and I don't expect it to be difficult.

My changes are here: https://github.com/davidflowerday/nebula/tree/userspace-network

I've only put in a few hours on it so far but I have tested it on Windows, Mac, and Linux and it works fine on all of them. You can access any TCP IP:port combo through the host running it as long as the host is configured properly with unsafe_routes. Currently you just set tun.netstack in your config.yaml to enable it:

tun:
  disabled: false
  netstack: true

[clarkmcc]

@davidflowerday this is very interesting, thank you for your work on this and for sharing! I just glanced over the gvisor library and I see that it does provide some utilities for implementing UDP forwarding. I'll do some testing and possibly dive into taking a stab at the UDP forwarding implementation later this week.

[brad-defined]

Is there a reason that Nebula does not do something similar? I.e. is that userland implementation is too slow for Nebula's primary goals, or has it just not been implemented yet?

@clarkmcc I think pulling the TCP/IP stack into the Nebula process would be sub-optimal performance-wise, but would also enable some capabilities like the one you're asking for. It would also, for example, also enable Nebula to be embedded directly into programs and route traffic without creating a TUN device on the host. So, it's possible / likely that this will happen, but just hasn't happened yet.

In your use case, IIUC, you'll have a couple of problems to solve. Your agents are going to be installed on machines in Client networks. It's possible that some of the Client's networks will overlap in their IP ranges. You'll not only need a way for the terminating Nebula node to pass and receive traffic on the LAN - you'll also need a way for your local Nebula node to determine which of your Client's terminating Nebula nodes to use.

Meaning, it's possible that you'll have two clients, each of whom are using a 10.10/16 LAN network, and you have an agent in each network. Your local Nebula Remote Controller node will have to decide which terminal Nebula node to route traffic to 10.10.100.3 to - client A's network, or client B's network.

With the SSH port forwarding solution, you would set up a local listening port that would map to some arbitrary Client IP and port. Then, any connections made to the local port would be tunneled through to the Client IP and port. The local port would be set up as something like: ssh -L 127.0.0.1:<local port>:<target client IP>:<target client port> <nebula IP of client Nebula host>

The tunnel command includes which destination IP you want to hit (10.10.100.3, say), which port, and which Nebula node to route the traffic through. Using SSH would be limited to TCP forwarded connections only, I believe.

With the Unsafe Routes solution, you'll need a way to configure the local Nebula node to know which terminal Nebula node to use when the local Nebula node receives traffic for 10.10.100.3. (This would likely look like some reloadable config options, with the caveat that the entire local Nebula would only be connected to one Client network at a time.)

[brad-defined]

@davidflowerday that looks awesome, thanks for sharing!

[davidflowerday]

@clarkmcc I added some basic UDP support as well. I'm curious to know if that implementation would work now for the use case you have in mind.

[clarkmcc]

@davidflowerday @brad-defined thank you for bringing up those issues (and for the UDP support), to summarize:

• Collectors need to be able to collect to specific devices within an agent's network even when multiple isolated agent networks have overlapping ranges.
• Nebula currently creates a TUN device but I probably don't want to touch network interfaces on the agent side. This brings up the point that ideally, connections would only be initiated from connector to agent but not the other way around.

I've been running through POCs and I'm wondering if Nebula is the tool for the job, perhaps either of you can speak to that. Here's what I have in mind. Setting aside security concerns, can we just give the collector a network interface that behaves as if it's a network interface running in the customer network? What this translates to is the collector having a userspace network stack bound to a TUN device, where the network stack passes all packets over a pre-existing connection which reaches the agent's userspace network stack where we implement packet forwarders (i.e. no TUN device on the agent side).

Based on my testing so far, I could use something like libp2p to act as the "transport" which gives a lot of the same NAT traversal features that Nebula provides. The only reason I suggest libp2p is if Nebula does not appear to fit this use case.

[brad-defined]

@clarkmcc Without something like @davidflowerday 's changes, Nebula must create a TUN device to transport traffic

If you're in control of the Collector, you could solve your agent connectivity by having the Agent initiate the connection to the Collector, and then using that connection to transfer whatever traffic you require. WebSockets, long poll, or SSH would all work great. Having the Agent initiate a connection to a public Collector IP/port would solve all NAT and basic Firewall issues.

Once the connection is established, it can sit idle (with keepalives or similar to maintain NAT/Firewall state) until the Collector decides it wants to talk to the agent. The Collector can use the Agent-established TCP connection to send requests, which the Agent can then act on.

[clarkmcc]

@brad-defined sadly, we're not guaranteed to be in control of the collector and it's also not guaranteed to be publically accessible, hence why I've been looking into the Nebula/libp2p options. Thanks for your feedback on this so far, it's been extremely helpful.

[jasikpark]

ig https://github.com/slackhq/nebula/issues/559 is related in that they're both discussing user-space TCP stacks?

[davidflowerday]

Collectors need to be able to collect to specific devices within an agent's network even when multiple isolated agent networks have overlapping ranges.

That's a bit of a problem but I think there's at least 2 ways to solve it.

1. Set up each agent device with Nebula with the appropriate subnet specified in each agent's certificate. Then on the remote collector side, you could modify Nebula's configuration and restart Nebula, so that the remote collector would only have a single unsafe_route at a time to whichever agent you are trying to contact. I'm pretty sure the overlapping ranges on each agent wouldn't be an issue as long as the Nebula node (remote collector) that you're initiating the connection from only knows about 1 of them at a time.
2. It wouldn't be too hard to take what I've done for TCP/UDP userspace proxying and add some IP address rewriting on top of it such that from the Nebula network's perspective, each device would be on a unique subnet. In the Nebula code running on the agent, when it examines the remote IP address, it would modify the IP to be the proper target before initiating the connection. This would basically be a little mini-NAT. I'm already doing this somewhat when the target IP is the Nebula address, rewriting it to be localhost. You can see that here, which is about where you'd want to make such a change: https://github.com/davidflowerday/nebula/blob/935072f133cbe56601090ad85c35fcf668892c53/overlay/netstack_device.go#L302

I think Nebula with my changes could do what you want, though I do have to question if you already have an agent that you control running on the agent devices, why not just add the functionality to do the collection in the agent and then have the agent send that back to your remote collector over HTTPS or something like that? Nebula may be overkill for your goals unless there's some other reason to have Nebula there (e.g. SSH/Remote Desktop access/etc.).

Nebula currently creates a TUN device but I probably don't want to touch network interfaces on the agent side. This brings up the point that ideally, connections would only be initiated from connector to agent but not the other way around.

Just to be clear - the changes that I made in the branch I posted allow you to use Nebula without a TUN device at all. Instead of using a TUN device, the Nebula network connections are terminated into the gVisor userspace TCP/IP stack and then from there regular TCP/UDP connections are established to the target. My real goal here is to be able to run Nebula inside Docker without the need for any special permissions (which means no TUN devices). My next step is to add SOCKS5 proxy support so that applications running inside a container can establish outbound connections to the rest of the Nebula network via the SOCKS proxy. I have that all written now but I need to track down one small bug before it's ready.

I haven't used Tailscale myself but my understanding is that this is the same way that Tailscale works for TUN-less operation, both as far as establishing outbound connections and the SOCKS proxy.