Idea / feature request: Integrate with Nebula or ZeroTier to protect from DDoS

[qdrddr]

If you wish to access Console, Router, or Controller via the internet, you must open ports to the public internet and assign a public DNS record. This opens Ziti services to DDoS.

However, there are VPNs that can open connections between a host and server nodes (and overwrite DNS similarly to Ziti) without opening a single firewall port for the VPN connection to work, such as ZeroTier One (zerotier.com) and Nebula (https://github.com/slackhq/nebula). Using tools like ZeroTier and Nebula, you have only your VPN clients able to access Ziti Console, Router, and Controller ports. Those listed tools allow to keep open and access only ports within VPN that are needed for Console, Router, and Controller to operate, allowing VPN clients to access only Ziti services and prohibit clients from communicating with each other.

Thus significantly reducing the chances of a DDoS attack on the Ziti services. Even if the internal user tries a DDoS, such a user will be limited to several hosts, limiting the scale of DDoS and removing the possibility of unleashing zombi network attacks. Since the attacker must be authenticated via the VPN, admins can quickly identify abnormal activity from VPN clients. Such malicious clients can be banned or suspended from the VPN network.

Please consider integrating or making smooth configurations and provide a blueprint with best practices of how to configure these VPN services in conjunction with Ziti to reduce exposure to malicious activities directed toward the Ziti Console, Router, or Controller.

[mikegorman-nf]

There are other ways to manage potential DDoS attacks besides applying another networking technology. Edge Routers can be ephemeral, coming and going and since it is all API driven, one can use various tools to control the growth/degrowth. The ZAC can be protected with OpenZiti itself, if you like, leaving only the Controller port. With distributed controllers, this will be somewhat like the Edge Routers, able to add or delete Controllers under pressure.

There has to be an open port somewhere. For example, Nebula needs an open port on the Lighthouse node. With dynamic IP environments, something has to be able to talk from anywhere to some node to begin the process of connecting. With this in mind, OpenZiti has looked for other ways to mitigate potential attacks, like those above. We have run multiple table top exercises as well, to examine the ways that one could respond to such an attack. Once the HA controller features are complete, we will revisit this and devise new strategies based on what we have available.

I fully understand the concern, and share it, but rather than applying an additional network layer as you proposed, we have worked to address the issue with other mitigating controls.

[qdrddr]

I understand the need to keep open ports in the current OpenZiti architecture so that components such as the Controller, Router, and Tunneler need to be able to reach the Routers and the Controllers. To bring to your attention, there are techniques that can help hide all ports behind a firewall by completely hiding them using the UDP hole-punching technique with ZeroTier One and Nebula.

Please consider applying this UDP hole-punching technique to establish a connection between peers on the Internet, such as Controller and Routers that are reachable by Tunneler, without opening ports in the firewall.

The easiest way would be in the current stage by using existing products like ZeroTier One or Nebula, a relatively straightforward integration.

Alternatively, you can implement this UDP hole-punching technique directly into the Controller, Router, and Tunneler to communicate without opened ports in the first place, which may take time to implement if you decide to do so.

FYI, though I'm saying "No ports opened," the supernode still needs an opened port on the public Internet for ZeroTier and Nebula that allows peers to find each other and establish direct peer-to-peer communication using the UDP hole-punching technique, so the issue remains. But.

The supernode on the image below is called Rendezvous Server: https://camo.githubusercontent.com/53995a59d02716a522a0285e0a958fe66d4869829a5361336bff2e9390d0c944/687474703a2f2f692e696d6775722e636f6d2f645a4e456870772e706e67

Benefits of UDP hole-punching technique

UDP hole-punching in products such as ZeroTier and Nebula that may be useful for OpenZiti:

1. The supernode is a separate stand-alone service that does not influence sensitive services, such as the controller/router that does the main work and handles and enforces zero-trust, routing, and handle management. So, if the supernode is hacked, the other components will not be compromised.
2. There are many supernodes across the Internet, so it is hard to DDoS all of them simultaneously
3. Even if all supernodes are DDoSed and down, the peers that already established connection do not care about the supernode and continue working without an issue
4. Only one port is opened, and you must be concerned about being open on the public Internet to one service that is not viral and acceptable if not working for hours or days.

So please reconsider if you can utilize the UDP hole-punching technique similarly to how it is used with ZeroTier or Nebula in one way or another.

I propose implementing UDP hole-punching on the Controller and Router and Tunneler so they can see each other without opening the Controller and Router ports to the Internet. And since the Tunneler can now see the Controller and Router, you can also generate JWT locally on the Tunneller/Client.

[mikegorman-nf]

We had a discussion on this exact thought with another person, and it had some interesting ideas brought up. The first is that the enrollment process for adding an endpoint to a network can't be done this way, securely. The endpoint needs to verify the identity of the controller as well as the controller of the endpoint. We did postulate that we could separate the enrollment process away from the actual controller so that it would be the only service affected by a DDoS attack, allowing the rest of the network to function normally, which is interesting. The concerns with something like hole punching are the next set of problems. In a DDoS attack against the supernode, new connections are still impossible. Any connection that fails for whatever reason is gone, as the supernodes can't be used to create a new one, so the network decays across the timeline of the attack, for instance. OpenZiti has a lot of other options for how to mitigate such an attack, particularly once the multiple controller option is in production, which will be quite soon. Edge routers themselves are cattle, not pets, so they can be added to the network at any time, in any place, and the messaging about the new routers is distributed over the control channels. Having multiple options for controllers will make this stronger if particular controllers were attacked. There are also operational responses we've gamed out using various features to move controllers as well.

I'm not saying we wouldn't consider additional ideas, but any changes of this magnitude need a solid reason. We have methods of mitigating DDoS attacks, which is the main point here, and more are coming. So it comes down to a risk management problem; what is the threat, what is the risk, and what do the existing controls provide for mitigation? This is a conversation I'd love to have. I would suggest our Discourse server would be a better place, as it is likely to have a larger audience.

[qdrddr]

I like to repeat my own proverb: "If you have an option to do something, it means you must do that." or "If someone gives you an option, now you are stuck choosing between options and supporting them." In other words, "options" are not always a good thing :)

So, if we can enable/disable routers to protect from DDoS, we must do that. Emphasis on additional doing something manually. So it would be nice to have an easy way of doing that if users now must do that to protect from DDoS. And hopefully, you'll not forget people who self-host ;) @mikegorman-nf

[qdrddr]

Also, please consider the capability of hiding this new separate Enrollment behind an async reverse proxy such as Cloudflare. @mikegorman-nf

[dovholuknf]

As part of issue grooming, we decided to close this issue for now. This was a great discussion, thank you. There are numerous different ways to implement and we will end up choosing one in the future. It's a larger effort and will require more discussion.