search

threefoldtech / tfgrid-sdk-ts

Apache License 2.0
4 stars 8 forks source link

Wireguard Access with more than 1 VM #1396

Open Mik-TF opened 10 months ago

Mik-TF commented 10 months ago

Wireguard Access with more than 1 VM

By default, the wireguard IP address given to a VM is always the same. For example, on the new.playground.dev.grid.tf, it is 10.20.4.2.

If I deploy 2 VMs, they will both have the same wireguard address: 10.20.4.2.

The current situation

If I set vm1 with wg1.conf and vm2 with wg2.conf, I cannot SSH in both VMS at the same time.

To ssh into vm1, I do

If then I do

I get the error:

[#] ip link add wg2 type wireguard
[#] wg setconf wg2 /dev/fd/63
[#] ip -4 address add 100.64.20.3/32 dev wg2
[#] ip link set mtu 1270 up dev wg2
[#] ip -4 route add 100.64.20.0/32 dev wg2
RTNETLINK answers: File exists
[#] ip link delete dev wg2

So to connect to vm2, I must do

This will lead me to vm2.

Potential fix

Could there be a way to be able to SSH into both VMs with both wireguard access set as up? (wg-quick up wg1, wg-quick up wg2).

From @scottyeager : it would be possible to run two tunnels simultaneously, but they would need to use different address ranges, like 10.20.4.0/16 and 10.21.4.0/16, for example.

A solution would thus be to: add an input field to set the IP range. Of course, this option would be optional, and by default would give only one tunnel (e.g. 10.20.4.0/16)

xmonader commented 10 months ago

You need to bring the interface down indeed. Specifying the ranges is quite advanced for the user and we have our own assumptions for the internal IPs (and should be comfortable to provide so via scripting). Another option could be allowing vm2 to be part of the network of vm1, we used to expose that option for the user to set the network from before.

In zos4 we will be working with project concept were workloads live within an explicit project (we do so implicitly in the background, so the user won't be bothered)

scottyeager commented 10 months ago

Definitely users should be able to add new VMs to existing networks, but creating multiple networks their own IP ranges is also an important use case. Isolating VMs that don't need to communicate with each other will be more secure.

Specifying the ranges is quite advanced for the user and we have our own assumptions for the internal IPs

I'm not sure what's meant by the second part. We can keep it very simple though. Valid network ranges are 10.X.0.0/16, so we just need to allow the user to specify X and nothing more.

xmonader commented 10 months ago

but creating multiple networks their own IP ranges is also an important use case. Isolating VMs that don't need to communicate with each other will be more secure.

I agree with that, I don't think that's common enough though to be exposed for the average user, and it comes with a cost, and if the user is going with that I believe he is better off scripting that is with full control on everything .

I'm not sure what's meant by the second part. We can keep it very simple though. Valid network ranges are 10.X.0.0/16, so we just need to allow the user to specify X and nothing more.

There are assumptions about the order of hosts in the wireguard network. If we expose such option, probably we should allow the user to specify the host ID on the wireguard network as well, but that won't happen in case of kubernetes for example.

Another note: these ranges will still conflict too on testnet.

Either way, exposed or not the user will still have to keep track what is on what range and what wireguard interfaces are conflicting and definitely if they use other networks like testnet or devnet, they will need to track that too.

Let me know what you guys think.

scottyeager commented 10 months ago

So definitely this can be hidden under some kind of advanced menu.

Combining both ideas the flow could look like this:

  1. User expands the Network section and sees a new drop down box called, for example, Private Network
  2. In the case that there are existing networks from other deployments, those would be populated, along with a default option for Add to new network
  3. When the new network option is selected, another expandable Advanced section is available below for choosing an IP range

Personally I think it's okay to keep the assignment of actual addresses automated (just assigned sequentially in the order the VMs are added).

We can compare to what Hetzner does for some inspiration. Private networking with user selectable ranges is exposed through the GUI. They keep things simple by making this a separate and option feature, whereas network membership is mandatory on the Grid, but I think it demonstrates that this sort of thing shouldn't necessarily be relegated to other tools/scripts.

xmonader commented 10 months ago

I like your suggestion, that's how it started actually, but the network option was too complex for the users specially when we were promoting yggdrasil.

scottyeager commented 10 months ago

Cool. I think it makes sense to reconsider this now, as many users are now preferring WireGuard to Yggdrasil and I don't count on that changing with Mycelium especially until it becomes well supported on Windows.

With some finesse hopefully we can make the functionality available to users who want it without confusing those who don't.