Hyper-v: reconfigure the default interface to use a custom vSwitch

[abeermmukhemir]

This is more of queries about the state and plans to support this feature

What are you trying to do? I need to be able to configure the VM's default interface with static IP, currently, this is impossible since the default interface is always connected with the hyper-v default switch. My use case requires setting static IP to the default interface which cannot be done with the --network option.

I saw some open issues that generally request the same thing but the reasons differ. I really like multipass on Windows, it simply makes things much easier, however, the way it's tightly coupled to the hyper-v default switch is a blocker.

Q: Is there a plan to support this? if so, is there a timeline? I saw in the issues threads that there are some blockers like "relying on mshome.net to figure out the instance IP", what are all the blockers for this?

What's your proposed solution? Q: if the instance interface was to be configured with static IP from the beginning i.e. with cloud-init (eg. if https://github.com/canonical/multipass/issues/1040 is implemented), would it help unblock some of the requirements?

[ricab]

Hi @abeermmukhemir, thank you for your input, but I am afraid that this isn't really in our plans FTTB... That said, I would like to understand the need a little better. Why is it that you would need a static IP on the default interface specifically? Why wouldn't a static IP on an extra interface work for you? And why is the default switch a blocker?

[abeermmukhemir]

Hi @ricab, thanks for the quick response

Why is it that you would need a static IP on the default interface specifically? Why wouldn't a static IP on an extra interface work for you?

Basically, for hosting stuff that require static IP on the default interface, like a microstack or microk8s cluster, these take the IP of default interface and any changes in the IP will break their services. Now the problem with the Hyper-v default switch is that sometimes will even change the subnet, so workarounds like - after launch - configuring additional IP for the default interface (with netplan or whatever) wouldn't work here.

[ricab]

I understand that you'd need to be able to reach the instance with a static IP. What I don't get is why you'd need that static IP to be on the default interface. I mean, from the outside, all network interfaces should be indistinguishable, no? They are just an endpoint with an IP. And you can configure routes to your liking inside the instance. So why wouldn't a recipe like this work for your purposes?

Also, is there any way you could configure microstack or microk8s to use <instance-name>.mshome.net and rely on DNS instead of a static IP?

I may well be missing something, just trying to understand the use case better.

[abeermmukhemir]

I was talking in general so this thread wouldn't turn into a discussion about a specific behavior/issue in microstack/microk8s, but let me give more details:

I understand that you'd need to be able to reach the instance with a static IP. What I don't get is why you'd need that static IP to be on the default interface. I mean, from the outside, all network interfaces should be indistinguishable, no? They are just an endpoint with an IP

The problem isn't about reaching the instance from the outside, and you're right, using <instance-neme>.mshome.net and/or configuring static IP on extra interface are enough for reaching from the outside.

The requirement for a static IP on the default interface comes from within the instance, I'll explain the issue I had with microstack as an example in the hope it gives more clarity: A bit of context: with recent versions of microstack, the bootstrapping of the cluster is done through project sunbeam, which is supposed to automatically install all the required snaps (microk8s, openstack-hypervisor,, etc) and configure them, in addition to configuring juju as an operator for the whole thing, now, for simplicity reasons the things that can be configured during the bootstrap are very limited and there are assumptions and reliance on default values, and once the cluster is bootstraped there's no way (at least no documented way) to manually reconfigure those default values without breaking the cluster or some of its internal components (nova, neutron, etc). Now the specific part that relates to the default interface requirement is that during the bootstrap process the openstack-hypervisor snap is configured to take the IP of the interface associated with the default route which in turn translates to the default interface in a multipass instance (code for that), and even though the snap provides a way to configure this manually, I haven't been able to successfully reconfigure it without breaking the overall sunbeam cluster.

That said, I realize this is also a limitation from microstack side as well, however, this is only one occurrence of software relying on the default interface specifically and surely there other softwares that do the same thing.

Also, is there any way you could configure microstack or microk8s to use .mshome.net and rely on DNS instead of a static IP?

I haven't tested this so I can't confirm, but I think this would be useful when it comes to communication between nodes in multi-node cluster, however, I don't think it will help in the specific problem I described above.

[ricab]

Hi @abeermmukhemir, thank you for the detailed explanation. It is very useful to understand what people are looking for and why.

So, for this case, I looked at the code you linked and then the code of the netifaces being used and indeed, they select the default gateway just by whatever gets selected by default, i.e., the default route. So I tried a little experiment and launched an instance with an extra --network and then changed the configuration to convince netifaces to identify the second interface as the default:

ubuntu@cherished-krill:~$ python3 -mvenv env
ubuntu@cherished-krill:~$ cd env
ubuntu@cherished-krill:~/env$ ./bin/pip install netifaces
Collecting netifaces
  Using cached netifaces-0.11.0-cp312-cp312-linux_x86_64.whl
Installing collected packages: netifaces
Successfully installed netifaces-0.11.0
ubuntu@cherished-krill:~/env$ ./bin/python
Python 3.12.3 (main, Apr 10 2024, 05:33:47) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from netifaces import gateways
>>> gateways()
{'default': {2: ('10.130.108.1', 'enp5s0'), 10: ('fe80::216:3eff:fe47:e3cc', 'enp5s0')}, 2: [('10.130.108.1', 'enp5s0', True), ('192.168.1.254', 'enp6s0', False)], 10: [('fe80::216:3eff:fe47:e3cc', 'enp5s0', True), ('fe80::da78:7fff:fe05:f31f', 'enp6s0', False)]}
>>>
ubuntu@cherished-krill:~/env$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.130.108.1    0.0.0.0         UG    100    0        0 enp5s0
0.0.0.0         192.168.1.254   0.0.0.0         UG    200    0        0 enp6s0
10.130.108.0    0.0.0.0         255.255.255.0   U     100    0        0 enp5s0
10.130.108.1    0.0.0.0         255.255.255.255 UH    100    0        0 enp5s0
192.168.1.0     0.0.0.0         255.255.255.0   U     200    0        0 enp6s0
192.168.1.254   0.0.0.0         255.255.255.255 UH    200    0        0 enp6s0
ubuntu@cherished-krill:~/env$ sudo -i
root@cherished-krill:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: enp5s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 52:54:00:27:60:3f brd ff:ff:ff:ff:ff:ff
    inet 10.130.108.76/24 metric 100 brd 10.130.108.255 scope global dynamic enp5s0
       valid_lft 3574sec preferred_lft 3574sec
    inet6 fd42:209b:fcb9:33ed:5054:ff:fe27:603f/64 scope global mngtmpaddr noprefixroute
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe27:603f/64 scope link
       valid_lft forever preferred_lft forever
3: enp6s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 52:54:00:21:27:d6 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.118/24 metric 200 brd 192.168.1.255 scope global dynamic enp6s0
       valid_lft 86374sec preferred_lft 86374sec
    inet6 2001:8a0:ecd8:4100:5054:ff:fe21:27d6/64 scope global dynamic mngtmpaddr noprefixroute
       valid_lft 89976sec preferred_lft 89976sec
    inet6 fe80::5054:ff:fe21:27d6/64 scope link
       valid_lft forever preferred_lft forever
root@cherished-krill:~#
root@cherished-krill:~# cat << EOF > /etc/netplan/10-custom.yaml
> network:
>   ethernets:
>     enp6s0:                  # name of your second interface
>       dhcp4: true
>       dhcp4-overrides:
>         route-metric: 50     # make this the default gateway
>   version: 2
> EOF
root@cherished-krill:~# netplan apply

** (generate:13589): WARNING **: 11:22:41.959: Permissions for /etc/netplan/10-custom.yaml are too open. Netplan configuration should NOT be accessible by others.

** (process:13588): WARNING **: 11:22:42.156: Permissions for /etc/netplan/10-custom.yaml are too open. Netplan configuration should NOT be accessible by others.

** (process:13588): WARNING **: 11:22:42.208: Permissions for /etc/netplan/10-custom.yaml are too open. Netplan configuration should NOT be accessible by others.
root@cherished-krill:~#
root@cherished-krill:~# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         GEN8            0.0.0.0         UG    50     0        0 enp6s0
default         _gateway.lxd    0.0.0.0         UG    100    0        0 enp5s0
10.130.108.0    0.0.0.0         255.255.255.0   U     100    0        0 enp5s0
_gateway.lxd    0.0.0.0         255.255.255.255 UH    100    0        0 enp5s0
192.168.1.0     0.0.0.0         255.255.255.0   U     50     0        0 enp6s0
GEN8            0.0.0.0         255.255.255.255 UH    50     0        0 enp6s0
root@cherished-krill:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: enp5s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 52:54:00:27:60:3f brd ff:ff:ff:ff:ff:ff
    inet 10.130.108.76/24 metric 100 brd 10.130.108.255 scope global dynamic enp5s0
       valid_lft 3586sec preferred_lft 3586sec
    inet6 fd42:209b:fcb9:33ed:5054:ff:fe27:603f/64 scope global mngtmpaddr noprefixroute
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe27:603f/64 scope link
       valid_lft forever preferred_lft forever
3: enp6s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 52:54:00:21:27:d6 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.118/24 metric 50 brd 192.168.1.255 scope global dynamic enp6s0
       valid_lft 86386sec preferred_lft 86386sec
    inet6 2001:8a0:ecd8:4100:5054:ff:fe21:27d6/64 scope global dynamic mngtmpaddr noprefixroute
       valid_lft 89987sec preferred_lft 89987sec
    inet6 fe80::5054:ff:fe21:27d6/64 scope link
       valid_lft forever preferred_lft forever
root@cherished-krill:~#
logout
ubuntu@cherished-krill:~/env$ ./bin/python
Python 3.12.3 (main, Apr 10 2024, 05:33:47) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from netifaces import gateways
>>> gateways()['default']
{2: ('192.168.1.254', 'enp6s0'), 10: ('fe80::da78:7fff:fe05:f31f', 'enp6s0')}
>>>

If you configure the second interface with a static IP, that should work for your case, hopefully :crossed_fingers: Let me know how it goes.

[ricab]

BTW, for your case, instead of dhcp4_overrides in the netplan config, you would use something like:

      routes:
        - to: 0.0.0.0/0
          via: 10.13.31.1
          metric: 50

Or you could apply a higher metric to the default interface with dhcp4_overrides.

[abeermmukhemir]

Hi @ricab I got a chance to test setting up microstack on a multipass instance configured with the suggestion above and I can see the traffic going through the extra interface instead of the default one, which might solve the issue discussed above regarding the "openstack-hypervisor" component in the cluster bootstrap process. However, apparently, other components/parts of the bootstrap process still use and rely on the IP of the default interface, for example: Juju API addresses are collected from all the interfaces, and then (for some reason) the connection is done using the IP of the default interface

ubuntu@microstack:~$ juju --debug --verbose controllers --refresh
21:44:27 INFO  juju.cmd supercommand.go:56 running juju [3.5.2 cbe7810e21f7395219e6e39b34106428ed4013e4 gc go1.21.11]
21:44:27 DEBUG juju.cmd supercommand.go:57   args: []string{"/snap/juju/27745/bin/juju", "--debug", "--verbose", "controllers", "--refresh"}
21:44:27 INFO  juju.juju api.go:86 connecting to API addresses: [172.32.144.2:17070 172.33.144.2:17070 172.20.24.85:17070]
21:44:27 DEBUG juju.api apiclient.go:1094 successfully dialed "wss://172.20.24.85:17070/api"
21:44:27 INFO  juju.api apiclient.go:629 connection established to "wss://172.20.24.85:17070/api"
21:44:27 DEBUG juju.api monitor.go:35 RPC connection died
Controller           Model             User                  Access     Cloud/Region  Models  Nodes    HA  Version
sunbeam-controller*  admin/controller  microstack.multipass  superuser                     2      1  none  3.5.2
21:44:27 INFO  cmd supercommand.go:556 command finished

ubuntu@microstack:~$ sudo netplan get ethernets
eth0:
  match:
    macaddress: "52:54:00:8e:0e:6b"
  dhcp4: true
  dhcp6: true
  set-name: "eth0"
eth1:
  match:
    macaddress: "52:54:00:16:cc:74"
  addresses:
  - "172.32.144.2/20"
  dhcp4: false
  routes:
  - metric: 50
    to: "default"
    via: "172.32.144.1"
eth2:
  match:
    macaddress: "52:54:00:44:cf:6f"
  addresses:
  - "172.33.144.2/20"
  dhcp4: false
ubuntu@microstack:~$

ubuntu@microstack:~$ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         mp-host              0.0.0.0         UG    50     0        0 eth1
default         mp-host              0.0.0.0         UG    100    0        0 eth0
10.1.191.64     0.0.0.0         255.255.255.192 U     0      0        0 *
10.1.191.65     0.0.0.0         255.255.255.255 UH    0      0        0 cali7bb9b0c8fc4
.
.
.
10.1.191.103    0.0.0.0         255.255.255.255 UH    0      0        0 cali040c76ae798
172.20.16.0      0.0.0.0         255.255.240.0   U     100    0        0 eth0
mp-host          0.0.0.0         255.255.255.255 UH    100    0        0 eth0
172.32.144.0    0.0.0.0         255.255.240.0   U     0      0        0 eth1
172.33.144.0    0.0.0.0         255.255.240.0   U     0      0        0 eth2
ubuntu@microstack:~$

I haven't managed yet to figure out why Juju is connecting with the default interface IP specifically or whether I can reconfigure this behavior

[abeermmukhemir]

BTW, for your case, instead of dhcp4_overrides in the netplan config, you would use something like:

      routes:
        - to: 0.0.0.0/0
          via: 10.13.31.1
          metric: 50

Or you could apply a higher metric to the default interface with dhcp4_overrides.

It seems this approach would also help in hosting microk8s on multipass, I see the right IP is configured as the node IP

ubuntu@microstack:~$ microk8s kubectl get nodes -o wide
NAME         STATUS   ROLES    AGE     VERSION    INTERNAL-IP    EXTERNAL-IP   OS-IMAGE         KERNEL-VERSION       CONTAINER-RUNTIME
microstack   Ready    <none>   4h32m   v1.28.10   172.32.144.2   <none>        Ubuntu Core 20   5.15.0-116-generic   containerd://1.6.28
ubuntu@microstack:~$

[ricab]

Hi @abeermmukhemir, thanks for sharing your experience.

I just want to add some context for where Multipass currently stands and where it could go. Multipass currently relies on that default interface to manage the instance but, in the future, we would like to do that with a VSOCK instead. When we have that, users will be able to modify the default interface as they see fit. We have been contemplating that idea for a long time, but we're a small team — and two of us left recently — with 3 platforms and multiple backends to support. I am afraid that there have always been more important things to do first, so that hasn't been selected for development.

[abeermmukhemir]

Hi @ricab, thanks for explaining the situation I'd like to note that I've got invested in this for a while now and I really want to see multipass succeed with hyper-v backend (for all of us who are required to stay with Hyper-v), so, if there is a clear path to develop this I can contribute

[ricab]

Thanks for the enthusiasm @abeermmukhemir! That would be nice, but quite difficult in this case. Not only is this something that would need to be implemented on all platforms and made to work with all backends, I am afraid that Windows and macOS specific bits are closed FTTB.