search

lxc / incus

Powerful system container and virtual machine manager
https://linuxcontainers.org/incus
Apache License 2.0
2.62k stars 218 forks source link

Request to clean up instance's previous state's virtual NIC from host's stack. #983

Closed markrattray closed 3 weeks ago

markrattray commented 3 months ago

Required information

   * Storage backend in use: Incus managed ZFS on local disks.

# Issue description

At the moment we are still using macvlan type NICs with Incus instances until we're able to move to OVN.

Sometimes VMs (mostly Windows) are unable to start because the previous state of the virtual NIC ( `volatile.eth0.host_name` ) is still bound to the Incus node's parent NIC. 

It's not limited to one Incus node and the software installed in them are quite different:
  * incus node 01: MS SQL server with additional block device on enterprise grade NVMe for data, root disk on enterprise grade
  * incus node 03: MS IIS server with no additional devices, root disk on enterprise grade RAID10 SSDs
  * Both are running WS2022
  * I tested another instance created from the same image and it has 4 additional block devices in it, and after an update and reboot, it came up fine.

This happened before under LXD and was partially resolved:

incus start {instance-name} Error: Failed to start device "eth0": Failed adding link: Failed to run: ip link add name mace74a984e link br0 address 00:16:3e:24:a3:7a allmulticast on up type macvtap mode bridge: exit status 2 (RTNETLINK answers: Address already in use)

The manual steps to get the VM back up and running are:
  1. find the Incus node that the instance is running on
  2. on that node run: `ip link show | grep -B 1 '{instance-mac-address}'`
  3. delete the virtual NIC: `sudo ip link delete mac0f01152c`
  4. start the instance

My request is to get Incus to the following during the VM startup process:
  1. get the instance's MAC address/s
  2. check the Incus node's IP stack for lingering virtual NICs with the MACs and delete them
  3. continue with the VM startup

I think the previous attempts at fixing this may have been too granular and perhaps there might not be a need to fix multiple scenarios where this can happen, so maybe it just needs to clean up on every start because I have observed that these virtual NICs change at every startup. Also I might not be aware of other scenarios where there will be problems with what I'm suggesting.

Thanks

# Steps to reproduce

 1. VM might crash or reboot (in this scenario 2 affected VMs rebooted after Windows updates)
 2. Doesn't come up
 3. Try start manually and observe the error: `Address already in use`

# Information to attach

 - [x] Any relevant kernel output (`dmesg`)
 none
 - [ ] Container log (`incus info NAME --show-log`)
 - [ ] Container configuration (`incus config show NAME --expanded`)
 - [x] Main daemon log (at /var/log/incus/incusd.log)

This is the only entry around the time. time="2024-07-11T00:44:15Z" level=error msg="Failed to cleanly stop instance" err="Failed to start device \"eth0\": Failed adding link: Failed to run: ip link add name mac450200c0 link br0 address 00:16:3e:24:a3:7a allmulticast on up type macvtap mode bridge: exit status 2 (RTNETLINK answers: Address already in use)" instance=someinstance instanceType=virtual-machine project=someproject


 - [ ] Output of the client with --debug
 - [ ] Output of the daemon with --debug (alternatively output of `incus monitor --pretty` while reproducing the issue)
stgraber commented 3 months ago

Iterating over all the host interfaces to try to clean up potential conflicts shouldn't be needed and may actually be dangerous as it's perfectly valid in some environments to have the same MAC on multiple interfaces and starting to arbitrarily delete them may just cause a whole bunch of issues.

I spent around 30min trying to reproduce the issue you're describing, both by killing QEMU to simulate a hard crash and by triggering reboots from within a VM as seems to be the trigger for you, but I never managed to get the issue to happen here, so we're going to need some kind of somewhat reliable reproducer.

Looking at the macvlan nic cleanup logic, I'm not seeing anything wrong in there. As soon as the VM comes down, it triggers the onStop action which then iterates over all the devices on the instance and calls their Stop command. In the macvlan case, this will return a function that will delete the host device. I also did a test build here to make sure that code path is properly being hit during an instance initiated reboot and it did hit.

stgraber commented 3 months ago

If you can reproduce this somewhat reliably with a VM, it'd be good to run incus monitor --pretty on the system it's running, then reboot the VM and see it hit the issue. That should show us a better trace of all the calls being made.

Having the full incus config show --expanded output for an affected VM would also help as it's certainly possible that other devices or configuration are impacting this.

markrattray commented 3 months ago

Good morning. Sorry had a few emergencies so been away. Thank you for your efforts and checking out all this.

It's a bit random unfortunately and I've been rebooting VMs based on the same image regularly. The problematic ones did have a lot more workload than what I was rebooting. I'm working this Sunday so I'll see if I can reproduce the scenario again.

It might have something to do with the network setup on these hosts.... OVN wanted a dedicated NIC or a bridge, so to test OVN I deployed a bridge then OVN on a single NIC, but still using macvlan NICs for instances due to a routing issue to/from external networks and routed OVN networks.

stgraber commented 3 weeks ago

@markrattray did you have any luck on reproducing this somewhat reliably?

markrattray commented 3 weeks ago

Good morning.

I'll close this now because it hasn't happened in a while. It might have had something to do with the post cluster upgrade issue that you fixed for us, where we had FQDN to localhost entries in the hosts file. which caused issues on this cluster.

markrattray commented 3 weeks ago

Issue as not reoccurred in a while.