Documentation: missing Xen Hypervisor chapter

SigmaSquadron commented 2 months ago

Problem

There is zero user-facing documentation for the Xen hypervisor on Nixpkgs. Of particular importance is documentation on how to set up networking interfaces, as the relevant options were removed in 24.11.

Proposal

Three things:

[ ] Write a NixOS documentation chapter that defines how to enable the module and set up networking. In the future, this documentation chapter can be expanded to explain how to configure more advanced features of Xen, such as a nix-based declarative configuration syntax for xl.
[ ] Write a Wiki page with step-by-step instructions that teaches the user how to get from a normal installation to a full NixOS dom0 with accompanying NixOS domUs.
[ ] Move the Xen README in pkgs/by-name/xe/xen/README.md to the Nixpkgs Manual.

Checklist

[x] checked latest Nixpkgs manual (source) and latest NixOS manual (source)
[x] checked open documentation issues for possible duplicates
[x] checked open documentation pull requests for possible solutions

Add a :+1: reaction to issues you find important.

hehongbo commented 1 month ago

The network setup may open up to diversity. For example one may consider the default scripted networking versus systemd-networkd, or choose between the built-in bridge of Linux, and OpenVSwitch. Also, I don't know if networkd and OVS are having questionable support with each other.

Another question is, the user might have different needs on their networking:

Create the bridge, add the primary physical interface into the bridge, connect VIFs of DomUs onto this bridge, Dom0's IP addresses are on the bridge, Dom0 and DomUs in the same L2 network (Default behavior of XCP-ng/Citrix, VMware ESXi, Proxmox, suitable for provisioning a bunch of VM servers)
Create the bridge, give the bridge it's own subnet and IP address, provide IP forwards/NAT/DNS into the network. (Default behavior of VMware Workstation/Fusion, VirtualBox, suitable for performing tests or running different OSes on a desktop environment.)

I believe the old Dom0 module performs the second one. The first one can cover both needs and reduces the complexity of network stack on Dom0, but may need careful setup to avoid disrupting the main network.

nixos-discourse commented 1 month ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixos-xen-project-reviving-the-xen-project-hypervisor-on-nixpkgs/52768/7

hehongbo commented 1 month ago

Setting up Open vSwitch with scripted networking, main network interface added as bridge member and with static IP can result in weird behaviors. Reported as 346857.

However, when used in combination with systemd-networkd, the Open vSwitch and its corresponding module might be okay with it.

networking.useDHCP = false;
networking.useNetworkd = true;

systemd.network = {
  enable = true;
  networks = {
    "xenbr0" = {
      name = "xenbr0";
      address = [ "192.168.0.102/24" ];
      gateway = [ "192.168.0.1" ];
      dns = [ "192.168.0.1" ];
    };
  };
};

systemd.network.wait-online.ignoredInterfaces = [ "eno1" ];

virtualisation.vswitch = {
  enable = true;
  resetOnStart = true;
};

networking.vswitches.xenbr0 = {
  interfaces = {
    eno1 = {};
  };
};

By the way, there is no "real" support of Open vSwitch according to Feature Request: systemd-networkd support for ovs (openvswitch) #2613. With a such setup, it's only telling systemd-networkd to work, waiting for the xenbr0 to be created and then eno1 added as a member. Either systemd-networkd and Open vSwitch would be okay later if vif-openvswitch adds more VIF members into the bridge on DomU creation.

One tiny thing I haven't figured out is, networkd would be "configuring" it forever.

[root@xen-dom0:~]# networkctl
IDX LINK       TYPE     OPERATIONAL SETUP
  1 lo         loopback carrier     unmanaged
  2 enp1s0     ether    enslaved    configuring
  3 enp2s0     ether    off         unmanaged
  4 eno1       ether    off         unmanaged
  5 enp4s0     ether    off         unmanaged
  6 ovs-system ether    off         unmanaged
  7 xenbr0     ether    routable    configured
  8 vif2.0     ether    enslaved    unmanaged
  9 vif1.0     ether    enslaved    unmanaged
 10 vif1.1     ether    enslaved    unmanaged

10 links listed.

This won't break anything but looked somehow imperfect. However, since network stacks are initialized on the bridge, systemd.network.wait-online.ignoredInterfaces should be set, or systemd-networkd-wait-online.service will wait forever as well, blocking future services/units from starting.

hehongbo commented 1 month ago

Also regarding the networkd+ovs setup, setting Unmanaged on the member interface won't fix.

     "xenbr0" = {
       name = "xenbr0";
       address = [ "192.168.0.102/24" ];
       gateway = [ "192.168.0.1" ];
       dns = [ "192.168.0.1" ];
     };
+    "eno1" = {
+      name = "eno1";
+      DHCP = "no";
+      linkConfig = {
+        Unmanaged = "yes";
+      };
+    };
   };
 };

 systemd.network.wait-online.ignoredInterfaces = [ "eno1" ];

Maybe I missed something, and I will update or reply once I figured it out.

NixOS / nixpkgs