Support for Xen Guest Tools

[danielr1996]

First and foremost: I recognize that this feature request conflicts the principles of a small and immutable os, but I'll elaborate why I would need this and how it can also benefit others.

Feature Request

I'd like to have the xen guest tools installed inside talos, so that I can retrieve the ip address of my nodes to be automatically used in terraform.

Use Case

Infrastructure as Code (IaC) approaches like terraform are more and more taking off these days, if they are not already pretty standard for many people. Im working on starting a kubernetes cluster with a single terraform apply command. With Talos this should be really easy possible because it generates all the certificate stuff needed to join nodes beforehand. However on piece is currently missing for my approach, to provision the created vms I need terraform to know the ip addresses. I'm using Xen Orchestra (xoa) as my virtualization solution which has a terraform provider that would theoretically allow me to retrieve the ip addresses for the vms, however in order to do this inside my vm I need to install the guest tools which is easy for a standard ubuntu vm, but not for Talos.

I have four ideas, sorted from worst to best in my opinion

Allow ssh access to vms or expose a package management interface

This defeats the purpose of a immutable os so I'm not considering it any further.

Integrate xen guest tools by default

For me it would be the easiest to have the xen guest tools just installed by default. However it would defeat the purpose of a small os for other people that don't run talos inside xoa, also other people might need guest tools for kvm, aws, etc. This would make talos very bloated.

Create own iso with xen guest tools enabled

I don't know the process how to do that, but theoretically I could just build my own iso starting from the official talos release with xen guest tools installed. Maybe someone has more information on how to package my own iso, I'm thinking of packer or something like that.

Activate xen guest tools through a kernel boot flag

The simplest way from a user perspective would be to have a boot flag that activates xen guest tools, or other guest tools that other users might need.

Solutions 3 and 4 would be my favourites because in my opinion it aligns with the projects pilosophy. Solution 3 has the advantage of shifting the work needed from the Talos developers to the user. Solution 4 needs work from the developers, but in the end is easier for the end user.

Let me know what you think of that and if this is a use case that you would want to support in some way, or if this is even already possible.

[danielr1996]

Reading a bit through the documentation I noticed there is already a platform kernel argument. However it's not really clear what excactly changes between the various options. Would it be possible to add a xen option which enables the xen guest tools?

[rsmitty]

Can you run the guest tools as a daemonset in Kubernetes? That's what other folks are doing for things like open-vmtools for vmware (see https://github.com/mologie/talos-vmtoolsd)

[danielr1996]

Didn't even think of that, but I fear this won't work for my use case. While it's fine for enabling gracefull shutdown / restart I need terraform to know the ip address before I even have a cluster that I can deploy anything to.

[mologie]

You can reference manifests to auto-install upon cluster setup in Talos' configuration. This would be the preferred installation method for talos-vmtoolsd after #4422 is implemented. Chances are that it works for Xen guest tools right now as long as you are not specifically integrating with Talos' API like talos-vmtoolsd.

https://www.talos.dev/docs/v0.13/reference/configuration/ ctrl+f "extraManifests"

[danielr1996]

I guess you are referring to the extraManifests attribute in the configuration, right? This has still the problem that I need to know the ips of the nodes before I even can send the configuration to these nodes.

[dvicory]

@danielr1996 Have you (or anyone else) figured out the best way to proceed on this? I want to run Talos on xcp-ng and running into a similar roadblock.

[danielr1996]

@danielr1996 Have you (or anyone else) figured out the best way to proceed on this? I want to run Talos on xcp-ng and running into a similar roadblock.

Nope, i've given up on talos completely and moved to provisioning my kubernetes cluster manually with ansible and kube-adm

[smira]

The proper way from Talos point of view is to package that as a system extension: https://github.com/siderolabs/extensions

In this case it comes with the OS, it is available at early boot, it has enough permissions to e.g. reboot the node.

[smira]

And also extension makes it opt-in, so enabled only if needed

[smira]

If anyone wants to build one, we're happy to help with the general direction.

[brunobriante]

I'm slowly working on enabling support for xen guest tools (the xcp-ng flavoured one) on my spare time and would like to hop in with some more information on how xen tooling works and where i'm stuck at the moment. It would be fantastic if you could point me in the right direction if possible @smira.

There is a really nice breakdown on how xe-guest-utilities works internally on this rancher/os issue. The TL;DR being there is a frontend application suite (xe-daemon, xe-linux-distribution and xenstore), a backend for the DomU to communicate with Dom0 (xenbus) and a udev rule for hotplugging vcpus.

---

The udev rule is easy to work with as a system extension is allowed to write in /usr/etc/udev/rules.d/ which should take care of that as far as i know.

A first roadblock exists on the frontend applications as they all depends internally on a package called guestmetric that hardcode /var/cache/xe-linux-distribution (see) as the path for this specific file, which causes a crash on Talos as /var/cache is read only. For now i've forked the repo and changed this path do /tmp/cache/xe-linux-distribution which is an hacky solution but works. I do understand that the best path forward would be allowing this container to write on /var/cache.

The second roadblock (and the one i'm currently working on) is access to xenbus on /proc/xen/xenbus, /dev/xen/xenbus or /kern/xen/xenbus. The linux kernel has support for xen DomU since 3.0 but from what i saw in the xen wiki there is some tweaking necessary on the kernel side to enable it. It looks like the kernel customization feature on Talos is enough for keeping this as an opt-in but i'm still working on it as docker is crashing constantly at my machine which is not giving me enough time to finish rebuilding the kernel.

[smira]

@brunobriante it's exciting to hear that the project is moving forward!

Talos should have by default first-class support for hypervisors, so if anything is missing from the kernel config, please create a PR to https://github.com/siderolabs/pkgs/tree/main/kernel/build to add any options missing

[smira]

as for the /var/cache file, I'm not sure what runs this... is this triggered via udev or part of a system extension?

if it's a part of system extension, as it runs in a container, anything can be mounted to the container to make sure /var/cache is writeable

[smira]

and in general /var is writeable in Talos... as soon as EPHEMERAL is mounted (on the host)

but inside the container with the system extension by default root fs is read-only, but it can be changed: https://www.talos.dev/v1.2/advanced/extension-services/#containersecurity

[frezbo]

I think this can be closed now, since starting Talos 1.6.0 extensions can be started early in the boot process.