Open smira opened 6 months ago
This feature is going to be shifted to Talos 1.8.0 (only first bits might appear in Talos 1.7.0).
Talos 1.8.0 will be released as soon as this feature is ready.
Some software like longhorn might not respect limits and fill the whole disk. It would be great if a misbehaving pod cannot destroy etcd or other core parts of talos by just claiming all the available disk space.
Some software like longhorn might not respect limits and fill the whole disk. It would be great if a misbehaving pod cannot destroy etcd or other core parts of talos by just claiming all the available disk space.
This is good to know. I always liked the separation of Talos and it using a dedicated disk to prevent unknowns/complications like this. Any ideas on how we could impose those limitations?
From my point of view something like lvm and partitions for each part would help. I used a similar setup in k3s and had never issues like this.
LVM would also make the encryption part easy because you only have to encrypt one device…
Allow the choice of any block device.
A partition is also a block device. People could partition a single SSD with sufficient space for Talos and then an additional partition for general use. Filling up the general use partition isn’t going to affect the Talos partition(s)
Allow the choice of any block device.
A partition is also a block device. People could partition a single SSD with sufficient space for Talos and then an additional partition for general use. Filling up the general use partition isn’t going to affect the Talos partition(s)
Similar to how the newer ESXi installer does when using the systemMediaSize
option, this allows the installer to make the system & OS partitions at the beginning of the disk.While leaving free space at the end of the disk.
I think at minimal we would need two partitions or lvm volumes: Talos (etcd and other stuff) General purpose (container data and so on)
It would be great if we could have an option to say okay we also need 100gb longhorn space and 50gb local path space. That are just some examples we would just need a volume size and mount path. All remaining space could be assigned to the general purpose partition. Here the default setting should be to use all space.
With something like lvm we could also allow to fix the general volume to a specific size and leave the remaining space unused. The would allow for expansion of other volumes or ensure that all nodes are the same even if one has a bigger disk.
This feature is going to be shifted to Talos 1.8.0 (only first bits might appear in Talos 1.7.0).
Talos 1.8.0 will be released as soon as this feature is ready.
Thank you for clarifying @smira! If I set up a cluster with 1.7 today, will there be a migration path in 1.8 to have talos managing disks as proposed in this issue?
Thank you for clarifying @smira! If I set up a cluster with 1.7 today, will there be a migration path in 1.8 to have talos managing disks as proposed in this issue?
Talos is always backwards compatible, so upgrade to 1.8 will always work. You would be able to start using volume management features, but some of them (e.g. shrinking /var
) might require a wipe of some volumes.
OpenEBS had a component called ndm (node-disk-manager) that was quite handy to manage block-devices. HostPath and OS disks could be excluded with filters, e.g.:
filterconfigs:
- key: os-disk-exclude-filter
name: os disk exclude filter
state: true
exclude: "/,/etc/hosts,/boot,/var/mnt/openebs/nvme-hostpath-xfs"
This was used by the localpc-device sc, letting you assign a whole block device to a pod. Unfortunately they have stopped supporting ndm and localpv-device with the release of OpenEBS 4.0.
It would be great if talos had a similar feature!
this is incredibly exciting, happy to give it a whirl once you get an RC/beta or something @smira . thank you!
@smira TopoLVM requires a pvcreate
and vgcreate
to be able to allocate remaining free disk space.
What I get from this issue, is that we would be able to at least allocate the system disk with free-space remaining, which is already 99% of the way there!
Does it also allow us to use lvm using pvcreate
and vgcreate
to consume the rest of the disk space?
Note: not a lvm specialist at all.
As soon as #8901 is merged, we will release the last alpha version.
At this point, the internals for volume management got completely implemented from scratch, while there are new configuration options available to use new features.
Before 1.8.0-beta.0, we plan to add minimal configuration for EPHEMERAL
- creating on a different disk, controlling maximum/minimum size, and growth.
All other changes got shifted towards 1.9.0.
Hi @smira, huge congratulations for your incredible work on this PR 🎉
Having a quick look at the content of the PR + the documentation added to it, I couldn't find an answer to the following question:
With Talos v1.8 will we be able to do a full-ram PXE boot without having to install the OS on a disk? Or is this one of the features shifting to v1.9?
Thanks a lot 🙏
With Talos v1.8 will we be able to do a full-ram PXE boot without having to install the OS on a disk? Or is this one of the features shifting to v1.9?
not in 1.8 (probably 1.9), two pieces are missing:
tmpfs
volumes for required EPHEMERAL/STATE (will come as part of this workflow)Well done on the work done on 1.8.0-alpha.2, can't wait to see the results in practice 👏 🎉 (Your newly added get discoveredvolume
will help us a lot)
If I understand correctly, the changes made mainly concern the new concept of volumes (which will be used for the new features in 1.9). This means that Machine Configurations are not yet impacted, am I right?
Thank you in advance 🙏
If I understand correctly, the changes made mainly concern the new concept of volumes (which will be used for the new features in 1.9). This means that Machine Configurations are not yet impacted, am I right?
yes, for 1.8.0-alpha.2, there are no machine configuration changes, and everything "should just work as before".
Before 1.8.0-beta.0, there will be new machine configuration document to configure some aspects of EPHEMERAL
volume (this is the most requested feature).
Everything else for 1.9.
Closely related: #8016
Problem Statement
Talos Linux is not flexible in the way it manages volumes, it occupies the whole system disk, creating an EPHEMERAL partition covering 99% of the disk space. User disk management is fragile, requires extra steps to get it to work properly (mounting into the kubelet), doesn’t support wiping disks, etc. Talos does not detect properly various partition types which leads to wiping user data (e.g. Ceph bluestore).
There were following requests from the users/customers which can’t be addressed in the current design:
STATE
/EPHEMERAL
ontmpfs
)/var
from an NVMe/SSDcontainerd
stateetcd
data directory to a separate disk/var
:containerd
state)/var/mnt/foo
)/data
The proposed design provides an option to solve the issues mentioned above.
Groundwork
Before we move into volume management operations, there is some amount of work that needs to be done to improve the blockdevice management operations:
blkid
, to allow easier identification of storage objects.Installation Process
Talos installation should do the bare minimum to make sure that Talos can be booted from the disk, without touching the pieces which are not strictly required to boot Talos. This might include installing Talos without having machine configuration.
So the install should only touch following partitions/objects:
BOOT
/EFI
partitions (boot assets, boot loader itself)META
partitionMBR
Any management of the storage/volumes should be deferred to the Talos running on the
host
(i.e. creating/var
,/system/state
, etc.)Volumes
Let’s introduce a new concept of the
volumes
, which will solve the ideas mentioned above and allow us to take storage management to the next level.There are two kinds of volumes:
Every volume has several most important features:
Volumes support a basic set of operations:
Volume types:
Volume formats:
Volume additional options:
System Volumes
As of today, Talos implicitly has the following volume types:
Volume Lifecycle
Talos services can express their dependency on the volumes. For example,
kubelet
service can only be started whenkubelet data
volume is available. Same way, ifkubelet data
volume is going to be unmounted,kubelet
should be stopped first.The boot process should naturally stop when the required volume is not available. E.g. maintenance mode of Talos implies that the boot can’t proceed as long as the volume configuration is not available.
Volume Configuration
System volumes have implicit configuration, which is applied as long as
v1alpha1.Config
is applied to the machine. Some properties are configurable inv1alpha1.Config
, e.g. disk encryption. If an explicit volume configuration is provided, Talos uses that.For example, if the user configures
EPHEMERAL
to betmpfs
of size 10 GiB, it will be created on each boot as instructed.Users might provide configuration for user volumes (similar to the user disks feature today), which might be critical for the pods to be started, an otherwise e.g. extension services might provide a dependency on the additional volumes.
Some system volumes might be optional, i.e. configured by the users - for example, container image cache.
Upgrades and Wiping
Talos Linux upgrades should not wipe anything by default, and wiping should be an additional operation which can be done without an upgrade, or can optionally be combined with an upgrade.
Update itself should only modify boot assets/boot loader, i.e. ensure that the new version of Talos Linux can be booted up from the disk device.
Wiping is volume-based, examples:
EPHEMERAL
, which implies wiping all volumes which haveEPHEMERAL
as a parent (e.g. subdirectory volume of/var/lib/etcd
); all services which depend onEPHEMERAL
or its children should be stopped, but reboot is not necessary, as theEPHEMERAL
will be re-provisioned after the wipeetcd data
, which in the default configuration implies leavingetcd
, stoppingetcd
services, performingrm -rf /var/lib/etcd
, and re-startingetcd
join processNotes
As pointed out by @utkuozdemir,
EPHEMERAL
might be a bad name given that the partition is not supposed to be forced wiped by default.