search

siderolabs / omni

SaaS-simple deployment of Kubernetes - on your own hardware.
Other
402 stars 23 forks source link

[feature] Omni should allow wiping user disks when resetting nodes #211

Open cullenmcdermott opened 2 months ago

cullenmcdermott commented 2 months ago

Problem Description

When resetting a node in Omni it does not currently wipe disks other than the install disk.

Solution

I would like to be able to specify omni managed disks when creating a machine/machine class so that omni will handle wiping them for me when resetting a node.

Alternative Solutions

No response

Notes

No response

utkuozdemir commented 2 months ago

Edit: Was mistaken, see the comment below.

~This is more about Talos than Omni - Talos claims ownership of a single disk and manages that, so wiping will also affect only that disk.~ ~We are working on a much more powerful volume management on Talos, you can see its status here: https://github.com/siderolabs/talos/issues/8367~

~Only after we get this released (planned for Talos 1.8), we can support it in Omni (which would take some additional time to land ofc.)~

Unix4ever commented 2 months ago

@utkuozdemir this issue isn't about the system disk, but about the extra disks. We have everything prepared for that.

Talos support was already added for that: it's possible to both see disk types there and set extra disks to wipe in the reset command. We just never got to Omni part due to that there was not any demand for this feature.

utkuozdemir commented 2 months ago

@utkuozdemir this issue isn't about the system disk, but about the extra disks. We have everything prepared for that.

Talos support was already added for that: it's possible to both see disk types there and set extra disks to wipe in the reset command. We just never got to Omni part due to that there was not any demand for this feature.

Oh, wasn't aware that it was possible. Apologies for the confusion.

bernardgut commented 2 months ago

Hello again. If I may, IMO :

  1. talosct reset should have an option to keep the "extra disks" data. More specifically --wipe-devices-data flag that is off by default. If you don't do that people are going to come back at Omni/Talos with You ate all my data. I hate you complaints (Especially from the beginners doing K8s-on-the-edge use-cases).
  2. Right now omnictl cluster delete does not delete the "extra disks" data. In particular, it does not run the CSI finalizers before tearing down the cluster. So if you create/delete multiple clusters in a row with some CSI (Piraeus for example) You end up with a lot of dangling volumes on the physical disks that do not show up in your CSI as they do not have any corresponding resource assignment in K8s. I will open a ticket for this, but in the meantime It would make sense to have something like --wipe-devices-data for omnictl delete cluster as well, such that the user does not need to run both omnictl delete cluster AND talosctl reset node for each node to do a wipe. (Note: this is NOT a good long term solution as you wipe everything including what is not "owned" by omni from the nodes, but its a good quickfix for now).
Unix4ever commented 2 months ago

The idea is to let users explicitly mark some disks as managed by Omni. Only then they'll be wiped.

willbush commented 1 month ago

I suspect one use-case is to cleanup rook-ceph storage. However, this will require more than just wiping the disk.

bernardgut commented 1 month ago

In my case, the use-case is wiping the Linstor-created lvm/zfs PVs. Which is something similar as what you described above @willbush for Ceph.

I tested this a bit when I saw my "free space" shrink with every cluster instantiation in Omni, and I can report that the zfs/lvm partition tables do not get wiped with a dd command (and it is also a waste of time) on the whole device. My current understanding is that You need to specifically wipe the device partitions with something like wipefs,which should be enough for the underlying lvm/zfs "strings" to be completely removed from the device. I couldn't test this theory because it is not shipped with Talos but I think it should work.

willbush commented 1 month ago

@bernardgut Perhaps a more general solution is for Omni to have some kind of hook to run a script on each node when resetting?

Omni makes it easier to destroy a cluster than most websites do to unsubscribe from their newsletter.

So I personally went with the approach to make a ceph prepare bash script that has to be run manually when first setting up the cluster. I can share if you want.

It basically:

The job:

I found block devices that were used by rook-ceph will mention ceph in the blkid output.

The slowest step is installing gdisk. I probably should have tried wipefs. I read a random comment online that said it can leave the disk in a state that ceph doesn't like. dd generally works from my testing, but seemed really hacky. Probably wipefs + dd would be enough and not require installing gdisk.

@rsmitty I thought about making a PR to contrib for Omni for the script, but I'm not sure if such a thing is really desired.