rancher / elemental

Elemental is an immutable Linux distribution built to run Rancher and its corresponding Kubernetes distributions RKE2 and k3s. It is built using the Elemental-toolkit
https://elemental.docs.rancher.com/
Apache License 2.0
295 stars 39 forks source link

Longhorn directory not persistent #1199

Open SanekP opened 7 months ago

SanekP commented 7 months ago

What steps did you take and what happened: Installed Elemental extension. helm upgrade --create-namespace -n cattle-elemental-system --install elemental-operator-crds oci://registry.suse.com/rancher/elemental-operator-crds-chart helm upgrade --create-namespace -n cattle-elemental-system --install elemental-operator oci://registry.suse.com/rancher/elemental-operator-chart OS Version Channel elemental-channel which is registry.suse.com/rancher/elemental-channel:1.4.2 contains ManagedOSVersion v2.0.2. Install this version from fresh Seed Image or upgrade to it through Update Groups. [A clear and concise description of what the bug is.] Installation does not contain persistent path /var/lib/longhorn.

What did you expect to happen: As it stated in documentation https://elemental.docs.rancher.com/installation path /var/lib/longhorn must be persistent. So I expected that it has mount to persistent partition as it was in previous versions e.g. v1.2.3. m# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS loop0 7:0 0 1.5G 1 loop / sda 8:0 0 465.8G 0 disk ├─sda1 8:1 0 1M 0 part ├─sda2 8:2 0 64M 0 part /oem ├─sda3 8:3 0 4G 0 part ├─sda4 8:4 0 8G 0 part /run/initramfs/cos-state └─sda5 8:5 0 453.7G 0 part /var/lib/calico /var/lib/cni /var/lib/NetworkManager /var/lib/kubelet /var/lib/rancher /var/lib/elemental /var/log /usr/libexec /root /opt /home /etc/cni /etc/iscsi /etc/ssh /etc/rancher /etc/systemd /usr/local sdb 8:16 1 0B 0 disk sr0 11:0 1 1024M 0 rom

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Environment:

frelon commented 6 months ago

Hi @SanekP!

Looks like that documentation is outdated, /var/lib/longhorn was removed as a default persistent path since there was problems trying to use longhorn with LVM when that path was already a persistent mount.

I would recommend following the https://elemental.docs.rancher.com/customizing/#adding-extra-lvm-volume-group-disks-during-the-installation guide to add a custom rootfs mount layout like the following:

name: "Custom Rootfs Layout Settings"
stages:
  rootfs.after:
    - if: '[ ! -f "/run/elemental/recovery_mode" ]'
      name: "Layout configuration"
      environment_file: /run/elemental/mount-layout.env
      environment:
        OVERLAY: "tmpfs:25%"
        RW_PATHS: "/var /etc /srv"
        PERSISTENT_STATE_BIND: "true"
        PERSISTENT_STATE_PATHS: >-
          /etc/systemd
          /etc/rancher
          /etc/ssh
          /etc/iscsi
          /etc/cni
          /home
          /opt
          /root
          /usr/libexec
          /usr/local
          /var/log
          /var/lib/elemental
          /var/lib/longhorn
          /var/lib/rancher
          /var/lib/kubelet
          /var/lib/NetworkManager
          /var/lib/cni
          /var/lib/calico
SanekP commented 6 months ago

This all looks so overcomplicated! I don't want to make deep customization, I just want to make /var/lib/longhorn path persistent (using existing COS_PERSISTENT partition). Is there somewhere an original of this rootfs mount layout? I mean something in RW_PATHS could change or I can miss some of PERSISTENT_STATE_PATHS.

kkaempf commented 6 months ago

Agreed @SanekP. I've created #1239 to address your concern.

SanekP commented 6 months ago

Also it is not expected after upgrading from previous OS version. Maybe it worth to create a Troubleshooting upgrade article how to mitigate this issue.

As I understand for the new installation I need somehow to put content from frelon to the /iso-config folder of ISO file. Is it possible to do same from MachineRegistration resource spec? It would be nice to download already ready ISO file.