search

kairos-io / kairos

:penguin: The immutable Linux meta-distribution for edge Kubernetes.
https://kairos.io
Apache License 2.0
1.11k stars 97 forks source link

Filesystem doesn't resize on RPI4 and alpine flavor #2863

Open Ludea opened 3 weeks ago

Ludea commented 3 weeks ago

Kairos version:

tested version : 3.1.2 and 3.1.3 flavor: alpine

CPU architecture, OS, and Version:

Rp4 aarch64

Describe the bug

Filesystem is not resizing Partition resizing works as expected

To Reproduce

Expected behavior

Filesystem expand all available free space

Logs

df -h:

Filesystem                         Size  Used Avail Use% Mounted on
 /dev/disk/by-label/COS_ACTIVE      2.0G  697M  1.2G  38% /
 /dev/disk/by-label/COS_OEM          55M   32K   52M   1% /oem
 /dev/disk/by-label/COS_PERSISTENT   55M  1.9M   50M   4% /usr/local
 overlay                            2.0G  180K  2.0G   1% /var
 overlay                            2.0G  180K  2.0G   1% /etc
 overlay                            2.0G  180K  2.0G   1% /srv
 devtmpfs                            10M     0   10M   0% /dev
 tmpfs                              3.9G  544K  3.9G   1% /run
 shm                                3.9G     0  3.9G   0% /dev/shm
 tmpfs                              3.9G     0  3.9G   0% /tmp
 /dev/mmcblk0p2                     5.9G  1.4G  4.3G  25% /run/initramfs/cos-state
 tmpfs                              2.0G     0  2.0G   0% /run/overlay

dmesg:

[    4.651890] udevd[995]: failed to execute '/usr/libexec/nfsrahead' '/usr/libexec/nfsrahead 1:1': No such file or directory
 [    4.652275] udevd[997]: failed to execute '/usr/libexec/nfsrahead' '/usr/libexec/nfsrahead 179:0': No such file or directory
 [    4.652795] udevd[996]: failed to execute '/usr/libexec/nfsrahead' '/usr/libexec/nfsrahead 1:10': No such file or directory
 [    4.653498] udevd[998]: failed to execute '/usr/libexec/nfsrahead' '/usr/libexec/nfsrahead 1:0': No such file or directory
 [    4.653902] udevd[999]: failed to execute '/usr/libexec/nfsrahead' '/usr/libexec/nfsrahead 1:11': No such file or directory
 [    4.654311] udevd[1000]: failed to execute '/usr/libexec/nfsrahead' '/usr/libexec/nfsrahead 1:12': No such file or directory
 [    4.654535] udevd[1001]: failed to execute '/usr/libexec/nfsrahead' '/usr/libexec/nfsrahead 1:13': No such file or directory
 [    4.677703] NET: Registered PF_PACKET protocol family
 [    4.683017] udevd[1007]: failed to execute '/usr/libexec/nfsrahead' '/usr/libexec/nfsrahead 1:7': No such file or directory
 [    4.686944] udevd[1009]: failed to execute '/usr/libexec/nfsrahead' '/usr/libexec/nfsrahead 7:0': No such file or directory

immucore.log:

1970-01-01T00:00:11Z INF Immucore commit=none compiled with=go1.22.5 version=v0.4.2
 1970-01-01T00:00:11Z INF creating a runtime
 1970-01-01T00:00:11Z INF detecting boot state
 1970-01-01T00:00:11Z INF Booting on active/passive/recovery.
 1970-01-01T00:00:11Z INF 1.
  <init> (background: false) (weak: false) (run: false)
 2.
  <lvm-activation> (background: false) (weak: false) (run: false)
  <mount-state> (background: false) (weak: false) (run: false)
  <mount-tmpfs> (background: false) (weak: false) (run: false)
  <create-sentinel> (background: false) (weak: false) (run: false)
 3.
  <discover-state> (background: false) (weak: false) (run: false)
  <upgrade-kcrypt> (background: false) (weak: false) (run: false)
 4.
  <mount-root> (background: false) (weak: false) (run: false)
 5.
  <mount-oem> (background: false) (weak: false) (run: false)
 6.
  <rootfs-hook> (background: false) (weak: false) (run: false)
  <unlock-all> (background: false) (weak: false) (run: false)
 7.
  <load-config> (background: false) (weak: false) (run: false)
 8.
  <mount-base-overlay> (background: false) (weak: false) (run: false)
  <custom-mount> (background: false) (weak: false) (run: false)
 9.
  <overlay-mount> (background: false) (weak: false) (run: false)
 10.
  <mount-bind> (background: false) (weak: false) (run: false)
 11.
  <write-fstab> (background: false) (weak: false) (run: false)
 12.
  <initramfs-hook> (background: false) (weak: false) (run: false)

 1970-01-01T00:00:11Z INF creating a runtime
 1970-01-01T00:00:11Z INF detecting boot state
 1970-01-01T00:00:11Z INF creating a runtime
 1970-01-01T00:00:11Z INF detecting boot state
 1970-01-01T00:00:11Z INF creating a runtime
 1970-01-01T00:00:11Z INF Setting sentinel file to=active_mode
 1970-01-01T00:00:11Z INF detecting boot state
 1970-01-01T00:00:12Z INF mount done options=["rw"] type=tmpfs what=tmpfs where=/tmp
 1970-01-01T00:00:12Z INF mount done options=["ro"] type=ext3 what=/dev/disk/by-label/COS_STATE where=/sysroot/run/initramfs/cos-state
 1970-01-01T00:00:13Z INF mount done options=["ro","suid","dev","exec","async"] type=ext4 what=/dev/disk/by-label/COS_ACTIVE where=/sysroot
 1970-01-01T00:00:13Z INF creating a runtime
 1970-01-01T00:00:13Z INF detecting boot state
 1970-01-01T00:00:14Z INF mount done options=["rw","suid","dev","exec","async"] type=ext3 what=/dev/disk/by-label/COS_OEM where=/sysroot/oem
 1970-01-01T00:00:14Z INF Running rootfs stage
 1970-01-01T00:00:14Z INF triggering udev to populate disk info
 1970-01-01T00:00:21Z INF mount done options=["rw"] type=ext4 what=/dev/disk/by-label/COS_PERSISTENT where=/sysroot/usr/local
 1970-01-01T00:00:21Z WRN
 1970-01-01T00:00:21Z WRN executing mount callback error="mkdir /sysroot/snap: read-only file system" options=["bind"] type=overlay what=/sysroot/usr/local/.state/snap.bind where=/sysroot/snap
 1970-01-01T00:00:21Z ERR error="mkdir /sysroot/snap: read-only file system"
 1970-01-01T00:00:24Z WRN executing mount callback error="mkdir /sysroot/usr/share/pki: read-only file system" options=["bind"] type=overlay what=/sysroot/usr/local/.state/usr-share-pki-trust.bind where=/sysroot/usr/share/pki/trust
 1970-01-01T00:00:24Z ERR error="mkdir /sysroot/usr/share/pki: read-only file system"
 1970-01-01T00:00:24Z WRN executing mount callback error="mkdir /sysroot/usr/share/pki: read-only file system" options=["bind"] type=overlay what=/sysroot/usr/local/.state/usr-share-pki-trust-anchors.bind where=/sysroot/usr/share/pki/trust/anchors
 1970-01-01T00:00:24Z ERR error="mkdir /sysroot/usr/share/pki: read-only file system"
 1970-01-01T00:00:24Z WRN error="3 errors occurred:\n\t* mkdir /sysroot/snap: read-only file system\n\t* mkdir /sysroot/usr/share/pki: read-only file system\n\t* mkdir /sysroot/usr/share/pki: read-only file system\n\n"
 1970-01-01T00:00:24Z INF Running initramfs stage
 2006-08-14T08:35:02Z INF 1.
  <init> (background: false) (weak: false) (run: false)
 2.
  <create-sentinel> (background: false) (weak: false) (run: true)
  <mount-tmpfs> (background: false) (weak: false) (run: true)
  <lvm-activation> (background: false) (weak: false) (run: true)
  <mount-state> (background: false) (weak: false) (run: true)
 3.
  <upgrade-kcrypt> (background: false) (weak: false) (run: true)
  <discover-state> (background: false) (weak: false) (run: true)
 4.
  <mount-root> (background: false) (weak: false) (run: true)
 5.
  <mount-oem> (background: false) (weak: false) (run: true)
 6.
  <rootfs-hook> (background: false) (weak: false) (run: true)
  <unlock-all> (background: false) (weak: false) (run: true)
 7.
  <load-config> (background: false) (weak: false) (run: true)
 8.
  <custom-mount> (background: false) (weak: false) (run: true)
  <mount-base-overlay> (background: false) (weak: false) (run: true)
 9.
  <overlay-mount> (background: false) (weak: false) (run: true)
 10.
  <mount-bind> (error: 3 errors occurred:
         * mkdir /sysroot/snap: read-only file system
         * mkdir /sysroot/usr/share/pki: read-only file system
         * mkdir /sysroot/usr/share/pki: read-only file system

 ) (background: false) (weak: false) (run: true)
 11.
  <write-fstab> (background: false) (weak: false) (run: true)
 12.
  <initramfs-hook> (background: false) (weak: false) (run: true)

Additional context

Itxaka try to reproduce issue on his Rpi, but FS resize as expected. With Tumbleweed flavor, no issue, partition and fs resizing work as expected

Itxaka commented 13 hours ago

going to retry and put the logs here to see.

Itxaka commented 11 hours ago

just intalled from scratch and cant reproduce:

Disk /dev/sda: 115.1 GiB, 123589361664 bytes, 241385472 sectors
Disk model: 3.2 Gen 1       
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 33553920 bytes
Disklabel type: gpt
Disk identifier: 85800D3E-DE8D-4BE1-8D6A-FDE3987F8CD8

Device        Start       End   Sectors   Size Type
/dev/sda1      2048    198655    196608    96M EFI System
/dev/sda2    198656  12896255  12697600   6.1G Linux filesystem
/dev/sda3  12896256  21497855   8601600   4.1G Linux filesystem
/dev/sda4  21497856  21628927    131072    64M Linux filesystem
/dev/sda5  21628928 241385436 219756509 104.8G Linux filesystem
localhost:~$ df -h
Filesystem                         Size  Used Avail Use% Mounted on
/dev/disk/by-label/COS_ACTIVE      2.0G  697M  1.2G  38% /
/dev/disk/by-label/COS_OEM          55M   32K   52M   1% /oem
/dev/disk/by-label/COS_PERSISTENT   99G  1.8M   93G   1% /usr/local
overlay                            2.0G  176K  2.0G   1% /var
overlay                            2.0G  176K  2.0G   1% /etc
overlay                            2.0G  176K  2.0G   1% /srv
devtmpfs                            10M     0   10M   0% /dev
tmpfs                              3.9G  608K  3.9G   1% /run
efivarfs                           128K   920  128K   1% /sys/firmware/efi/efivars
shm                                3.9G     0  3.9G   0% /dev/shm
tmpfs                              3.9G     0  3.9G   0% /tmp
/dev/sda2                          5.9G  1.4G  4.3G  25% /run/initramfs/cos-state
tmpfs                              2.0G     0  2.0G   0% /run/overlay
localhost:~$ cat /etc/kairos-release 
KAIROS_PRETTY_NAME="kairos-standard-alpine-3.19 v3.2.1-7-ga068314-v1.31.1-k3s1"
KAIROS_IMAGE_REPO="quay.io/kairos/alpine:3.19-standard-arm64-rpi4-v3.2.1-7-ga068314-k3sv1.31.1-k3s1"
KAIROS_IMAGE_LABEL="3.19-standard-arm64-rpi4-v3.2.1-7-ga068314-k3sv1.31.1-k3s1"
KAIROS_REGISTRY_AND_ORG="quay.io/kairos"
KAIROS_BUG_REPORT_URL="https://github.com/kairos-io/kairos/issues"
KAIROS_SOFTWARE_VERSION="v1.31.1+k3s1"
KAIROS_ID="kairos"
KAIROS_VERSION="v3.2.1-7-ga068314-v1.31.1-k3s1"
KAIROS_FLAVOR="alpine"
KAIROS_FAMILY="alpine"
KAIROS_SOFTWARE_VERSION_PREFIX="k3s"
KAIROS_NAME="kairos-standard-alpine-3.19"
KAIROS_VERSION_ID="v3.2.1-7-ga068314-v1.31.1-k3s1"
KAIROS_MODEL="rpi4"
KAIROS_HOME_URL="https://github.com/kairos-io/kairos"
KAIROS_ID_LIKE="kairos-standard-alpine-3.19"
KAIROS_FLAVOR_RELEASE="3.19"
KAIROS_VARIANT="standard"
KAIROS_TARGETARCH="arm64"
KAIROS_RELEASE="v3.2.1-7-ga068314"
KAIROS_GITHUB_REPO="kairos-io/kairos"
KAIROS_ARTIFACT="kairos-alpine-3.19-standard-arm64-rpi4-v3.2.1-7-ga068314-k3sv1.31.1+k3s1"

Stage is correctly executed

1970-01-01T00:00:20Z INF Processing stage step 'RPI configs.Grow persistent'. ( commands: 0, files: 0, ... )
1970-01-01T00:00:20Z DBG Stage: schema.Stage{
  If: "[ ! -f /run/cos/recovery_mode ] && [ ! -f /run/cos/live_mode ] && [ -f \"/sys/firmware/devicetree/base/model\" ] && grep -i \"Raspberry Pi 4\" \"/sys/firmware/devicetree/base/model\"",
  Name: "Grow persistent",
  Layout: schema.Layout{
    Device: &schema.Device{
      Label: "COS_PERSISTENT",
    },
    Expand: &schema.Expand{},
  },
}
1970-01-01T00:00:20Z DBG Using label COS_PERSISTENT for layout expansion
1970-01-01T00:00:20Z DBG Output of udevadm settle: 
1970-01-01T00:00:20Z DBG Got device /dev/sda for label COS_PERSISTENT
1970-01-01T00:00:21Z INF Extending last partition to max space
1970-01-01T00:00:21Z DBG Expanding partition 5 up to 0 sectors
1970-01-01T00:00:22Z DBG Output of growpart: CHANGED: partition=5 start=21628928 old: size=131072 end=21759999 new: size=219756510 end=241385437

1970-01-01T00:00:22Z DBG Trying to reread the partition table of /dev/sda (try number 1)
1970-01-01T00:00:22Z DBG output of partprobe: 
1970-01-01T00:00:22Z DBG Output of sync: 
1970-01-01T00:00:22Z DBG Trying to resize filesystem for device /dev/sda5
1970-01-01T00:00:22Z DBG Found filesystem ext3
 for device /dev/sda5
1970-01-01T00:00:22Z DBG Output from running e2fsck e2fsck 1.47.0 (5-Feb-2023)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
COS_PERSISTENT: 11/16384 files (0.0% non-contiguous), 9525/65536 blocks

1970-01-01T00:01:57Z DBG Output from running resize2fs resize2fs 1.47.0 (5-Feb-2023)
Resizing the filesystem on /dev/sda5 to 109878252 (1k) blocks.
The filesystem on /dev/sda5 is now 109878252 (1k) blocks long.

1970-01-01T00:01:57Z DBG Trying to reread the partition table of /dev/sda (try number 1)
1970-01-01T00:01:57Z DBG output of partprobe: 
1970-01-01T00:01:57Z DBG Output of sync:

I have no idea whats going on @Ludea Please retry from scratch and run the Kairos entry adding rd.immucore.debug to the cmdline and then paste the /run/immucore/immucore.log and /run/immucore/rootfs_stage.log full logs to see whats going on.