flatcar / Flatcar

Flatcar project repository for issue tracking, project documentation, etc.
https://www.flatcar.org/
Apache License 2.0
774 stars 32 forks source link

ign-converter: formatting root device does not work if called 'root' #888

Open dahicks opened 2 years ago

dahicks commented 2 years ago

Description

Device not booting on stable versions > 3139.2.3 using ipxe. Root drive is not being properly labeled and is causing machine to not boot. Disk and filesystem partition configuration provided via ignition.

Impact

OS Never boots. Same (exact) configuration works on stable 3139.2.3

Environment and steps to reproduce

  1. Set-up: Flatcar Stable 3227.2.4 on Equinix Metal custom iPXE, c3.small.x86
  2. Task: OS Boot / Ignition Initialization
  3. Action(s): a. Booted Device via iPXE with supplied Ignition Spec
  4. Error: Could Not Boot. /dev/disk/by-label/root
    [   12.265610] localhost systemd[1]: Starting Ignition (disks)...
    [   12.274447] localhost ignition[716]: Stage: disks
    [   12.280439] localhost ignition[716]: no configs at "/usr/lib/ignition/base.d"
    [   12.289399] localhost ignition[716]: no config dir at "/usr/lib/ignition/base.platform.d/packet"
    [   12.300440] localhost ignition[716]: disks: createPartitions: op(1): [started]  waiting for devices [/dev/sda /dev/sdb]
    [   12.314525] localhost ignition[716]: disks: createPartitions: op(1): [finished] waiting for devices [/dev/sda /dev/sdb]
    [   12.328477] localhost ignition[716]: disks: createPartitions: created device alias for "/dev/sda": "/run/ignition/dev_aliases/dev/sda" -> "/dev/sda"
    [   12.343522] localhost ignition[716]: disks: createPartitions: created device alias for "/dev/sdb": "/run/ignition/dev_aliases/dev/sdb" -> "/dev/sdb"
    [   12.358483] localhost ignition[716]: disks: createPartitions: op(2): [started]  partitioning "/run/ignition/dev_aliases/dev/sda"
    [   12.372528] localhost ignition[716]: disks: createPartitions: op(2): wiping partition table requested on "/run/ignition/dev_aliases/dev/sda"
    [   12.387476] localhost ignition[716]: disks: createPartitions: op(2): running sgdisk with options: [--zap-all /run/ignition/dev_aliases/dev/sda]
    [   12.402525] localhost ignition[716]: disks: createPartitions: op(2): op(3): [started]  deleting 0 partitions and creating 0 partitions on "/run/ignition/dev_aliases/dev/sda"
    [   12.420494] localhost ignition[716]: disks: createPartitions: op(2): op(3): executing: "sgdisk" "--zap-all" "/run/ignition/dev_aliases/dev/sda"
    [   15.259605] localhost kernel: ata1.00: Enabling discard_zeroes_data
    [   15.266321] localhost kernel: ata1.00: Enabling discard_zeroes_data
    [   13.279278] localhost ignition[716]: disks: createPartitions: op(2): op(3): [finished] deleting 0 partitions and creating 0 partitions on "/run/ignition/dev_aliases/dev/sda"
    [   15.272649] localhost kernel: ata1.00: Enabling discard_zeroes_data
    [   13.302500] localhost ignition[716]: disks: createPartitions: op(2): op(4): [started]  reading partition table of "/run/ignition/dev_aliases/dev/sda"
    [   13.317482] localhost ignition[716]: disks: createPartitions: op(2): op(4): [finished] reading partition table of "/run/ignition/dev_aliases/dev/sda"
    [   13.333524] localhost ignition[716]: disks: createPartitions: op(2): running sgdisk with options: [--pretend --new=1:0:+0 --info=1 /run/ignition/dev_aliases/dev/sda]
    [   13.350471] localhost ignition[716]: disks: createPartitions: op(2): running sgdisk with options: [--new=1:0:+937701007 --change-name=1:root /run/ignition/dev_aliases/dev/sda]
    [   13.368520] localhost ignition[716]: disks: createPartitions: op(2): op(5): [started]  deleting 0 partitions and creating 1 partitions on "/run/ignition/dev_aliases/dev/sda"
    [   13.386479] localhost ignition[716]: disks: createPartitions: op(2): op(5): executing: "sgdisk" "--new=1:0:+937701007" "--change-name=1:root" "/run/ignition/dev_aliases/dev/sda"
    [   16.282065] localhost kernel: ata1.00: Enabling discard_zeroes_data
    [   14.298995] localhost ignition[716]: disks: createPartitions: op(2): op(5): [finished] deleting 0 partitions and creating 1 partitions on "/run/ignition/dev_aliases/dev/sda"
    [   16.288695] localhost kernel:  sda: sda1
    [   14.317494] localhost ignition[716]: disks: createPartitions: op(2): op(6): [started]  waiting for triggered uevent
    [   14.329502] localhost ignition[716]: disks: createPartitions: op(2): op(6): executing: "udevadm" "trigger" "--settle" "/dev/sda"
    [   14.413842] localhost ignition[716]: disks: createPartitions: op(2): op(6): [finished] waiting for triggered uevent
    [   16.407817] localhost kernel: ata2.00: Enabling discard_zeroes_data
    [   14.431452] localhost ignition[716]: disks: createPartitions: op(2): [finished] partitioning "/run/ignition/dev_aliases/dev/sda"
    [   14.446551] localhost ignition[716]: disks: createPartitions: op(7): [started]  partitioning "/run/ignition/dev_aliases/dev/sdb"
    [   14.460542] localhost ignition[716]: disks: createPartitions: op(7): wiping partition table requested on "/run/ignition/dev_aliases/dev/sdb"
    [   14.475439] localhost ignition[716]: disks: createPartitions: op(7): running sgdisk with options: [--zap-all /run/ignition/dev_aliases/dev/sdb]
    [   14.490484] localhost ignition[716]: disks: createPartitions: op(7): op(8): [started]  deleting 0 partitions and creating 0 partitions on "/run/ignition/dev_aliases/dev/sdb"
    [   14.508466] localhost ignition[716]: disks: createPartitions: op(7): op(8): executing: "sgdisk" "--zap-all" "/run/ignition/dev_aliases/dev/sdb"
    [   17.414721] localhost kernel: ata2.00: Enabling discard_zeroes_data
    [   17.420529] localhost kernel: ata2.00: Enabling discard_zeroes_data
    [   15.433681] localhost ignition[716]: disks: createPartitions: op(7): op(8): [finished] deleting 0 partitions and creating 0 partitions on "/run/ignition/dev_aliases/dev/sdb"
    [   17.426878] localhost kernel: ata2.00: Enabling discard_zeroes_data
    [   15.457445] localhost ignition[716]: disks: createPartitions: op(7): op(9): [started]  reading partition table of "/run/ignition/dev_aliases/dev/sdb"
    [   15.472451] localhost ignition[716]: disks: createPartitions: op(7): op(9): [finished] reading partition table of "/run/ignition/dev_aliases/dev/sdb"
    [   15.487438] localhost ignition[716]: disks: createPartitions: op(7): running sgdisk with options: [--pretend --new=1:0:+41943040 --new=2:0:+0 --info=2 /run/ignition/dev_aliases/dev/sdb]
    [   15.507467] localhost ignition[716]: disks: createPartitions: op(7): running sgdisk with options: [--new=1:0:+41943040 --change-name=1:LOGS --new=2:0:+895757967 --change-name=2:OPT /run/ignition/dev_aliases/dev/sdb]
    [   15.530438] localhost ignition[716]: disks: createPartitions: op(7): op(a): [started]  deleting 0 partitions and creating 2 partitions on "/run/ignition/dev_aliases/dev/sdb"
    [   15.548477] localhost ignition[716]: disks: createPartitions: op(7): op(a): executing: "sgdisk" "--new=1:0:+41943040" "--change-name=1:LOGS" "--new=2:0:+895757967" "--change-name=2:OPT" "/run/ignition/dev_aliases/dev/sdb"
    [   18.436665] localhost kernel: ata2.00: Enabling discard_zeroes_data
    [   16.454012] localhost ignition[716]: disks: createPartitions: op(7): op(a): [finished] deleting 0 partitions and creating 2 partitions on "/run/ignition/dev_aliases/dev/sdb"
    [   18.443250] localhost kernel:  sdb: sdb1 sdb2
    [   16.472497] localhost ignition[716]: disks: createPartitions: op(7): op(b): [started]  waiting for triggered uevent
    [   16.484496] localhost ignition[716]: disks: createPartitions: op(7): op(b): executing: "udevadm" "trigger" "--settle" "/dev/sdb"
    [   16.541554] localhost ignition[716]: disks: createPartitions: op(7): op(b): [finished] waiting for triggered uevent
    [   16.554505] localhost ignition[716]: disks: createPartitions: op(7): [finished] partitioning "/run/ignition/dev_aliases/dev/sdb"
    [   16.569483] localhost ignition[716]: disks: createFilesystems: op(c): [started]  waiting for devices [/dev/disk/by-partlabel/OPT /dev/disk/by-partlabel/LOGS]
    [   16.586473] localhost ignition[716]: disks: createFilesystems: op(c): [finished] waiting for devices [/dev/disk/by-partlabel/OPT /dev/disk/by-partlabel/LOGS]
    [   16.603516] localhost ignition[716]: disks: createFilesystems: created device alias for "/dev/disk/by-partlabel/OPT": "/run/ignition/dev_aliases/dev/disk/by-partlabel/OPT" -> "/dev/sdb2"
    [   16.623508] localhost ignition[716]: disks: createFilesystems: created device alias for "/dev/disk/by-partlabel/LOGS": "/run/ignition/dev_aliases/dev/disk/by-partlabel/LOGS" -> "/dev/sdb1"
    [   16.643516] localhost ignition[716]: disks: createFilesystems: op(d): [started]  determining filesystem type of "/dev/disk/by-partlabel/OPT"
    [   16.658532] localhost ignition[716]: disks: createFilesystems: op(d): op(e): [started]  determining filesystem type of "/dev/disk/by-partlabel/LOGS"
    [   16.674527] localhost ignition[716]: disks: createFilesystems: op(d): op(e): [finished] determining filesystem type of "/dev/disk/by-partlabel/OPT"
    [   16.690515] localhost ignition[716]: disks: createFilesystems: op(d): found  filesystem at "/dev/disk/by-partlabel/OPT" with uuid "" and label ""
    [   16.706544] localhost ignition[716]: disks: createFilesystems: op(d): op(f): [started]  wiping filesystem signatures from "/run/ignition/dev_aliases/dev/disk/by-partlabel/OPT"
    [   16.725534] localhost ignition[716]: disks: createFilesystems: op(d): op(f): executing: "wipefs" "-a" "/run/ignition/dev_aliases/dev/disk/by-partlabel/OPT"
    [   16.725570] localhost ignition[716]: disks: createFilesystems: op(d): op(f): [finished] determining filesystem type of "/dev/disk/by-partlabel/LOGS"
    [   16.741531] localhost ignition[716]: disks: createFilesystems: op(d): found  filesystem at "/dev/disk/by-partlabel/LOGS" with uuid "" and label ""
    [   16.757521] localhost ignition[716]: disks: createFilesystems: op(d): op(10): [started]  wiping filesystem signatures from "/run/ignition/dev_aliases/dev/disk/by-partlabel/LOGS"
    [   16.776531] localhost ignition[716]: disks: createFilesystems: op(d): op(10): executing: "wipefs" "-a" "/run/ignition/dev_aliases/dev/disk/by-partlabel/LOGS"
    [   16.776566] localhost ignition[716]: disks: createFilesystems: op(d): op(10): [finished] wiping filesystem signatures from "/run/ignition/dev_aliases/dev/disk/by-partlabel/OPT"
    [   16.795533] localhost ignition[716]: disks: createFilesystems: op(d): op(11): [started]  creating "ext4" filesystem on "/run/ignition/dev_aliases/dev/disk/by-partlabel/OPT"
    [   16.814534] localhost ignition[716]: disks: createFilesystems: op(d): op(11): executing: "mkfs.ext4" "-F" "-L" "OPT" "/run/ignition/dev_aliases/dev/disk/by-partlabel/OPT"
    [   16.814570] localhost ignition[716]: disks: createFilesystems: op(d): op(11): [finished] wiping filesystem signatures from "/run/ignition/dev_aliases/dev/disk/by-partlabel/LOGS"
    [   16.833525] localhost ignition[716]: disks: createFilesystems: op(d): op(12): [started]  creating "ext4" filesystem on "/run/ignition/dev_aliases/dev/disk/by-partlabel/LOGS"
    [   16.852541] localhost ignition[716]: disks: createFilesystems: op(d): op(12): executing: "mkfs.ext4" "-F" "-L" "LOGS" "/run/ignition/dev_aliases/dev/disk/by-partlabel/LOGS"
    [   19.383318] localhost ignition[716]: disks: createFilesystems: op(d): op(12): [finished] creating "ext4" filesystem on "/run/ignition/dev_aliases/dev/disk/by-partlabel/LOGS"
    [   19.402505] localhost ignition[716]: disks: createFilesystems: op(d): op(13): [started]  waiting for triggered uevent
    [   19.414444] localhost ignition[716]: disks: createFilesystems: op(d): op(13): executing: "udevadm" "trigger" "--settle" "/dev/sdb1"
    [   20.971192] localhost ignition[716]: disks: createFilesystems: op(d): op(13): [finished] waiting for triggered uevent
    [   27.423439] localhost ignition[716]: disks: createFilesystems: op(d): [finished] creating "ext4" filesystem on "/run/ignition/dev_aliases/dev/disk/by-partlabel/OPT"
    [   27.440446] localhost ignition[716]: disks: createFilesystems: op(14): [started]  waiting for triggered uevent
    [   27.453439] localhost ignition[716]: disks: createFilesystems: op(14): executing: "udevadm" "trigger" "--settle" "/dev/sdb2"
    [   27.458367] localhost ignition[716]: disks: createFilesystems: op(14): [finished] waiting for triggered uevent
    [   27.470464] localhost ignition[716]: disks: disks passed
    ...
    135.715462] localhost dracut-initqueue[566]: Warning: dracut-initqueue: timeout, still waiting for following initqueue hooks:
    [  135.729452] localhost dracut-initqueue[566]: Warning: /lib/dracut/hooks/initqueue/finished/devexists-\x2fdev\x2fdisk\x2fby-label\x2froot.sh: "if ! grep -q After=remote-fs-pre.target /run/systemd/generator/systemd-cryptsetup@*.service 2>/dev/null; then
    [  135.729452] localhost dracut-initqueue[566]:     [ -e "/dev/disk/by-label/root" ]
    [  135.729452] localhost dracut-initqueue[566]: Warning: dracut-initqueue: starting timeout scripts
    [  136.249388] localhost dracut-initqueue[566]: Warning: dracut-initqueue: timeout, still waiting for following initqueue hooks:
    [  136.263449] localhost dracut-initqueue[566]: Warning: /lib/dracut/hooks/initqueue/finished/devexists-\x2fdev\x2fdisk\x2fby-label\x2froot.sh: "if ! grep -q After=remote-fs-pre.target /run/systemd/generator/systemd-cryptsetup@*.service 2>/dev/null; then
    [  136.263449] localhost dracut-initqueue[566]:     [ -e "/dev/disk/by-label/root" ]
    [  136.2634
    ...
    [  197.730465] localhost dracut-initqueue[566]: Warning: dracut-initqueue: starting timeout scripts
    [  198.241427] localhost dracut-initqueue[566]: Warning: dracut-initqueue: timeout, still waiting for following initqueue hooks:
    [  198.255456] localhost dracut-initqueue[566]: Warning: /lib/dracut/hooks/initqueue/finished/devexists-\x2fdev\x2fdisk\x2fby-label\x2froot.sh: "if ! grep -q After=remote-fs-pre.target /run/systemd/generator/systemd-cryptsetup@*.service 2>/dev/null; then
    [  198.255456] localhost dracut-initqueue[566]:     [ -e "/dev/disk/by-label/root" ]
    [  198.255456] localhost dracut-initqueue[566]: Warning: dracut-initqueue: starting timeout scripts
    [  198.255456] localhost dracut-initqueue[566]: Warning: Could not boot.
    [  198.263659] localhost systemd[1]: Starting Dracut Emergency Shell...

    Expected behavior

Machine should boot. Root FS is mounted to disk as expected. Additional partitions / filesystems should be created as expected.

Additional information

Behavior can also be reproduced using QEMU, similarly specifying --kernel and --initrd options, so I don't believe the issue is platform specific.

Configuration

storage:
  disks:
    - device: /dev/sda
      wipe_table: true
      partitions:
      - label: root
        number: 1
        size: 0
    - device: /dev/sdb
      wipe_table: true
      partitions:
      - label: LOGS
        number: 1
        size: 20GiB
      - label: OPT
        number: 2
        size: 0

  filesystems:
    - name: root
      mount:
        device: /dev/disk/by-partlabel/root
        format: ext4
        wipe_filesystem: true
        label: root
    - name: OPT
      mount:
        device: /dev/disk/by-partlabel/OPT
        format: ext4
        wipe_filesystem: true
        label: OPT
    - name: LOGS
      mount:
        device: /dev/disk/by-partlabel/LOGS
        format: ext4
        wipe_filesystem: true
        label: LOGS

ipxe Script

#!ipxe

set base-url https://stable.release.flatcar-linux.net/amd64-usr/3227.2.4

kernel ${base-url}/flatcar_production_pxe.vmlinuz initrd=flatcar_production_pxe_image.cpio.gz root=LABEL=root flatcar.first_boot=1  flatcar.oem.id=packet console=ttyS1,115200n8 flatcar.autologin ignition.config.url=<URL>
initrd ${base-url}/flatcar_production_pxe_image.cpio.gz
boot

Successful Boot (Stable / 3139.2.3)

Oct 17 21:26:57 localhost ignition[447]: disks: createPartitions: op(1): [started]  waiting for devices [/dev/sda /dev/sdb]
Oct 17 21:26:57 localhost ignition[447]: disks: createPartitions: op(1): [finished] waiting for devices [/dev/sda /dev/sdb]
Oct 17 21:26:57 localhost ignition[447]: disks: createPartitions: created device alias for "/dev/sda": "/run/ignition/dev_aliases/dev/sda" -> "/dev/sda"
Oct 17 21:26:57 localhost ignition[447]: disks: createPartitions: created device alias for "/dev/sdb": "/run/ignition/dev_aliases/dev/sdb" -> "/dev/sdb"
Oct 17 21:26:57 localhost ignition[447]: disks: createPartitions: op(2): [started]  partitioning "/run/ignition/dev_aliases/dev/sda"
Oct 17 21:26:57 localhost ignition[447]: disks: createPartitions: op(2): wiping partition table requested on "/run/ignition/dev_aliases/dev/sda"
Oct 17 21:26:57 localhost ignition[447]: disks: createPartitions: op(2): running sgdisk with options: [--zap-all /run/ignition/dev_aliases/dev/sda]
Oct 17 21:26:57 localhost ignition[447]: disks: createPartitions: op(2): op(3): [started]  deleting 0 partitions and creating 0 partitions on "/run/ignition/dev_aliases/dev/sda"
Oct 17 21:26:57 localhost ignition[447]: disks: createPartitions: op(2): op(3): executing: "/usr/sbin/sgdisk" "--zap-all" "/run/ignition/dev_aliases/dev/sda"
Oct 17 21:26:58 localhost ignition[447]: disks: createPartitions: op(2): op(3): [finished] deleting 0 partitions and creating 0 partitions on "/run/ignition/dev_aliases/dev/sda"
Oct 17 21:26:58 localhost ignition[447]: disks: createPartitions: op(2): op(4): [started]  reading partition table of "/run/ignition/dev_aliases/dev/sda"
Oct 17 21:26:58 localhost ignition[447]: disks: createPartitions: op(2): op(4): [finished] reading partition table of "/run/ignition/dev_aliases/dev/sda"
Oct 17 21:26:58 localhost ignition[447]: disks: createPartitions: op(2): running sgdisk with options: [--pretend --new=1:0:+0M --info=1 /run/ignition/dev_aliases/dev/sda]
Oct 17 21:26:58 localhost ignition[447]: disks: createPartitions: op(2): running sgdisk with options: [--new=1:0:+0M --change-name=1:root /run/ignition/dev_aliases/dev/sda]
Oct 17 21:26:58 localhost ignition[447]: disks: createPartitions: op(2): op(5): [started]  deleting 0 partitions and creating 1 partitions on "/run/ignition/dev_aliases/dev/sda"
Oct 17 21:26:58 localhost ignition[447]: disks: createPartitions: op(2): op(5): executing: "/usr/sbin/sgdisk" "--new=1:0:+0M" "--change-name=1:root" "/run/ignition/dev_aliases/dev/sda"
Oct 17 21:26:59 localhost ignition[447]: disks: createPartitions: op(2): op(5): [finished] deleting 0 partitions and creating 1 partitions on "/run/ignition/dev_aliases/dev/sda"
Oct 17 21:26:59 localhost ignition[447]: disks: createPartitions: op(2): op(6): [started]  waiting for triggered uevent
Oct 17 21:26:59 localhost ignition[447]: disks: createPartitions: op(2): op(6): executing: "/usr/bin/udevadm" "trigger" "--settle" "/dev/sda"
Oct 17 21:26:59 localhost ignition[447]: disks: createPartitions: op(2): op(6): [finished] waiting for triggered uevent
Oct 17 21:26:59 localhost ignition[447]: disks: createPartitions: op(2): [finished] partitioning "/run/ignition/dev_aliases/dev/sda"
Oct 17 21:26:59 localhost ignition[447]: disks: createPartitions: op(7): [started]  partitioning "/run/ignition/dev_aliases/dev/sdb"
Oct 17 21:26:59 localhost ignition[447]: disks: createPartitions: op(7): wiping partition table requested on "/run/ignition/dev_aliases/dev/sdb"
Oct 17 21:26:59 localhost ignition[447]: disks: createPartitions: op(7): running sgdisk with options: [--zap-all /run/ignition/dev_aliases/dev/sdb]
Oct 17 21:26:59 localhost ignition[447]: disks: createPartitions: op(7): op(8): [started]  deleting 0 partitions and creating 0 partitions on "/run/ignition/dev_aliases/dev/sdb"
Oct 17 21:26:59 localhost ignition[447]: disks: createPartitions: op(7): op(8): executing: "/usr/sbin/sgdisk" "--zap-all" "/run/ignition/dev_aliases/dev/sdb"
Oct 17 21:27:00 localhost ignition[447]: disks: createPartitions: op(7): op(8): [finished] deleting 0 partitions and creating 0 partitions on "/run/ignition/dev_aliases/dev/sdb"
Oct 17 21:27:00 localhost ignition[447]: disks: createPartitions: op(7): op(9): [started]  reading partition table of "/run/ignition/dev_aliases/dev/sdb"
Oct 17 21:27:00 localhost ignition[447]: disks: createPartitions: op(7): op(9): [finished] reading partition table of "/run/ignition/dev_aliases/dev/sdb"
Oct 17 21:27:00 localhost ignition[447]: disks: createPartitions: op(7): running sgdisk with options: [--pretend --new=1:0:+20480M --new=2:0:+0M --info=2 /run/ignition/dev_aliases/dev/sdb]
Oct 17 21:27:00 localhost ignition[447]: disks: createPartitions: op(7): running sgdisk with options: [--new=1:0:+20480M --change-name=1:LOGS --new=2:0:+0M --change-name=2:OPT /run/ignition/dev_aliases/dev/sdb]
Oct 17 21:27:00 localhost ignition[447]: disks: createPartitions: op(7): op(a): [started]  deleting 0 partitions and creating 2 partitions on "/run/ignition/dev_aliases/dev/sdb"
Oct 17 21:27:00 localhost ignition[447]: disks: createPartitions: op(7): op(a): executing: "/usr/sbin/sgdisk" "--new=1:0:+20480M" "--change-name=1:LOGS" "--new=2:0:+0M" "--change-name=2:OPT" "/run/ignition/dev_aliases/dev/sdb"
Oct 17 21:27:01 localhost ignition[447]: disks: createPartitions: op(7): op(a): [finished] deleting 0 partitions and creating 2 partitions on "/run/ignition/dev_aliases/dev/sdb"
Oct 17 21:27:01 localhost ignition[447]: disks: createPartitions: op(7): op(b): [started]  waiting for triggered uevent
Oct 17 21:27:01 localhost ignition[447]: disks: createPartitions: op(7): op(b): executing: "/usr/bin/udevadm" "trigger" "--settle" "/dev/sdb"
Oct 17 21:27:01 localhost ignition[447]: disks: createPartitions: op(7): op(b): [finished] waiting for triggered uevent
Oct 17 21:27:01 localhost ignition[447]: disks: createPartitions: op(7): [finished] partitioning "/run/ignition/dev_aliases/dev/sdb"
Oct 17 21:27:01 localhost ignition[447]: disks: createFilesystems: op(c): [started]  waiting for devices [/dev/disk/by-partlabel/root /dev/disk/by-partlabel/OPT /dev/disk/by-partlabel/LOGS]
Oct 17 21:27:01 localhost ignition[447]: disks: createFilesystems: op(c): [finished] waiting for devices [/dev/disk/by-partlabel/root /dev/disk/by-partlabel/OPT /dev/disk/by-partlabel/LOGS]
Oct 17 21:27:01 localhost ignition[447]: disks: createFilesystems: created device alias for "/dev/disk/by-partlabel/root": "/run/ignition/dev_aliases/dev/disk/by-partlabel/root" -> "/dev/sda1"
Oct 17 21:27:01 localhost ignition[447]: disks: createFilesystems: created device alias for "/dev/disk/by-partlabel/OPT": "/run/ignition/dev_aliases/dev/disk/by-partlabel/OPT" -> "/dev/sdb2"
Oct 17 21:27:01 localhost ignition[447]: disks: createFilesystems: created device alias for "/dev/disk/by-partlabel/LOGS": "/run/ignition/dev_aliases/dev/disk/by-partlabel/LOGS" -> "/dev/sdb1"
Oct 17 21:27:01 localhost ignition[447]: disks: createFilesystems: op(d): [started]  determining filesystem type of "/dev/disk/by-partlabel/root"
Oct 17 21:27:01 localhost ignition[447]: disks: createFilesystems: op(d): op(e): [started]  determining filesystem type of "/dev/disk/by-partlabel/OPT"
Oct 17 21:27:01 localhost ignition[447]: disks: createFilesystems: op(d): op(e): op(f): [started]  determining filesystem type of "/dev/disk/by-partlabel/LOGS"
Oct 17 21:27:01 localhost ignition[447]: disks: createFilesystems: op(d): op(e): op(f): found  filesystem at "/dev/disk/by-partlabel/root" with uuid "" and label ""
Oct 17 21:27:01 localhost ignition[447]: disks: createFilesystems: op(d): op(e): op(f): [finished] determining filesystem type of "/dev/disk/by-partlabel/root"
Oct 17 21:27:01 localhost ignition[447]: disks: createFilesystems: op(d): op(e): op(10): [started]  creating "ext4" filesystem on "/run/ignition/dev_aliases/dev/disk/by-partlabel/root"
Oct 17 21:27:01 localhost ignition[447]: disks: createFilesystems: op(d): op(e): op(10): executing: "/usr/sbin/mkfs.ext4" "-F" "-L" "root" "/run/ignition/dev_aliases/dev/disk/by-partlabel/root"
Oct 17 21:27:01 localhost ignition[447]: disks: createFilesystems: op(d): op(e): op(10): found  filesystem at "/dev/disk/by-partlabel/LOGS" with uuid "" and label ""
Oct 17 21:27:01 localhost ignition[447]: disks: createFilesystems: op(d): op(e): op(10): [finished] determining filesystem type of "/dev/disk/by-partlabel/LOGS"
Oct 17 21:27:01 localhost ignition[447]: disks: createFilesystems: op(d): op(e): op(11): [started]  creating "ext4" filesystem on "/run/ignition/dev_aliases/dev/disk/by-partlabel/LOGS"
Oct 17 21:27:01 localhost ignition[447]: disks: createFilesystems: op(d): op(e): op(11): executing: "/usr/sbin/mkfs.ext4" "-F" "-L" "LOGS" "/run/ignition/dev_aliases/dev/disk/by-partlabel/LOGS"
Oct 17 21:27:01 localhost ignition[447]: disks: createFilesystems: op(d): op(e): op(11): found  filesystem at "/dev/disk/by-partlabel/OPT" with uuid "" and label ""
Oct 17 21:27:01 localhost ignition[447]: disks: createFilesystems: op(d): op(e): op(11): [finished] determining filesystem type of "/dev/disk/by-partlabel/OPT"
Oct 17 21:27:01 localhost ignition[447]: disks: createFilesystems: op(d): op(e): op(12): [started]  creating "ext4" filesystem on "/run/ignition/dev_aliases/dev/disk/by-partlabel/OPT"
Oct 17 21:27:01 localhost ignition[447]: disks: createFilesystems: op(d): op(e): op(12): executing: "/usr/sbin/mkfs.ext4" "-F" "-L" "OPT" "/run/ignition/dev_aliases/dev/disk/by-partlabel/OPT"
Oct 17 21:27:04 localhost ignition[447]: disks: createFilesystems: op(d): op(e): op(12): [finished] creating "ext4" filesystem on "/run/ignition/dev_aliases/dev/disk/by-partlabel/LOGS"
Oct 17 21:27:04 localhost ignition[447]: disks: createFilesystems: op(d): op(e): op(13): [started]  waiting for triggered uevent
Oct 17 21:27:04 localhost ignition[447]: disks: createFilesystems: op(d): op(e): op(13): executing: "/usr/bin/udevadm" "trigger" "--settle" "/dev/sdb1"
Oct 17 21:27:05 localhost ignition[447]: disks: createFilesystems: op(d): op(e): op(13): [finished] waiting for triggered uevent
Oct 17 21:27:11 localhost ignition[447]: disks: createFilesystems: op(d): op(e): [finished] creating "ext4" filesystem on "/run/ignition/dev_aliases/dev/disk/by-partlabel/root"
Oct 17 21:27:11 localhost ignition[447]: disks: createFilesystems: op(d): op(14): [started]  waiting for triggered uevent
Oct 17 21:27:11 localhost ignition[447]: disks: createFilesystems: op(d): op(14): executing: "/usr/bin/udevadm" "trigger" "--settle" "/dev/sda1"
Oct 17 21:27:12 localhost ignition[447]: disks: createFilesystems: op(d): op(14): [finished] waiting for triggered uevent
Oct 17 21:27:12 localhost ignition[447]: disks: createFilesystems: op(d): [finished] creating "ext4" filesystem on "/run/ignition/dev_aliases/dev/disk/by-partlabel/OPT"
Oct 17 21:27:12 localhost ignition[447]: disks: createFilesystems: op(15): [started]  waiting for triggered uevent
Oct 17 21:27:12 localhost ignition[447]: disks: createFilesystems: op(15): executing: "/usr/bin/udevadm" "trigger" "--settle" "/dev/sdb2"
Oct 17 21:27:12 localhost ignition[447]: disks: createFilesystems: op(15): [finished] waiting for triggered uevent
Oct 17 21:27:12 localhost ignition[447]: disks: disks passed
tormath1 commented 2 years ago

Hi @dahicks,

Thanks for the clear bug report - we can clearly see that the following line is missing on the 3227.2.4 logs:

Oct 17 21:27:01 localhost ignition[447]: disks: createFilesystems: op(d): op(e): op(10): executing: "/usr/sbin/mkfs.ext4" "-F" "-L" "root" "/run/ignition/dev_aliases/dev/disk/by-partlabel/root"

Between 3139.2.3 and 3227.2.4 there is a major bump in the Ignition version: while we tried to provide the seamless transition, it seems that reformatting the filesystem named "root" is now a no-op operation (https://github.com/flatcar/ign-converter/blob/b03b4e55d1f74d81af0a1a6528e87fbb4ea56879/translate/v24tov31/v24tov31.go#L451-L454).

So your configuration is translated to:

Filesystem "root" shadows exising filesystem definition{
  "ignition": {
    "version": "3.1.0"
  },
  "storage": {
    "disks": [
      {
        "device": "/dev/sda",
        "partitions": [
          {
            "label": "root",
            "number": 1,
            "sizeMiB": 0
          }
        ],
        "wipeTable": true
      },
      {
        "device": "/dev/sdb",
        "partitions": [
          {
            "label": "LOGS",
            "number": 1,
            "sizeMiB": 20480
          },
          {
            "label": "OPT",
            "number": 2,
            "sizeMiB": 0
          }
        ],
        "wipeTable": true
      }
    ],
    "filesystems": [
      {
        "device": "/dev/disk/by-partlabel/OPT",
        "format": "ext4",
        "label": "OPT",
        "path": "/tmp/OPT-ign1",
        "wipeFilesystem": true
      },
      {
        "device": "/dev/disk/by-partlabel/LOGS",
        "format": "ext4",
        "label": "LOGS",
        "path": "/tmp/LOGS-ign2",
        "wipeFilesystem": true
      }
    ]
  }
}

Where we can see that the root is simply ignored.

What you can do: try to use a different name for the root filesystem. Let us know how it goes :)

pothos commented 2 years ago

Our tests are using ROOT: https://github.com/flatcar/mantle/blob/flatcar-master/kola/tests/ignition/filesystem.go#L34 This label is also documented in https://www.flatcar.org/docs/latest/reference/developer-guides/sdk-disk-partitions/ (Edit: An example config can be found in https://www.flatcar.org/docs/latest/setup/storage/raid/#example-container-linux-config) Maybe we can fix that case in ign-converter, but for the mean time I suggest using ROOT instead of root.

dahicks commented 2 years ago

Thanks for looking into this so quickly! Changing to something other than root does indeed work.

pothos commented 2 years ago

I guess we can leave this open, or?

dahicks commented 2 years ago

I'm fine with leaving it open, if y'all are planning a fix vs the workaround.