flatcar-install fails on CentOS7

[till]

Description

I booted a server using CentOS7 and tried to run flatcar-install, but it fails to complete.

Impact

Would love to install flatcar linux.

Environment and steps to reproduce

1. Set-up:

   • booted hetzner cloud instance with CentOS7
   • downloaded Flatcar ISO (there are other issues verifying the iso)

2. Task: installing Flatcar linux

3. Action(s): a. ./flatcar-install -d /dev/sda1 -i /root/ignition.json -f ./flatcar_production_iso_image.iso -v

Output:

+ getopts V:B:C:I:d:o:c:e:i:t:b:k:f:nsyvh OPTION
+ [[ -z /dev/sda1 ]]
+ [[ -n '' ]]
++ lsblk -n -d -o TYPE /dev/sda1
+ [[ part =~ ^(disk|loop|lvm)$ ]]
+ echo './flatcar-install: Target block device (/dev/sda1) is not a full disk.'
./flatcar-install: Target block device (/dev/sda1) is not a full disk.
+ exit 1
[root@node-001 ~]# ./flatcar-install -d /dev/sda -i /root/ignition.json -f ./flatcar_production_iso_image.iso -v
+ getopts V:B:C:I:d:o:c:e:i:t:b:k:f:nsyvh OPTION
+ [[ -z /dev/sda ]]
+ [[ -n '' ]]
++ lsblk -n -d -o TYPE /dev/sda
+ [[ disk =~ ^(disk|loop|lvm)$ ]]
+ [[ ! -w /dev/sda ]]
+ [[ -n '' ]]
+ [[ -n /root/ignition.json ]]
+ [[ ! -f /root/ignition.json ]]
+ [[ -n '' ]]
+ _disk_status=
++ mktemp --tmpdir -d flatcar-install.XXXXXXXXXX
+ WORKDIR=/tmp/flatcar-install.cJNz6cYTpw
+ trap 'error_output ; is_modified && wipefs --all --backup "${DEVICE}" ; rm -rf "${WORKDIR}"' EXIT
+ '[' -n ./flatcar_production_iso_image.iso ']'
+ install_from_file
+ '[' -r ./flatcar_production_iso_image.iso ']'
+ echo 'Writing ./flatcar_production_iso_image.iso...'
Writing ./flatcar_production_iso_image.iso...
+ [[ ./flatcar_production_iso_image.iso =~ \.bz2$ ]]
+ write_to_disk
+ mkfifo -m 0600 /tmp/flatcar-install.cJNz6cYTpw/disk_modified
+ trap '(exec 2>/dev/null ; echo done > "${WORKDIR}/disk_modified") &' RETURN
++ blockdev --getsz /dev/sda
+ dd conv=nocreat count=1024 if=/dev/zero of=/dev/sda seek=80002048 status=none
+ dd bs=1M conv=nocreat of=/dev/sda status=none
+ udevadm settle
+ local try
+ for try in 0 1 2 4
+ sleep 0
+ blockdev --rereadpt /dev/sda
blockdev: ioctl error on BLKRRPART: Device or resource busy
+ echo 'Failed to reread partitions on /dev/sda'
Failed to reread partitions on /dev/sda
+ for try in 0 1 2 4
+ sleep 1
+ blockdev --rereadpt /dev/sda
blockdev: ioctl error on BLKRRPART: Device or resource busy
+ echo 'Failed to reread partitions on /dev/sda'
Failed to reread partitions on /dev/sda
+ for try in 0 1 2 4
+ sleep 2
+ blockdev --rereadpt /dev/sda
blockdev: ioctl error on BLKRRPART: Device or resource busy
+ echo 'Failed to reread partitions on /dev/sda'
Failed to reread partitions on /dev/sda
+ for try in 0 1 2 4
+ sleep 4
+ blockdev --rereadpt /dev/sda
blockdev: ioctl error on BLKRRPART: Device or resource busy
+ echo 'Failed to reread partitions on /dev/sda'
Failed to reread partitions on /dev/sda
+ '[' -z 4 ']'
+ exit 1
+ error_output
+ echo 'Error: return code 1 from exit 1'
Error: return code 1 from exit 1
+ is_modified
+ [[ -e /tmp/flatcar-install.cJNz6cYTpw/disk_modified ]]
+ wipefs --all --backup /dev/sda
wipefs: unrecognized option '--backup'

Usage:
 wipefs [options] <device>

Options:
 -a, --all           wipe all magic strings (BE CAREFUL!)
 -f, --force         force erasure
 -h, --help          show this help text
 -n, --no-act        do everything except the actual write() call
 -o, --offset <num>  offset to erase, in bytes
 -p, --parsable      print out in parsable instead of printable format
 -q, --quiet         suppress output messages
 -t, --types <list>  limit the set of filesystem, RAIDs or partition tables
 -V, --version       output version information and exit

For more information see wipefs(8).

4. Error:

   • not entirely sure if the first "error" is the busy device
   • or maybe usage of wipefs

Expected behavior

flatcar-install completes, and I can boot into Flatcar Linux.

[till]

When I boot into CentOS8 and re-run my steps:

# ./flatcar-install -d /dev/sda -i /root/ignition.json -f ./flatcar_production_iso_image.iso -v
+ getopts V:B:C:I:d:o:c:e:i:t:b:k:f:nsyvh OPTION
+ [[ -z /dev/sda ]]
+ [[ -n '' ]]
++ lsblk -n -d -o TYPE /dev/sda
+ [[ disk =~ ^(disk|loop|lvm)$ ]]
+ [[ ! -w /dev/sda ]]
+ [[ -n '' ]]
+ [[ -n /root/ignition.json ]]
+ [[ ! -f /root/ignition.json ]]
+ [[ -n '' ]]
+ _disk_status=
++ mktemp --tmpdir -d flatcar-install.XXXXXXXXXX
+ WORKDIR=/tmp/flatcar-install.GXy8SI3sqB
+ trap 'error_output ; is_modified && wipefs --all --backup "${DEVICE}" ; rm -rf "${WORKDIR}"' EXIT
+ '[' -n ./flatcar_production_iso_image.iso ']'
+ install_from_file
+ '[' -r ./flatcar_production_iso_image.iso ']'
+ echo 'Writing ./flatcar_production_iso_image.iso...'
Writing ./flatcar_production_iso_image.iso...
+ [[ ./flatcar_production_iso_image.iso =~ \.bz2$ ]]
+ write_to_disk
+ mkfifo -m 0600 /tmp/flatcar-install.GXy8SI3sqB/disk_modified
+ trap '(exec 2>/dev/null ; echo done > "${WORKDIR}/disk_modified") &' RETURN
++ blockdev --getsz /dev/sda
+ dd conv=nocreat count=1024 if=/dev/zero of=/dev/sda seek=80002048 status=none
+ dd bs=1M conv=nocreat of=/dev/sda status=none
+ udevadm settle
+ local try
+ for try in 0 1 2 4
+ sleep 0
+ blockdev --rereadpt /dev/sda
blockdev: ioctl error on BLKRRPART: Device or resource busy
+ echo 'Failed to reread partitions on /dev/sda'
Failed to reread partitions on /dev/sda
+ for try in 0 1 2 4
+ sleep 1
+ blockdev --rereadpt /dev/sda
blockdev: ioctl error on BLKRRPART: Device or resource busy
+ echo 'Failed to reread partitions on /dev/sda'
Failed to reread partitions on /dev/sda
+ for try in 0 1 2 4
+ sleep 2
+ blockdev --rereadpt /dev/sda
blockdev: ioctl error on BLKRRPART: Device or resource busy
+ echo 'Failed to reread partitions on /dev/sda'
Failed to reread partitions on /dev/sda
+ for try in 0 1 2 4
+ sleep 4
+ blockdev --rereadpt /dev/sda
blockdev: ioctl error on BLKRRPART: Device or resource busy
+ echo 'Failed to reread partitions on /dev/sda'
Failed to reread partitions on /dev/sda
+ '[' -z 4 ']'
+ exit 1
+ error_output
+ echo 'Error: return code 1 from exit 1'
Error: return code 1 from exit 1
+ is_modified
+ [[ -e /tmp/flatcar-install.GXy8SI3sqB/disk_modified ]]
+ wipefs --all --backup /dev/sda
wipefs: error: /dev/sda: probing initialization failed: Device or resource busy

[pothos]

Hello, thanks for the report and trying it out on CentOS8, too. There is something similar: https://github.com/flatcar-linux/Flatcar/issues/120 Do you have any systemd-fsck services started at that time (see sudo journalctl -e) and could you try the workaround systemctl mask systemd-fsck@.service mentioned in the CoreOS issue?

[till]

I think it didn't help.

[root@node-001 ~]# systemctl mask systemd-fsck@.service
Created symlink /etc/systemd/system/systemd-fsck@.service → /dev/null.
[root@node-001 ~]# ./flatcar-install -d /dev/sda -i /root/ignition.json -f ./flatcar_production_iso_image.iso -v
+ getopts V:B:C:I:d:o:c:e:i:t:b:k:f:nsyvh OPTION
+ [[ -z /dev/sda ]]
+ [[ -n '' ]]
++ lsblk -n -d -o TYPE /dev/sda
+ [[ disk =~ ^(disk|loop|lvm)$ ]]
+ [[ ! -w /dev/sda ]]
+ [[ -n '' ]]
+ [[ -n /root/ignition.json ]]
+ [[ ! -f /root/ignition.json ]]
+ [[ -n '' ]]
+ _disk_status=
++ mktemp --tmpdir -d flatcar-install.XXXXXXXXXX
+ WORKDIR=/tmp/flatcar-install.kimFsog3zh
+ trap 'error_output ; is_modified && wipefs --all --backup "${DEVICE}" ; rm -rf "${WORKDIR}"' EXIT
+ '[' -n ./flatcar_production_iso_image.iso ']'
+ install_from_file
+ '[' -r ./flatcar_production_iso_image.iso ']'
+ echo 'Writing ./flatcar_production_iso_image.iso...'
Writing ./flatcar_production_iso_image.iso...
+ [[ ./flatcar_production_iso_image.iso =~ \.bz2$ ]]
+ write_to_disk
+ mkfifo -m 0600 /tmp/flatcar-install.kimFsog3zh/disk_modified
+ trap '(exec 2>/dev/null ; echo done > "${WORKDIR}/disk_modified") &' RETURN
++ blockdev --getsz /dev/sda
+ dd conv=nocreat count=1024 if=/dev/zero of=/dev/sda seek=80002048 status=none
+ dd bs=1M conv=nocreat of=/dev/sda status=none
+ udevadm settle
+ local try
+ for try in 0 1 2 4
+ sleep 0
+ blockdev --rereadpt /dev/sda
blockdev: ioctl error on BLKRRPART: Device or resource busy
+ echo 'Failed to reread partitions on /dev/sda'
Failed to reread partitions on /dev/sda
+ for try in 0 1 2 4
+ sleep 1
+ blockdev --rereadpt /dev/sda
blockdev: ioctl error on BLKRRPART: Device or resource busy
+ echo 'Failed to reread partitions on /dev/sda'
Failed to reread partitions on /dev/sda
+ for try in 0 1 2 4
+ sleep 2
+ blockdev --rereadpt /dev/sda
blockdev: ioctl error on BLKRRPART: Device or resource busy
+ echo 'Failed to reread partitions on /dev/sda'
Failed to reread partitions on /dev/sda
+ for try in 0 1 2 4
+ sleep 4
+ blockdev --rereadpt /dev/sda
blockdev: ioctl error on BLKRRPART: Device or resource busy
+ echo 'Failed to reread partitions on /dev/sda'
Failed to reread partitions on /dev/sda
+ '[' -z 4 ']'
+ exit 1
+ error_output
+ echo 'Error: return code 1 from exit 1'
Error: return code 1 from exit 1
+ is_modified
+ [[ -e /tmp/flatcar-install.kimFsog3zh/disk_modified ]]
+ wipefs --all --backup /dev/sda
wipefs: error: /dev/sda: probing initialization failed: Device or resource busy

To add more context, here is what I am doing:

• I am booting an instance on Hetzner Cloud (with CentOS 8)
• I ssh into it, and attempt to run flatcar-install

Maybe that's not how it can work in general? Reason why I am trying this without mounting Flatcar (for PXE) is that I can't control anything when it PXE boots on Hetzner Cloud (it boots, but basically no SSH, just console access — was looking to automate this step a bit). But maybe that is not how it works? Or do you think it should?

[pothos]

Haven't reproduced it, but why are you using the ISO if it's a a regular hard disk? You should use https://stable.release.flatcar-linux.net/amd64-usr/current/flatcar_production_image.bin.bz2 (after bunzip) or just leave the -f imagefile out to have it download the correct image.

In general, you can specify the Ignition config as platform user data for the instance if Ignition already knows about that platform (and the OEM ID is set as kernel parameter) or you can specify the Ignition config as URL (could be a URL to the platform user-data endpoint, too, or a third party server) by setting the ignition.config.url= kernel command line parameter in, e.g., an iPXE script.

[pothos]

Another thing to look out for is whether there are existing LVM partitions that are active. One can try something like lvchange -an /dev/mapper/vg* until logic to cover this is part of flatcar-install.