Packet s3.xlarge.x86 instances can't provision flatcar OS

[kgwaters6]

Description

Provisioning failures of Flatcar OS (Alpha, Beta, Stable, and Edge) on the s3.xlarge. Suspect issue is related to installing on NVMe disks in BIOS mode (not UEFI/EFI).

Impact

Flatcar Container is not working on these instances.

Environment and steps to reproduce

1. Set-up: Select Flatcar for a s3.xlarge instance on Packet
2. Task: Provisioning
3. Action(s): Attempt to provision Flatcare OS (Alpha, Beta, Stable, or Edge) on s3.xlarge
4. Error: Provisioning failure

Expected behavior Provision successful

Additional information Flatcar for s3.xlarge is currently toggled off. When there is a desire to test, let me know and I'll turn it back on.

@marga-kinvolk Happy to provide any additional detail.

[pothos]

Hello, thanks for creating the issue here. Can you tell which Alpha version got installed? The latest Alpha has a fix for the c3.medium and some m2.xlarge instances which probably is relevant: https://www.flatcar-linux.org/releases/#release-2592.0.0 Without the fix the instance would hang in GRUB.

About checking the version: The first update happens after a couple of minutes, so either be fast or disable updates with the following user-data:

{
  "ignition": {
    "version": "2.2.0"
  },
  "systemd": {
    "units": [
      {
        "mask": true,
        "name": "update-engine.service"
      }
    ]
  }
}

[kgwaters6]

Thanks, we have confirmed we have the latest releases for all versions with @t-lo

Some more detail that may be helpful: We see it get this far:

Flatcar Container Linux by Kinvolk stable (2512.2.1) Update Strategy: No Reboots Failed Units: 1 tcsd.service core@localhost ~ $ core@localhost ~ $

Then it reboots again and never loads an OS

Ends with:

Reboot and Select proper Boot device or Insert Boot Media in selected Boot device and press a key

[pothos]

Ah, I see, the GRUB menu never showed up for me and it's only trying FlexBoot.

What I did was booting via iPXE: https://alpha.release.flatcar-linux.net/amd64-usr/current/flatcar_production_packet.ipxe with this user-data:

{
  "ignition": { "version": "2.2.0" },
  "storage": {
    "links": [{
      "filesystem": "root",
      "path": "/etc/systemd/system/multi-user.target.wants/coreos-metadata-sshkeys@core.service",
      "target": "/usr/lib/systemd/system/coreos-metadata-sshkeys@.service"
    }]
  }
}

and then installing to disk with: sudo flatcar-install -o packet -s This chose the an NVMe block device for installation. You said it works with UEFI but not BIOS? Then the block device would be ok and maybe GRUB is missing something. But I wonder why I didn't get the GRUB menu – is there something in the BIOS which configures not to boot from that drive?

[pothos]

(Installing to /dev/sda worked)

[pothos]

To install to a /dev/sdX device add the -e MAJOR flag with the major device number of the NVMe disk (on my laptop it's -e 259 but maybe the device major number is different on the server).

[margamanterola]

Hi Kelsey!

Were you able to work around the problem with the information we provided?

Do you need any more information from the flatcar side?

Cheers!

[pothos]

@kgwaters6 This was resolved recently by excluding the NVMe drives' major device numbers from the installation, or was this for another instance type?

[dustinmiller]

@pothos https://github.com/tinkerbell/boots/pull/86 accomplished this goal, I believe this issue can be resolved

[pothos]

Thanks