bootc enablement for ppc64le

[sacsant]

I am trying to figure out how bootc (bootable container) works and started experimenting with it on Power(ppc64le arch).

I came across https://developers.redhat.com/articles/2024/05/07/image-mode-rhel-bootable-containers# and attempted to create a bootable container image on Power using the recipe mentioned in bootc image builder project .

I used following command (Fedora40 running on PowerVM Logical Partition [LPAR])

podman run --pull=newer --rm --privileged --pid=host quay.io/sacsant/fedora-bootc:40 bootc install to-disk --target-no-signature-verification --wipe --block-setup direct --filesystem xfs /dev/sdc WARN[0000] Failed to decode the keys ["storage.options.thinpool"] from "/usr/share/containers/storage.conf" time="2024-05-25T06:03:33-04:00" level=warning msg="Failed to decode the keys [\"storage.options.thinpool\"] from \"/usr/share/containers/storage.conf\"" time="2024-05-25T06:03:34-04:00" level=warning msg="Failed to decode the keys [\"storage.options.thinpool\"] from \"/usr/share/containers/storage.conf\"" Installing image: docker://quay.io/sacsant/fedora-bootc:40 Digest: sha256:6c88359c24a8c351bf2f742c0f5cfd9a5b32b1e6e475499fc9612c017e2cf860 Wiping /dev/sdc1 Wiping device /dev/sdc1 Wiping /dev/sdc2 Wiping device /dev/sdc2 Wiping /dev/sdc3 Wiping device /dev/sdc3 /dev/sdc3: 4 bytes were erased at offset 0x00000000 (xfs): 58 46 53 42 Wiping /dev/sdc Wiping device /dev/sdc /dev/sdc: 2 bytes were erased at offset 0x000001fe (dos): 55 aa /dev/sdc: calling ioctl to re-read partition table: Success ERROR Installing to disk: Creating rootfs: Unsupported architecture: powerpc64

Can someone help me debug this problem? Is code missing for ppc64le arch?

[prb112]

It's x86_64 and aarch64 only L251 anyhow::bail!("Unsupported architecture: {}", std::env::consts::ARCH);

It'd be great to get ppc64le support.

[sacsant]

Following the s390x enablement tracker, I was able to make some progress on this.

• Updated the image.go to include ppc64le arch
• Updated partition_tables.go with ppc64le specific data
• To get around the (panic: required mounts for bootupd stage [/boot/efi] missing) issue for now commented the /boot/efi line from images repo
• AnacondaContainerInstaller specific customisation code in image.go also possibly needs some update to avoid following errors. For now I commented these statements to move forward

cmd/bootc-image-builder/image.go:237:7: img.Users undefined (type *"github.com/osbuild/images/pkg/image".AnacondaContainerInstaller has no field or method Users)
cmd/bootc-image-builder/image.go:238:7: img.Groups undefined (type *"github.com/osbuild/images/pkg/image".AnacondaContainerInstaller has no field or method Groups)
cmd/bootc-image-builder/image.go:240:50: img.KickstartKernelOptionsAppend undefined (type *"github.com/osbuild/images/pkg/image".AnacondaContainerInstaller has no field or method KickstartKernelOptionsAppend)
cmd/bootc-image-builder/image.go:242:7: img.KickstartNetworkOnBoot undefined (type *"github.com/osbuild/images/pkg/image".AnacondaContainerInstaller has no field or method KickstartNetworkOnBoot)
cmd/bootc-image-builder/image.go:246:14: img.Users undefined (type *"github.com/osbuild/images/pkg/image".AnacondaContainerInstaller has no field or method Users)
cmd/bootc-image-builder/image.go:246:36: img.Groups undefined (type *"github.com/osbuild/images/pkg/image".AnacondaContainerInstaller has no field or method Groups)

I have run into following boot loader related error:

Installing image: docker://quay.io/sacsant/fedora-bootc:latest Initializing ostree layout Initializing sysroot ostree/deploy/default initialized as OSTree stateroot Deploying container image Deployment complete Running bootupctl to install bootloader Installing for powerpc-ieee1275 platform. /usr/sbin/grub2-install: error: the chosen partition is not a PReP partition. error: boot data installation failed: installing component BIOS: Failed to run "/usr/sbin/grub2-install" "--target" "powerpc-ieee1275" "--boot-directory" "/run/osbuild/mounts/boot" "--no-nvram" "/dev/loop0" ERROR Installing to filesystem: Installing bootloader: Task Running bootupctl to install bootloader failed: ExitStatus(unix_wait_status(256)) Traceback (most recent call last): File "/run/osbuild/bin/org.osbuild.bootc.install-to-filesystem", line 53, in r = main(args["options"], args["inputs"], args["paths"]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/run/osbuild/bin/org.osbuild.bootc.install-to-filesystem", line 48, in main subprocess.run(pargs, env=env, check=True) File "/usr/lib64/python3.12/subprocess.py", line 571, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['bootc', 'install', 'to-filesystem', '--source-imgref', 'containers-storage:[overlay@/run/osbuild/containers/storage+/run/containers/storage]b3a026083aa7f61628b5fdc3ab90b5b692139668ec583485d0f2e48fe34e46d9', '--skip-fetch-check', '--generic-image', '--karg', 'rw', '--karg', 'console=tty0', '--karg', 'console=ttyS0', '--target-imgref', 'quay.io/sacsant/fedora-bootc:latest', '/run/osbuild/mounts']' returned non-zero exit status 1.

Any suggestions on how to solve this error? I am unable to connect the dots and having trouble identifying code that needs a change to fix this.

[sacsant]

To be specific I am currently focussed on creating a qcow2 based bootable image.

podman run --rm -it --privileged --pull=newer --security-opt label=type:unconfined_t -v $(pwd)/config.toml:/config.toml -v $(pwd)/output:/output -v /var/lib/containers/storage:/var/lib/containers/storage quay.io/sacsant/bootc-image-builder:latest --type qcow2 --rootfs xfs --local quay.io/sacsant/fedora-bootc:latest

[cgwalters]

/usr/sbin/grub2-install: error: the chosen partition is not a PReP partition. error: boot data installation failed: installing component BIOS: Failed to run "/usr/sbin/grub2-install" "--target" "powerpc-ieee1275" "--boot-directory" "/run/osbuild/mounts/boot" "--no-nvram" "/dev/loop0" ERROR Installing to filesystem: Installing bootloader: Task Running bootupctl to install bootloader failed: ExitStatus(unix_wait_status(256))

That's in https://github.com/coreos/bootupd/

[sacsant]

Thanks Colin for the pointer. Since I am completely new to this project/code, I am still unable to connect the dots.

Should I use the coreos/bootupd code as reference and make similar changes to other repositories (bootc-image-builder or images or perhaps bootc??) or should I change coreos/bootupd? Can you guide me on this?

I noticed that coreos/bootupd uses powerpc64 as a value to identify IBM Power architecture while other projects like bootc-image-builder and likes use ppc64le as a value. Is there a theory behind this difference ?

[mkumatag]

I noticed that coreos/bootupd uses powerpc64 as a value to identify IBM Power architecture while other projects like bootc-image-builder and likes use ppc64le as a value. Is there a theory behind this difference ?

This is based on how your programming language detects the platform

e.g: golang

# go env GOARCH
ppc64le      <=============================

rust:

# rustc --target=powerpc64le-unknown-linux-gnu --print=cfg
debug_assertions
panic="unwind"
target_abi=""
target_arch="powerpc64"             <=============================
target_endian="little"
target_env="gnu"
target_family="unix"
target_has_atomic="16"
target_has_atomic="32"
target_has_atomic="64"
target_has_atomic="8"
target_has_atomic="ptr"
target_os="linux"
target_pointer_width="64"
target_vendor="unknown"
unix

[cgwalters]

What's extra fun here is that it's actually only for ppc64le on Fedora derivatives that the "Rust vs RPM" architecture naming is different. Rust=powerpc64 RPM=ppc64le whereas they match for s390x, x86_64 and aarch64. Note that the Go and RPM names are otherwise different for amd64 vs x86_64 etc.

[sacsant]

Going back to the original error of "Unsupported architecture: powerpc64" , I added following code in bootc } else if cfg!(target_arch = "powerpc64") { // BIOS-BOOT sgdisk_partition( &mut sgdisk.cmd, 1, "0:+4M", "BIOS-BOOT", Some("9E1A2D38-C612-4316-AA26-8B49521E5A8B"), );

Is the above code correct?

With this code using install to-disk command the operation proceeds further but fails eventually with

DEBUG exec: "sgdisk" "-Z" "/dev/sdb" "-U" "R" "-n" "1:0:+4M" "-c" "1:BIOS-BOOT" "-t" "1:9E1A2D38-C612-4316-AA26-8B49521E5A8B" "-n" "3:0:0" "-c" "3:root" "-t" "3:0FC63DAF-8483-4772-8E79-3D69D8477DE4" [14381.355893] sdb: sdb1 sdb3 DEBUG Created partition table [14381.356751] sdb: sdb1 sdb3 DEBUG argv0=Some("exe") ERROR Installing to disk: Creating rootfs: Missing partition for index 3

Following partition layout was written to the disk Device Start End Sectors Size Type /dev/sdb1 256 1279 1024 4M PowerPC PReP boot /dev/sdb3 1280 26214394 26213115 100G Linux filesystem

What partition is missing and is referenced by index 3? I guess there should be /dev/sdb2 pointing to /boot . Can someone help decode this error?

[sacsant]

What partition is missing and is referenced by index 3? I guess there should be /dev/sdb2 pointing to /boot . Can someone help decode this error?

I was able to modify the code to move past this issue.

Using block setup: direct Initializing partitions Creating boot filesystem (xfs) on device /dev/sdb2 (size=510M)

mkfs.xfs -m uuid=e14c7dd8-4297-4cb4-b41b-eb027167d941 -L boot /dev/sdb2 Creating root filesystem (xfs) on device /dev/sdb3 (size=99.5G) mkfs.xfs -m uuid=ad5cdc91-0ca9-4cf5-afda-5aeaf53b1370 -L root /dev/sdb3 Mounting /run/bootc/mounts/rootfs Mounting /run/bootc/mounts/rootfs/boot Initializing ostree layout Initializing sysroot ostree/deploy/default initialized as OSTree stateroot Deploying container image...Freed objects: 118 bytes done Running bootupctl to install bootloader

Now both the methods qcow2 and install to-disk fail during boot loader install

Running bootupctl to install bootloader

bootupctl backend install --write-uuid --update-firmware --auto --device /dev/sdb /run/bootc/mounts/rootfs Installing for powerpc-ieee1275 platform. /usr/sbin/grub2-install: error: the chosen partition is not a PReP partition. error: boot data installation failed: installing component BIOS: Failed to run "/usr/sbin/grub2-install" "--target" "powerpc-ieee1275" "--boot-directory" "/run/bootc/mounts/rootfs/boot" "--no-nvram" "/dev/sdb" ERROR Installing to disk: Installing bootloader: Task Running bootupctl to install bootloader failed: ExitStatus(unix_wait_status(256))

[sacsant]

With some additional changes (read hard coded values) I was able to successfully use bootc install to-disk and to-filesystem commands on ppc64le architecture. I was also able to successfully verify boot of these images (disk device as well as qcow2) on IBM Power.

[cgwalters]

Cool! Feel free to send a PR, it's OK if it's not polished, we can iterate on it!

[sacsant]

I have submitted following pull requests for initial set of changes PR 497 against bootc-image-builder PR 636 against bootc

Working on following changes:

• images : Remove dependency on /boot/efi for ppc64le arch
• bootc-image-builder compile time failure

  cmd/bootc-image-builder/image.go:237:7: img.Users undefined (type *"github.com/osbuild/images/pkg/image".AnacondaContainerInstaller has no field or method Users)
  cmd/bootc-image-builder/image.go:238:7: img.Groups undefined (type *"github.com/osbuild/images/pkg/image".AnacondaContainerInstaller has no field or method Groups)
  cmd/bootc-image-builder/image.go:240:50: img.KickstartKernelOptionsAppend undefined (type *"github.com/osbuild/images/pkg/image".AnacondaContainerInstaller has no field or method KickstartKernelOptionsAppend)
  cmd/bootc-image-builder/image.go:242:7: img.KickstartNetworkOnBoot undefined (type *"github.com/osbuild/images/pkg/image".AnacondaContainerInstaller has no field or method KickstartNetworkOnBoot)
  cmd/bootc-image-builder/image.go:246:14: img.Users undefined (type *"github.com/osbuild/images/pkg/image".AnacondaContainerInstaller has no field or method Users)
  cmd/bootc-image-builder/image.go:246:36: img.Groups undefined (type *"github.com/osbuild/images/pkg/image".AnacondaContainerInstaller has no field or method Groups)

• bootupd changes

[yoheiueda]

Hi @sacsant

I am working on s390x support, and I just raised this PR https://github.com/osbuild/images/pull/758

images : Remove dependency on /boot/efi for ppc64le arch

I think my patch may be relevant to this issue.

[sacsant]

Hi @sacsant

I am working on s390x support, and I just raised this PR osbuild/images#758

images : Remove dependency on /boot/efi for ppc64le arch

I think my patch may be relevant to this issue.

Indeed. Thanks @yoheiueda

[sacsant]

@cgwalters to identify correct partition for grub2-install, I have tried using following code in lib/src/bootloader.rs

// get PowerPC-PReP-boot device information let result = Command::new("realpath") .arg("/dev/disk/by-partlabel/PowerPC-PReP-boot") .output()?; if !result.status.success() { anyhow::bail!("realpath failed with {}", result.status); }

Unfortunately this code fails with return code 1 which if I read correctly corresponding to errno EPERM (Operation not permitted).

Creating root filesystem (xfs) on device /dev/sdb3 (size=99.5G)

mkfs.xfs -m uuid=2e332d8b-8cca-46df-8d63-9d874333cada -L root /dev/sdb3 Mounting /run/bootc/mounts/rootfs Mounting /run/bootc/mounts/rootfs/boot Initializing ostree layout Initializing sysroot ostree/deploy/default initialized as OSTree stateroot Deploying container image...Freed objects: 118 bytes done ERROR Installing to disk: Installing bootloader: realpath failed with exit status: 1

Is there a restriction on running commands like realpath in bootc environment? If I manually run the command it does return correct output $ realpath /dev/disk/by-partlabel/PowerPC-PReP-boot /dev/sdb1

The disk is partitioned correctly as follows.

fdisk -l /dev/sdb Disk /dev/sdb: 100 GiB, 107374182400 bytes, 26214400 sectors Disk model: VDASD
Units: sectors of 1 * 4096 = 4096 bytes Sector size (logical/physical): 4096 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disklabel type: gpt Disk identifier: DE468C00-7382-4C25-A58A-1802A3AC9666

Device Start End Sectors Size Type /dev/sdb1 256 1279 1024 4M PowerPC PReP boot /dev/sdb2 1280 131839 130560 510M Linux filesystem /dev/sdb3 131840 26214394 26082555 99.5G Linux filesystem

[sacsant]

mkfs.xfs -m uuid=2e332d8b-8cca-46df-8d63-9d874333cada -L root /dev/sdb3 Mounting /run/bootc/mounts/rootfs Mounting /run/bootc/mounts/rootfs/boot Initializing ostree layout Initializing sysroot ostree/deploy/default initialized as OSTree stateroot Deploying container image...Freed objects: 118 bytes done ERROR Installing to disk: Installing bootloader: realpath failed with exit status: 1

Is there a restriction on running commands like realpath in bootc environment? If I manually run the command it does return correct output $ realpath /dev/disk/by-partlabel/PowerPC-PReP-boot /dev/sdb1

The failure is due to non-existent directory

Initializing ostree layout Initializing sysroot ostree/deploy/default initialized as OSTree stateroot Deploying container image...Freed objects: 118 bytes done realpath: /dev/disk/by-partlabel/PowerPC-PReP-boot: No such file or directory

ERROR Installing to disk: Installing bootloader: realpath failed with exit status: 1

[sacsant]

I have submitted https://github.com/containers/bootc/pull/667 to include ppc64le specific boot loader install logic.

[cgwalters]

At this point, I think we've fixed everything that's needed in this specific project! There's clearly more work - such as enabling c9s bootc that are probably best tracked in e.g. https://gitlab.com/fedora/bootc/tracker/-/issues/15