Open cgwalters opened 1 month ago
@cgwalters thank you very much for creating this tracker.
So far, I have identified the following issues.
/boot/efi
partitions is required as follows while this partition is not necessary on s390x
panic: required mounts for bootupd stage [/boot/efi] missing
ERROR Installing to filesystem: Creating ostree deployment: Performing deployment: Final bootloader swap: Failed to execute child process “zipl” (No such file or directory)
I am currently working on the last issue.
Add the s390x architecture in image.go of bootc-image-builder
Yep, I filed https://github.com/osbuild/bootc-image-builder/issues/460 as that code is in a distinct project.
Install zipl, otherwise the following error occurs when performing image conversion
Right. A lot of useful code in https://github.com/coreos/coreos-installer/blob/main/src/s390x/zipl.rs that I think we can just copy into this project.
Also https://gitlab.com/fedora/bootc/tracker/-/issues/15 is related to this
After updating the base image to include bootupd on s390x, I got the following error.
device/disk (org.osbuild.loopback): loop0 acquired (locked: False)
mount/part2 (org.osbuild.xfs): mounting /dev/loop0p2 -> /store/tmp/buildroot-tmp-fcpd6h9i/mounts/
mount/part1 (org.osbuild.ext4): mounting /dev/loop0p1 -> /store/tmp/buildroot-tmp-fcpd6h9i/mounts/boot
mount/part3 (org.osbuild.fat): mounting /dev/loop0p3 -> /store/tmp/buildroot-tmp-fcpd6h9i/mounts/boot/efi
Host kernel does not have SELinux support, but target enables it by default; this is less well tested. See https://github.com/containers/bootc/issues/419
Installing image: docker://localhost/centos-stream-9-bootc:latest
Initializing ostree layout
Initializing sysroot
ostree/deploy/default initialized as OSTree stateroot
Deploying container image...Error: Config file '/lib/s390-tools/zipl.conf': Cannot build automenu: no IPL entries available
Looking for components in '/lib/s390-tools'
Using config file '/lib/s390-tools/zipl.conf'
done
ERROR Installing to filesystem: Creating ostree deployment: Performing deployment: Final bootloader swap: Child process exited with code 1
Traceback (most recent call last):
File "/run/osbuild/bin/org.osbuild.bootc.install-to-filesystem", line 53, in <module>
r = main(args["options"], args["inputs"], args["paths"])
File "/run/osbuild/bin/org.osbuild.bootc.install-to-filesystem", line 48, in main
subprocess.run(pargs, env=env, check=True)
File "/usr/lib64/python3.9/subprocess.py", line 528, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['bootc', 'install', 'to-filesystem', '--source-imgref', 'containers-storage:[overlay@/run/osbuild/containers/storage+/run/containers/storage]f276b7c877e9e130a787dc53a31fdc714edc95456a4fc10e67ff57c1a8038133', '--skip-fetch-check', '--generic-image', '--karg', 'rw', '--karg', 'console=tty0', '--karg', 'console=ttyS0', '--target-imgref', 'localhost/centos-stream-9-bootc:latest', '/run/osbuild/mounts']' returned non-zero exit status 1.
The error occurs during executing zipl
at
https://github.com/containers/bootc/blob/f152bfe8da47c3bfbafca33c2142d10611375fa3/lib/src/install.rs#L1184
using the ostree
library.
https://github.com/ostreedev/ostree/blob/ac6ba4392285d4f30a937fddf2963b879fe129bd/src/libostree/ostree-bootloader-zipl.c#L477-L478
I guess we need to generate a proper zipf.conf
here.
@cgwalters do we need to generate zipl.conf
before calling initialize_ostree_root_from_self
in bootc? Or, can we use the ostree
libbary to genearte zipl.conf
. I don't see such generation code in _ostree_bootloader_zipl_write_config
.
crate::bootloader::install_via_bootupd
is called after initialize_ostree_root_from_self
.
https://github.com/containers/bootc/blob/f152bfe8da47c3bfbafca33c2142d10611375fa3/lib/src/install.rs#L1194
It looks like bootupd does no-op for s390x. https://github.com/coreos/bootupd/pull/556
I also think we need a zipl_helper script for loop device like this. https://github.com/openstack/diskimage-builder/blob/ac180b9ac61b7c8577fe0d489a33b0dd805d8acb/diskimage_builder/elements/zipl/finalise.d/50-zipl#L29-L52
I further read the source code, and I think that zipl
needs to recognize the BLS config file at
/store/tmp/buildroot-tmp-ctk6b4of/mounts/boot/loader/entries/ostree-1.conf
We need to match some of the logic there. Now zipl is somewhat special so far in the ostree/bootc architecture because it basically always acts like a UKI, rolling the bootloader,kernel,initrd together and hence crosses the domain of bootupd and bootc.
Going a bit forward I'd like to merge bootupd into this project https://github.com/coreos/bootupd/issues/432 which would help a bit in making coordinated changes.
@cgwalters thank you for the information.
I managed to create a QCOW2 disk using bootc. To work around the issues I reported above, I need to prepare a wrapper script for zipl, and update centos-bootc
image to include it. I don't think this is a good solution, but I just confirmed that the created image works as expected on an s390x machine.
Next, I will investigate what changes are necessary in the bootc and bootupd source code to support zipl.
I further read the source code, and I think that zipl needs to recognize the BLS config file at /store/tmp/buildroot-tmp-ctk6b4of/mounts/boot/loader/entries/ostree-1.conf
This is also related to https://github.com/ostreedev/ostree/pull/3104 perhaps?
This is quite a complex thing going on as osbuild (as used by bootc-image-builder) is also doing its own container stuff, then we have podman, and we also have bubblewrap involved here.
I managed to create a QCOW2 disk using bootc. To work around the issues I reported above, I need to prepare a wrapper script for zipl, and update centos-bootc image to include it. I don't think this is a good solution, but I just confirmed that the created image works as expected on an s390x machine.
Can you post that script?
Also, did you try a direct bootc install to-disk
?
@cgwalters sorry for the late reply. I was on vacation.
My script is this one. https://gist.github.com/yoheiueda/c328ca7eccf313dd03292aeca71fbd02
I directly put this script in the centos bootc base image, and replaced with the original zipl binary in it.
This script bind-mounts /run/osbuild/mounts/boot
to /boot
, and also temporarily installs a helper script at /run/osbuild/mounts/s390-tools/zipl_helper.blkext
. The target disk is
A helper script is called for disk device types that are not natively supported by zipl. https://github.com/ibm-s390-linux/s390-tools/blob/0e3034c0f699cb311c0f63d0bb475a816f5fba91/zipl/src/disk.c#L359-L384
In the case of osbuild installation, the target disk type is loop
, and the disk type of its partition is blkext
, so a helper script named zipl_helper.blkext
will be called.
This work around needs to update the centos bootc base image, so I don't think it is an appropriate solution for permanent fix, but I just confirmed that QCOW2 image generated with this workaroud, so I now believe there is not other blockers for s390x.
This is also related to https://github.com/ostreedev/ostree/pull/3104 perhaps? Yes, I think so.
If target_deployment is set and /boot
is appropriately mounted, bind-mounting of /run/osbuild/mounts/boot
is not necessary. But, it looks like target_deployment is set to NULL, and zipl
is directly executed.
Also, did you try a direct bootc install to-disk?
I have only tried install-to-filesystem
via bootc-image-builder
. I will try a direct bootc install to-disk
next.
A helper script is called for disk device types that are not natively supported by zipl. https://github.com/ibm-s390-linux/s390-tools/blob/0e3034c0f699cb311c0f63d0bb475a816f5fba91/zipl/src/disk.c#L359-L384
Ahhh of course. This is the big difference between what we were doing in coreos-assembler using qemu (providing a native dasd disk to the build) vs what bib (and bootc can do) with loopback. Hmm...that said, it looks like osbuild's zipl stage isn't doing anything too special here?
If target_deployment is set and /boot is appropriately mounted, bind-mounting of /run/osbuild/mounts/boot is not necessary. But, it looks like target_deployment is set to NULL, and zipl is directly executed.
I think that may be one of the bugs here.
@cgwalters
The osbuild's zipl stage explicitly set parameters targetbase
, targettype
,targettype
, targetblocksize
and targetoffset
. In this case, the helper script is not necessary. The helper script is used to identify these values when the parameters are not specified.
So, is it an option to use the osbuild's zipl stage instead of ostree by generating a s390x-specific manifest by bootc
?
Regarding bootc install to-disk
, I needed to modify the code to prepare partitions for s390x.
Then, I got the following error. I guess the cause of the error is that BLS configs under /boot/loader/entries
are not visible from zipl
. In this case, I didn't use my zipl wrapper script here.
sudo podman run --rm --privileged --pid=host -v /var/lib/containers:/var/lib/containers -v /dev:/dev -v $PWD/output:/output --security-opt label=type:unconfined_t localhost/centos-stream-9-bootc:latest env RUST_LOG=debug bootc install to-disk --skip-fetch-check --via-loopback --wipe /output/disk.img
Error: Config file '/lib/s390-tools/zipl.conf': Cannot build automenu: no IPL entries available
Looking for components in '/lib/s390-tools'
Using config file '/lib/s390-tools/zipl.conf'
done
DEBUG exec: "losetup" "-d" "/dev/loop0"
ERROR Installing to disk: Creating ostree deployment: Performing deployment: Final bootloader swap: Child process exited with code 1
In the case of bootc install to-disk
, this command should work as a standalone tool, so use of osbuild's zipl stage may not be an option.
As you mentioned earlier, if bootupd code is integrated into bootc, we can directly add zipl support in bootc. I think this is a best option, but it will take some time?
The osbuild's zipl stage explicitly set parameters targetbase, targettype, targettype, targetblocksize and targetoffset. In this case, the helper script is not necessary. The helper script is used to identify these values when the parameters are not specified.
OK. Of those...do you understand where targetoffset
comes from? I looked through the code but I don't understand that.
So, is it an option to use the osbuild's zipl stage instead of ostree by generating a s390x-specific manifest by bootc?
We have a logistical problem in that so far, osbuild is something designed for build-time only; we don't ship it onto the end system. IOW it doesn't exist by default in a bootc container. That doesn't mean it can't, but it would be a big change to add it.
There's a larger picture problem around code sharing between osbuild, anaconda, bootc (and for that matter, a wide array of other tools in this space, some distro specific, others more generic like Kiwi etc.)
Looking at that osbuild stage, in the end it's just parsing some bootloader configs and invoking zipl mostly, and we already have all that code here too, right?
Regarding bootc install to-disk, I needed to modify the code to prepare partitions for s390x.
Can you send a patch for that? Even if it's not all complete, nothing stops us from merging it now.
Then, I got the following error. I guess the cause of the error is that BLS configs under /boot/loader/entries are not visible from zipl.
Hmm, yes. There's a whole lot of namespacing and mounts going on here. It's just complex since zipl so far is the only case where we need to run code to update the bootloader.
As you mentioned earlier, if bootupd code is integrated into bootc, we can directly add zipl support in bootc. I think this is a best option, but it will take some time?
I don't think we need to block on merging bootupd; it would help a bit, but it's not the core problem here.
OK. Of those...do you understand where targetoffset comes from? I looked through the code but I don't understand that.
According to this page, targetoffset
needs to be set to 2048 when the first partition is the boot partition.
https://bugs.launchpad.net/diskimage-builder/+bug/1730641
MBR consumes 512 Byte. Due to the alignment the next (1MiB -512 Bytes) are unused. The first partition starts at 1 MiB which is block 2048 ((1024 * 1024)B / 512 Bytes per Block).
-> zipls targetoffset should be set to 2048 when the first partition is the boot partition.
Currently, I specify the whole disk (/dev/loop0) to targetbase
, and its first partition (/dev/loop0p1) is /boot
, so we need to specify targetoffset=2048
. If I directly specify the first partition (/dev/loop0p1) to targetbase
, do I need to specify targetoffset=0
? I will confirm it later.
Looking at that osbuild stage, in the end it's just parsing some bootloader configs and invoking zipl mostly, and we already have all that code here too, right?
zipl
has a mechanism to find kernel and ramdisk paths using BLS info under /boot/loader/entries/
, but the mechanism is only used when the command line options (targetbase, targetoffset, ...) are not specified if I remember correctly. When the target options are specified, the options --image
, --ramdisk
, and --parameters
are also needed to be specified directly.
So, I think we may need to copy the code that searchs BLS info from https://github.com/coreos/coreos-installer/blob/main/src/s390x/zipl.rs to identify kernel and ramdisk paths.
Regarding bootc install to-disk, I needed to modify the code to prepare partitions for s390x.
Can you send a patch for that? Even if it's not all complete, nothing stops us from merging it now.
Sure. I currently did quick dirty hacks to suppress errors on s390x. I'll prepare a cleaner patch to fix the errors.
I managed to run bootc install to-disk
. I updated my zipl wrapper script to calculate a targetoffset of /boot
partition.
https://gist.github.com/yoheiueda/c328ca7eccf313dd03292aeca71fbd02
@cgwalters
Regarding bootc install to-disk, I needed to modify the code to prepare partitions for s390x.
Can you send a patch for that? Even if it's not all complete, nothing stops us from merging it now.
I raised a PR for this issue. https://github.com/containers/bootc/pull/623
@cgwalters I raised an PR to run zipl
with correct options on a loop device. https://github.com/containers/bootc/pull/665
I think right now this is probably just porting some code from e.g. https://github.com/coreos/coreos-installer around setting up partitions etc. to enable
install to-disk
.What also clearly needs to happen here is fixes for https://github.com/osbuild/bootc-image-builder
(Which also leads into the overlap with Anaconda problem)