/etc/crypttab completely gone in zipl initrd (s390x image)

lcp commented 1 month ago

Problem description

I'm working on the encrypted image based on the s390x build test, test-image-disk, and the following luks parameters are added to <type>:

luks_version="luks2"
luks="1234"
luks_randomize="false"
luks_pbkdf="pbkdf2"

For the 'Virtual' image, additional bootpartition="true" and the related parameters are added to create an unencrypted zipl partition for booting the encrypted root partition.

Although the image was built successfully, the initrd in the zipl partition didn't contain /etc/crypttab which is necessary to unlock the root partition. However, /etc/crypttab did exist in the root partition, and the initrd in /boot also contained a valid crypttab.

I slightly modified python-klwi and grub2 to make them more verbose. Per the OBS build log(*), grub2-install strangely complained '/dev/mapper/luksRoot is not present.', and the followed dracut command seemed not detecting the LUKS partition and didn't include the 'crypt' module.

If I manually chroot and update the zipl initrd with a rescue media, then the new zipl initrd contains a valid crypttab. I have no idea why the chroot environment went insane when KIWI was invoking grub2-install.

(*) https://build.opensuse.org/build/home:gary_lin:branches:Virtualization:Appliances:Images:Testing_s390:tumbleweed/images/s390x/test-image-disk-enc:Virtual/_log To preserve the build log, it's attached as virtual-build.log.

Expected behaviour

The zipl initrd contains a valid /etc/crypttab.

Steps to reproduce the behaviour

Build my test image in OBS and mount the zipl partition to check initrd with 'lsinitrd'. https://build.opensuse.org/package/show/home:gary_lin:branches:Virtualization:Appliances:Images:Testing_s390:tumbleweed/test-image-disk-enc

OS and Software information

KIWI version: 10.0.25
Operating system host version: build.opensuse.org
Operating system target version: openSUSE Tumbleweed
Open Build Service version (N/A if not using OBS): build.opensuse.org
Koji version (N/A if not using Koji): N/A

lcp commented 1 month ago

I added a few debug code to kiwi to list the block devices in /dev/disk before invoking grub2-install. It turned out that there was only one entry in /dev/disk/by-uuid and none of the target partitions was there.

As a workaround, I add the following line to config.sh to ensure the crypt module is always included in dracut.

echo 'force_add_dracutmodules+=" crypt "' >> /etc/dracut.conf.d/15-kiwi-workaround.conf

Although this makes dracut to go through /etc/crypttab, it left an empty crypttab due to the missing by-uuid link. To work around the empty crypttab, I added 'force' to crypttab: https://github.com/lcp/kiwi/commit/2c21b4e72cc54e969d2d6710bf149351bd8d87f5

Now I have the image with the zipl initrd containing a valid crypttab. However, the image still failed to boot with the error:

[ TIME ] Timed out waiting for device /dev/disk/by-uuid/8e601b60-eb5d-42aa-93d8-11f053e62698.

What's funny is the link showed when I entered the dracut emergency shell. I still have no good idea of how this missing by-uuid happened...

lcp commented 1 month ago

It looks like the missing by-uuid device links is the main cause of the issue, and that's more about the kernel or udev. Close this kiwi issue.

OSInside / kiwi