Regression: custom kernels are no longer allowed in live builds

[v-fox]

WHY ?! So, I've been prompted to finally make updated iso for my live TW build only to find out that my custom kernel cannot be used and instead trashy SUSE default kernel is hardcoded. What the hell happened with all that ?

    <profiles>
        <profile name="default" description="Booting default setup"/>
        <profile name="custom" description="KERNEL: specified by bootinclude in system XML"/>
        <profile name="xen" description="Boot with xen mboot.c32 setup"/>
        <profile name="std" description="KERNEL: Default"/>
        <profile name="pae" description="KERNEL: pae kernel"/>
        <profile name="xenk" description="KERNEL: Xen kernel/mboot.c32"/>
        <profile name="desktop" description="KERNEL: Desktop kernel"/>
    </profiles

Not only that by it's not even packaged in neither kiwi-boot-descriptions or kiwi-templates-JeOS, it's nowhere to be found ! Is it even possible to make the live builds on openSUSE anymore without resorting to crutches and snots for old version of kiwi ?

And here I naively though that your rewrites would actually improve live builds and allow to finally auto-make exFAT partition for whole USB flash stick with overlay file on it. You know, the thing that would actually allow to use whole giant modern USB sticks as universal portable storage (which also requires for boot partition to be second, after storage partition due to Windows' old "one portable drive partition only" limitation) and have live OS on them ? The thing that we discussed years, among other things like saner auto RAM-drive size, ago but instead those discussions and patches got deleted ! And that after openSUSE installer was "rewritten and improved" to no longer install customized builds too.

[schaefi]

The above was needed at times when kiwi created its own initrd. These times were over when everything was switched to dracut. That having said if you want a custom kernel you only need to replace the line:

<package name="kernel-default"/>

with

<package name="your-kernel"/>

No other data, profiles or extra boot descriptions are needed anymore. Due to the move to dracut the custom kiwi initrd concept became obsolete. As we wanted to follow the upstream and distro standard and dracut also offers the more flexible system to handle initrd's, kiwi moved too. This has been ~2years ago and in the beginning we offered support for both during the transition period. For the image types oem and pxe we still allow custom kiwi initrd's because there are customers who still use the kiwi hook scripts which are handled as systemd unit hooks in dracut and needs to be rewritten. For the support of live ISO's the upstream community was fine with the two existing concepts below and thus isoboot got dropped:

For the live system concept there are two opportunities

<type image="iso" ... flags="overlay|dmsquash"/>

The first variant "overlay" is based on a small dracut module that comes with kiwi packaged as dracut-kiwi-live. It does not provide support for the exFAT partition, so probably not what you want

The second variant "dmsquash" is the dracut upstream live module. The module does not provide support for creating the extra partition but it can use a specified one if it has been created beforehand. I think you should be able to prepare your stick with the partition of your choice and setup a loopback grub boot using the generated live iso.

If all this is not sufficient to your use case you could consider to have a conversation with the upstream dracut people. We also like to have a conversation with them regarding the extra partition creation code to become an upstream part. But we are not there yet.

From our perspective it would be great if the upstream dmsquash and livenet dracut modules can cover all live use cases to make people happy. The missing "auto-make exFAT partition" can either go into the upstream dracut module or into the dracut-kiwi-live code.

I apologize if we did not port the exFat creation code from the legacy kiwi code base to the new one yet. The new dracut modules were written with regards to customer needs and nobody wanted to use exFat as the cow device so far.

I hope I could clarify the why on your initial report. Let us know how to continue from here.

Thanks

[schaefi]

If there are no further objections I guess this one can be closed ?

[v-fox]

If there are no further objections I guess this one can be closed ?

Well, in short, my expectations about a Live image creation would be:

• "Installation" by just dumping image straight to block device, with simple tools like dd.
• On first boot automatically:
  • a) assign some percentage of whole block device to R/W-boot&root partition and the rest - for "universal data partition" which previously would be FAT32 or UDF and now - exFAT.
  • b) make single universal data partition with kernel image, R/W-root image or R/O-root image + overlay image on it if bootloader is capable of booting that.
  • c) assign some percentage of whole block device to primary and secondary partitions where secondary is "universal data partition" while primary houses kernel, root and/or overlay images but is formatted in something that bootloader can chew.
  • d) "modern" UEFI-style boot with UEFI kernel images on "invisible" UEFI partition, native R/W root partition and non-native data partition. That would require actually having UEFI kernel images in packages which is not OBS-ready yet, I think. But even without that, bootloader should boot in UEFI mode and allow chainloading things like rEFInd, UEFI emergency shells and current releases of memtest.
• Live system root partition having all kernel modules and script for auto-configuration on any system. Meaning Image build system figuring out all the mandatory dependencies for:
  • setting up the bootloader,
  • opening up boot partition to read kernel and root image (finding root image/partition by filename, UUID or label),
  • setting up networking, time, language, (although, that could be skipped with NM, UTC and DM locale selection)
  • allowing to drop UEFI boot images to either be automatically added to boot list or concatenated from static bootloader config text.

Previously, kiwi did all that, including making a secondary partition on an USB stick. Or at least it would leave unused space as unformatted, so it could be assigned, formatted and marked in a way that Windows would garble up manually or by initrd script.

Even if setting custom kernel is no issue, I'm not even sure if I have to just rebuild old kiwi iso profiles with all the dependencies manually or it would be able to find dependencies for Live image itself. Because old kiwi didn't. It had this weird bootinclude="true"tag that had to be specified at everything that is needed for creation of every type of partition & image and functioning of initrd script for build not to fail at the end. And even weirder <packages type="delete"></packages> and bootinclude="true" bootdelete="true" directives which I still don't understand.

So, is it even possible to make Live image for USB that is simply dumped without manual preparation ? What will happen with excessive space ? What's with UEFI compatibility (is it visible in BIOS) ? If the answer is "Live image is easily built without guessing dependencies for its creation and booting but excessive space is simply unmapped and additional boot images have to be added by hand after installation and per-installation." then, I guess, it can be considered "on par" with old kiwi and closed. But I would still consider that incomplete for an automated Live image builder tool due to manual handling required for whole device to be usable.

For the live system concept there are two opportunities <type image="iso" ... flags="overlay|dmsquash"/>

If it uses overlay/squash then it doesn't use R/W root partition, but that means that it uses non-Linux boot partition. And there is not much options for those. What is it, FAT32 ? With exFAT now officially in kernel (even as built-in option), there shouldn't be much trouble in migrating to it. And could overlay approach be replaced entirely for R/W root (and maybe even boot) with auto-growth of FS on first boot (initrd would still need root-search script if boot is separate) ?

I apologize if we did not port the exFat creation code from the legacy kiwi code base to the new one yet. The new dracut modules were written with regards to customer needs and nobody wanted to use exFat as the cow device so far.

"cow device" as in device that houses kernel and root images ? Well, that only necessary if images are housed on Windows-visible partition, be it the only shared partition (b) or secondary (c) partition (which would make sense for overlay-image to be put there because of potentially unlimited size growth that could exhaust primary).

[schaefi]

ok according to your information I prepared an image build. Let's sort out all the missing parts. So here is the image

• https://build.opensuse.org/package/binaries/Virtualization:Appliances:Images:Testing_x86:suse/test-image-vfox/images

You should be able to fetch it from there. For testing I did the following

$ qemu-img create my-usb-stick 20g
$ dd if=kiwi-test-image-live-vfox.x86_64-1.42.3-Build1.1.iso of=my-usb-stick conv=notrunc
$ kvm -bios /usr/share/qemu/ovmf-x86_64-ms.bin -hda my-usb-stick -serial stdio

"Installation" by just dumping image straight to block device, with simple tools like dd.

As done above, should also work on a real stick. /usr/share/qemu/ovmf-x86_64-ms.bin is a virtual secure boot EFI image signed by Microsoft. So you can test secure boot with it

d) "modern" UEFI-style boot with UEFI kernel images on "invisible" UEFI partition

In the image description I have set

firmware="uefi"

This uses the shim approach for booting in secure boot. The design of the embedded partition table also allows to boot in CSM (BIOS legacy) mode. I can't tell how "invisible" this is according to your needs

b) make single universal data partition with kernel image,

This is done on first boot of the system. A data partition is created which currently spans the available rest space. The filesystem of the data partition is configurable via

hybridpersistent_filesystem="xfs"

currently ext4 and xfs are supported. If I got it right you want to use another fs for this part and also an opportunity to configure the size of the partition, correct ?

The resulting partition table is a hybrid table that looks like this

$ parted /dev/sda print

Number  Start   End     Size    Type     File system  Flags
 1      32.8kB  281MB   281MB   primary               boot, type=cd
 2      281MB   297MB   15.7MB  primary  fat16        esp, type=ef
 3      298MB   21.5GB  21.2GB  primary  xfs          type=83

If this table is not applicable to your use case please share your suggestion for a better understanding.

c) assign some percentage of whole block device to primary and secondary partitions

This part I did not fully understand. The live systems created by kiwi uses overlayfs for the system. The result looks like this

df -h
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           2.0G     0  2.0G   0% /run/overlay
/dev/sda1       269M  269M     0 100% /run/overlay/live
/dev/loop0      193M  193M     0 100% /run/overlay/squashfs_container
/dev/loop1      1.1G  763M  204M  79% /run/overlay/rootfsbase
/dev/sda3        20G  209M   20G   2% /run/overlay/overlayfs
LiveOS_rootfs    20G  209M   20G   2% /

Live system root partition having all kernel modules and script for auto-configuration on any system. Meaning Image build system figuring out all the mandatory dependencies for:

The live system looks like a standard system. It's only different on the level of the rootfs mounting. So you should not be hold back or blocked on any type of system configuration.

For a start I think we have collected quite some data. Maybe you can just fetch the image as I built it and deploy it virtually for a test. From there we have a common base to work on further issues / features / concerns

Hope this helps with the next steps.

Thanks

[schaefi]

I forgot, in the image description the following kernel is used:

<package name="kernel-default"/>

If you want another kernel, this is the only line to change, and that kernel package must be available in the repos of course

[schaefi]

Any thoughts ?

[schaefi]

I assume there is no further interest in this topic.