Support linux boot protocol for kexec

[rick-masters]

The live-bootstrap project boots Linux from Fiwix which requires support for the linux boot protocol.

The implementation in the forthcoming PR is similar to the multiboot1 solution:

• The kexec_protocol with value linux must be specified at the kernel command line.
• The kexec_size parameter is used to configure a ram drive which holds the kernel (and initrd ram drive).
• The kexec is performed when the kernel halts.

There are several design points that should be explained.

The kexec process for live-bootstrap uses both a linux kernel and an initial ram file system. These files are built under Fiwix and then must be passed to the Fiwix kernel for kexec. Similar to multiboot, the files are copied to a ram drive device. However, since there are two files the process is a little more complicated as explained later.

Assuming you boot Fiwix from a hard drive (and so there is no initrd ram drive device) then the kexec ram drive will be /dev/ram0.

--

By the way...

Note that live-bootstrap boots with a very large initrd ram drive, which is configured with the Fiwix kernel parameter ramdisksize=1179648 (approx. 1.2G). This is a problem because Fiwix compiles by default with one (additional) "general purpose" ram drive which uses the same parameter for sizing. But there is not enough memory for that.

Therefore, live-bootstrap compiles Fiwix with:

#define RAMDISK_DRIVES  0

In order to configure this setting, RAMDISK_DRIVES has been moved to include/fiwix/config.h.

--

A major way that linux boot differs from multiboot is the way that the size of the kernel and ram drive are conveyed to the Fiwix kernel for kexec. This is done by writing the length of the linux kernel in bytes to the ram drive as a 32-bit little-endian number followed by the length of the ram drive as another 32-bit little endian number. Then the kernel and ram drive are written to the ram drive, sequentially, starting right after the lengths.

I am open to feedback on this design. An alternative is to configure multiple ram drives but that would require additional kexec kernel parameters. Moreover, the size of the linux kernel and initramfs must be specified exactly in the linux boot protocol but those sizes will not be known until after Fiwix is booted so we cannot use boot parameters to tell Fiwix those sizes. So, those sizes need to be conveyed to the Fiwix kernel somehow as part of initiating the kexec. The design I came up with allows using the same kexec kernel parameters that multiboot uses.

The Linux boot parameters takes more than one page so the size of the boot memory was increased to two pages and the location adjusted accordingly.

Note that the disabling IRQs does not work for booting Linux so I moved that code to after the kexec code.

Note that some of the code to prepare the linux boot parameters was adapted from the Limine project and so the copyright notices reflect that.

--

Test files

The following files are used to aid in testing:

linux-4.9.10
kexec-linux.c
initramfs.cpio

Unfortunately I could not upload initramfs.cpio.gz because at 133MB it was too large for github. I'm not sure how to send it to you but you can email me at grick23 at gmail.com and perhaps we can arrange something. Or you can follow the following instructions to generate the file yourself. In the mean time it is possible to test kexec without a initramfs as explained later.

Note, for future reference, the following section describes how these test files were produced.

Producing test files

Clone and get ready to run the live-bootstrap project:

git clone https://github.com/fosslinux/live-bootstrap
cd live-bootstrap

In the file steps/jump/linux.sh replace this line:

    kexec-linux "/dev/ram1" "/boot/linux-4.9.10" "!$(command -v gen_init_cpio) /initramfs.list"

With this:

    $(command -v gen_init_cpio) /initramfs.list > /initramfs.cpio || true
    sync; read line
    kexec-linux "/dev/ram1" "/boot/linux-4.9.10" "!$(command -v gen_init_cpio) /initramfs.list"

Start live-bootstrap:

./rootfs.py --qemu

After 60 minutes or so live-bootstrap will hit the read line command and stop for input. You will see the following on the screen:

linux-4.9.10: install to fakeroot.
linux-4.9.10: postprocess binaries.
linux-4.9.10: creating package.
linux-4.9.10: cleaning up.
linux-4.9.10: installing package.
linux-4.9.10: build successful
Unrecognized nod format '/dev/sda 600 0 0 b Hr Lr' line 7071
Unrecognized nod format '/dev/sda1 600 0 0 b Hr Lr' line 7072
Unrecognized nod format '/dev/sda2 600 0 0 b Hr Lr' line 7073
Unrecognized nod format '/dev/sda3 600 0 0 b Hr Lr' line 7074
Unrecognized nod format '/dev/sdb 600 0 0 b Hr Lr' line 7075
Unrecognized nod format '/dev/sdb1 600 0 0 b Hr Lr' line 7076
Unrecognized nod format '/dev/sdb2 600 0 0 b Hr Lr' line 7077
Unrecognized nod format '/dev/sdc 600 0 0 b Hr Lr' line 7078
Unrecognized nod format '/dev/sdc1 600 0 0 b Hr Lr' line 7079
Unrecognized nod format '/dev/sdc2 600 0 0 b Hr Lr' line 7080
Unrecognized nod format '/dev/sdc3 600 0 0 b Hr Lr' line 7081

At this point, you must go to the qemu console by pressing ctrl-a followed by c. At the qemu prompt type the following commands, which will copy the ram disk to a file:

QEMU 4.2.1 monitor - type 'help' for more information
(qemu) pmemsave 0x001c6000 0x48000000 initrd.suspend
(qemu) quit

Now you can mount the ram disk and extract the linux kernel image, the ram drive cpio file used by linux, and the kexec-linux.c launching code:

cd ~/live-bootstrap
mkdir mnt
sudo mount -t ext2 initrd.suspend mnt
cp mnt/boot/linux-4.9.10 ~/
cp mnt/initramfs.cpio ~/
sudo umount mnt
cp steps/kexec-linux-1.0.0/files/kexec-linux.c ~/

Note that for testing on FiwixOS you'll need to change the following line in kexec-linux.c from this:

        reboot(RB_HALT_SYSTEM);

to this:

        system("halt");

The kexec-linux.c program I am providing here is also different in that it does not require providing an initial ram drive image.

--

Testing kexec linux

Once you have the test files, you can try to kexec Linux from Fiwix.

You'll need to build the Fiwix kernel with the CONFIG_KEXEC parameter.

echo "#define CONFIG_KEXEC" > include/fiwix/custom_config.h
make clean
make CONFFLAGS="-DCUSTOM_CONFIG_H"

You'll need to copy the test files to the Fiwix hard drive.

Note: I'm using the same FiwixOS-3.2-i386.raw downloaded from Fiwix.org except that the S0 terminal has been uncommented near the bottom of /etc/inittab to enable /dev/ttyS0 so text / serial console works with qemu.

BEWARE: RUNNING THIS TEST WILL REFORMAT THE HARD DRIVE! BEWARE: RUNNING THIS TEST WILL REFORMAT THE HARD DRIVE! BEWARE: RUNNING THIS TEST WILL REFORMAT THE HARD DRIVE!

You'll need to use the correct qemu command line to launch Fiwix:

qemu-system-i386 \
    -enable-kvm \
    -nographic \
    -machine pc \
    -cpu 486 \
    -drive file=FiwixOS-3.2-i386.raw,format=raw,if=ide,cache=writeback,index=0 \
    -nic user,ipv6=off,model=e1000 \
    -kernel fiwix \
    -append "console=/dev/ttyS0 root=/dev/hda2 kexec_proto=linux kexec_size=280000 kexec_cmdline=\"init=/init console=ttyS0\"" \
    -m size=4G

Login to Fiwix as root/root and run:

cc -m32 kexec-linux.c -o kexec-linux
./kexec-linux /dev/ram0 ./linux-4.9.10 ./initramfs.cpio
# After about 30 seconds linux should boot and start running gcc commands

Note that an initramfs image is optional. If not used, a size of zero should be written to the ram drive. I used this to boot Linux directly with a hard drive root using the FiwixOS 3.2 image. Yes, Linux can boot FiwixOS!

The steps required for this are to change the qemu append parameter:

    -append "console=/dev/ttyS0 root=/dev/hda2 kexec_proto=linux kexec_size=280000 kexec_cmdline=\"root=/dev/sda2 init=/sbin/init console=ttyS0\"" \

After booting Fiwix, change device from hda to sda:

# sed -i 's/hda/sda/g' /etc/rc.d/rc.sysinit

Then run kexec-linux without an initramfs file name:

# ./kexec-linux /dev/ram0 ./linux-4.9.10

kexec-linux.c.gz linux-4.9.10.gz

[mikaku]

I tried to kexec two different Linux kernels (2.0.30 and 4.19.254) with the parameters kexec_proto=linux kexec_size=2000 kexec_cmdline="ro root=/dev/hda2 but it doesn't work.

I see the messages:

kexec_linux: kernel file size: 1179403647
kexec_linux: Invalid kernel signature

**    Safe to Power Off    **
            -or-
** Press Any Key to Reboot **

The Linux kernel 2.0.30 has a size of 682886 bytes (666KB). The Linux kernel 4.19.254 has a size of 1261604 bytes (1232KB).

Does this implementation supports kexec the Linux kernel without any initramfs?

[rick-masters]

It does support kexec without a initramfs. The kernel file size sounds incorrect. Did you use the kexec-linux.c program to perform the kexec? Note that I attached a kernel for you to test with. If you want to use a different kernel could you upload the kernels you are testing with so I could try it?

[mikaku]

It does support kexec without a initramfs.

Great.

Did you use the kexec-linux.c program to perform the kexec?

No, I followed the documentation in file docs/kexec.txt. I just copied the kernel file directly to /dev/ram0 as I do when kexec the Fiwix kernel:

cp vmlinux-2.0.30 /dev/ram0

Note that I attached a kernel for you to test with. If you want to use a different kernel could upload the kernels you are testing with?

Yes. but I thought that any Linux kernel would work. I mean, I don't expect to prepare a specific Linux kernel to be kexec'd.

Here are the two Linux kernels:

vmlinux-2.0.30.gz vmlinux-4.19.254.gz

[mikaku]

Forgot to say that the Linux kernel 4.19.254 don't has the ATA disk driver, so it will panic because it cannot find the init program. This is harmless, I just wanted to see if it boots correctly.

[rick-masters]

In order to support multiple files, the cp command won't work. The docs/kexec.txt was updated at the bottom. I know a simple cp would be nice but the kernel needs to get multiple files and need the exact size of the files so the protocol is more sophisticated for linux, hence the kexec-linux program to help. Note that in Linux performing a kexec is a system call and requires a fairly complicated program to make it work so this isn't too unusual.

[mikaku]

Well, the information in docs/kexec.txt is a little vague. I think that it would be useful to have an example using the command dd to write the kernel size into the ramdisk drive and then using the same command again to write the whole kernel file with an offset of 8 bytes into the same ramdisk drive.

I'll work on this.

[rick-masters]

Well, the information in docs/kexec.txt is a little vague. I think that it would be useful to have an example using the command dd to write the kernel size into the ramdisk drive and then using the same command again to write the whole kernel file with an offset of 8 bytes into the same ramdisk drive.

I'll work on this.

Thanks, that's very helpful. If you need me to do it let me know. To be honest I did the documentation this morning and was a bit exhausted by the PR already and probably should have did a better job.

[mikaku]

Thanks, that's very helpful. If you need me to do it let me know. To be honest I did the documentation this morning and was a bit exhausted by the PR already and probably should have did a better job.

It's fine, no worries. You already did the big part. I'm just focusing on the details.

[mikaku]

With the Linux kernel vmlinux-4.19.254 with a file size of 1261604 bytes, I'm using the following commands:

# size=$(printf "%08x" `stat -c %s vmlinux-4.19.254` | tac -rs .. | echo "$(tr -d '\n')")
# echo -n $size | xxd -r -p - /dev/ram0
# dd if=vmlinux-4.19.254 of=/dev/ram0 bs=1c seek=4

But I still get the following messages:

kexec_linux: kernel file size: 1261604
kexec_linux: Invalid kernel signature

**    Safe to Power Off    **
            -or-
** Press Any Key to Reboot **

Any idea what I could be missing?

[rick-masters]

You will need to write an initrd size of zero and use a seek=8 offset for the kernel

[rick-masters]

I think there is something going on with the format of your kernel. This works for me:

KERNEL=linux-4.9.10
#KERNEL=vmlinux-4.19.254

size=$(printf "%08x" `stat -c %s $KERNEL` | tac -rs .. | echo "$(tr -d '\n')")
echo -n $size | xxd -r -p - /dev/ram0
size=$(printf "%08x" `echo 0` | tac -rs .. | echo "$(tr -d '\n')")
echo -n $size | xxd -r -p - | dd of=/dev/ram0 bs=1 seek=4
dd if=$KERNEL of=/dev/ram0 bs=1 seek=8
halt

[rick-masters]

Here it is with support for initramfs. This takes several minutes to dd the initramfs to the ram drive. Perhaps there is a way to speed this up but it works for me.

KERNEL=linux-4.9.10
#KERNEL=vmlinux-4.19.254
RAMFS=initramfs.cpio

size=$(printf "%08x" `stat -c %s $KERNEL` | tac -rs .. | echo "$(tr -d '\n')")
echo -n $size | xxd -r -p - /dev/ram0

if [ -n "$RAMFS" ]; then
        size=$(printf "%08x" `stat -c %s $RAMFS` | tac -rs .. | echo "$(tr -d '\n')")
else
        size=$(printf "%08x" `echo 0` | tac -rs .. | echo "$(tr -d '\n')")
fi
echo -n $size | xxd -r -p - | dd of=/dev/ram0 bs=1 seek=4

cat $KERNEL $RAMFS | dd of=/dev/ram0 bs=1 seek=8
halt

[rick-masters]

$ file linux-4.9.10 
linux-4.9.10: Linux kernel x86 boot executable bzImage, version 4.9.10-gnu_1 (@) #1 SMP PREEMPT @0, RO-rootFS, swap_dev 0x4, Normal VGA

$ file vmlinux-4.19.254 
vmlinux-4.19.254: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked, BuildID[sha1]=99e4ecad71ee683822d3f87e4a919612f2915908, d

Based on this: https://stackoverflow.com/questions/74325349/exact-difference-between-image-and-vmlinux#:~:text=Basing%20on%20this%20%22vmlinux%22%20ELF,the%20kernel%20on%20the%20system.

Apparently the ELF vmlinux needs to be processed further to create a boot image ?

[mikaku]

You will need to write an initrd size of zero and use a seek=8 offset for the kernel

Ah!, that's it.

I think there is something going on with the format of your kernel.

Yes, it also works here but only with your kernel. Using my kernel 4.19.254 still returns the message kexec_linux: Invalid kernel signature.

Here it is with support for initramfs. This takes several minutes to dd the initramfs to the ram drive. Perhaps there is a way to speed this up but it works for me.

Nice script! If you don't mind, I'll include your script into docs/kexec.txt, to help people to get an idea of how to do it.

Apparently the ELF vmlinux needs to be processed further to create a boot image ?

Well, I use the raw Linux kernel file on my boot floppy disks using GRUB v1 (now called GRUB Legacy). They work well, but indeed they don't work if you try to boot them directly with QEMU using the parameter -kernel. QEMU says:

qemu-system-i386: Error loading uncompressed kernel without PVH ELF Note

Hmm, now I've tried to boot yours (4.9.10) using QEMU and -kernel but I only get a blank screen. Weird.

[mikaku]

I've modified the file docs/kexec.txt. Please let me know if you find something misleading.

[rick-masters]

The docs/kexec.txt looks great. Thanks. I noticed this change was made on your PR66 branch...

[mikaku]

I noticed this change was made on your PR66 branch...

Damn, I thought I was changing your PR66. I'll try to remove that branch 'PR66' and I'll do the changes into your PR.

[mikaku]

OK, I've changed the docs/kexec.txt file in your own PR66 with the same modifications I did in my branch. Please, rebase your PR. Thanks.

[mikaku]

Thank you very much.