embedded initramfs does not boot with kernel 3.8.3

Hi,

I have build the hardened-sources-3.8.3 with kigen and using an embedded initramfs using kigen.

When I choose to embed the initramfs into the kernel it kernel panics upon booting. When I specify initramfs separately in grub the kernel boots fine.

I'm not sure if this is a kigen or 3.8.3 kernel issue, but I don't think I had this problem with the 3.7 linux kernel.

Hi,

OK, will look at it, give me a few days. Thanks for reporting ;o)

Can you copy/paste the output with verbose and command line? Thanks

I can't access the machine right now, but maybe I can reproduce it in a VM later.

The steps was to just generate a initramfs using host busybox, luks2, lvm2, and disk label from source (somehow my blkid was not linked statically). Then just a normal kernel compile with "kigen kernel" and path to the embedded initramfs.

I did not see any errors in the final step, it even said: "generated kernel with embbeded initramfs". Anyways, i'll try to get back to you with more details later this week.

mmm, sounds like a kernel config issue.

"somehow my blkid was not linked statically"

have you tried using --dynlibs with 'kigen i'?

when possible copy/paste your 'kigen i ...' and 'kigen k ...' commands and output too. could be a bug in --source-disklabel if blkid is not statically linked. Then I guess it's missing libs if it's dynamically linked and --dynlibs might help. if --host-disklabel is called then --dynlibs should always be called too to detect its lib deps and ship it. if that's the issue, then I can make any --host-* option pull --dynlibs.

just deployed a luks VM using kicktoo -p profiles/gentoo/kbin/gentoo-luks-kbin.profile merged hardened-sources-3.8.6:3.8.6 copied kernel config, select the new source and kigen k kigen i --source-disklabel --host-luks --dynlibs --source-lvm2 kigen k --initramfs=/boot/initramfs-kigen-x86-3.8.6-hardened added to grub conf kernel /boot/kernel-kigen-x86-3.8.6-hardened root=/dev/ram0 init=/linuxrc ramdisk=8192 real_root=/dev/mapper/root crypt_root=/dev/sda3 and boot works ok for me. what am I missing?

I don't think the issue is the blkid in initramfs, because the generated initramfs seems to work separately.

I tried 3.8.3 (which was the only stable hardened at that day). I know that i only use the following kernel flags:

root=LABEL=root real_root= crypt_root=/dev/sda2

I have to get back to you next monday, sorry.

ok, I'll retry using those parameters.

On 01-05-2013 22:19, yunzheng wrote:

I don't think the issue is the blkid in initramfs, because the generated initramfs seems to work separately.

I tried 3.8.3 (which was the only stable hardened at that day). I know that i only use the following kernel flags:

root=LABEL=root realroot= cryptroot=/dev/sda2

I have to get back to you next monday, sorry.

Reply to this email directly or view it on GitHub [1].

Links:

[1] https://github.com/r1k0/kigen/issues/23#issuecomment-17303910

do you mean real_root=crypt_root=/dev/sda2 or is it rather real_root=/dev/mapper/root crypt_root=/dev/sda3? I need more accurate info in order to replicate your error (copy/paste of CLI+logfile+exact grub conf) else I can't really visualize your settings. Closing, reopen with more info.

Ok, i can reproduce it:

# kigen k
 * Gentoo Base System release 2.1 on x86_64
 * Kernel sources Makefile version 3.8.6-hardened aka DisplacedHumerusAnterior
 * kernel.oldconfig 
 * kernel.prepare 
 * kernel.bzImage 
 * kernel.modules 
 * kernel.modules_install /lib/modules
 * saved /etc/kernels/dotconfig-kigen-x86_64-3.8.6-hardened
 * boot.mounted
 * success 2.6Mb /boot/System.map-kigen-x86_64-3.8.6-hardened
 * success 5.2Mb /boot/kernel-kigen-x86_64-3.8.6-hardened
 * boot.umounted

# kigen i --host-disklabel --host-luks --dynlibs --host-lvm2 --host-busybox
 * Gentoo Base System release 2.1 on x86_64
 * initramfs.append.base Gentoo linuxrc 3.4.38 patched
 * ... /init
 * ... /etc/initrd.scripts
 * ... /etc/initrd.defaults
 * ... /etc/fstab
 * ... /dev/console
 * ... /dev/null
 * ... /dev/tty1
 * ... /sbin/modprobe
 * initramfs.append.modules 3.8.6-hardened
 * ... MODULES_SATA  
 * ... MODULES_DMRAID    
 * ... MODULES_MDADM     
 * ... MODULES_VIDEO     
 * ... MODULES_ISCSI     
 * ... MODULES_MISC  
 * ... MODULES_FS    
 * ... MODULES_CRYPTO    
 * ... MODULES_WAITSCAN  
 * ... MODULES_USB   
 * ... MODULES_SCSI  
 * ... MODULES_PATA  
 * ... MODULES_FIREWIRE  
 * ... MODULES_NET   
 * ... MODULES_LVM   
 * ... MODULES_EVMS  
 * ... MODULES_ATARAID   
 * ... MODULES_PCMCIA    
 * initramfs.append.host.busybox /bin/busybox
 * initramfs.append.host.lvm2 /sbin/lvm.static
 * initramfs.append.host.luks /sbin/cryptsetup
 * initramfs.append.host.disklabel /sbin/blkid
 * ... warning: /sbin/blkid is dynamically linked, copying detected libraries
 * ... /lib64/libblkid.so.1
 * ... /lib64/libc.so.6
 * ... /lib64/libuuid.so.1
 * ... /lib64/ld-linux-x86-64.so.2
 * initramfs.append.keymaps all
 * ... azerty be bg br-a br-l 
 * ... by cf croat cz de dk 
 * ... dvorak es et fi fr gr 
 * ... hu il is it jp keymapList 
 * ... la lt mk nl no pl 
 * ... pt ro ru se sg sk-y 
 * ... sk-z slovene trf trq ua uk 
 * ... us wangbe 
 * initramfs.compress
 * boot.mounted
 * success 3.7Mb /boot/initramfs-kigen-x86_64-3.8.6-hardened
 * boot.umounted

# kigen k --initramfs=/boot/initramfs-kigen-x86-3.8.6-hardened
 * Gentoo Base System release 2.1 on x86_64
 * Kernel sources Makefile version 3.8.6-hardened aka DisplacedHumerusAnterior
 * kernel.copy_config /usr/src/linux/.config -> /usr/src/linux/.config-2013-05-07-16-26-44
 * kernel.import_user_initramfs /boot/initramfs-kigen-x86-3.8.6-hardened
 * kernel.oldconfig 
 * kernel.prepare 
 * kernel.bzImage 
 * kernel.modules 
 * kernel.modules_install /lib/modules
 * saved /etc/kernels/dotconfig-kigen-x86_64-3.8.6-hardened
 * boot.mounted
 * success 2.6Mb /boot/System.map-kigen-x86_64-3.8.6-hardened
 * success 5.2Mb /boot/kernel-kigen-x86_64-3.8.6-hardened with an embedded initramfs
 * boot.umounted

This is my /boot/grub/menu.lst config (using grub-static-0.97-r12):

title=Linux kernel-kigen-x86_64-3.8.6-hardened
root (hd0,0)
kernel /boot/kernel-kigen-x86_64-3.8.6-hardened real_root=LABEL=root crypt_root=/dev/sda2

This above grub doesn't work with the embedded initramfs, it will result in a kernel panic. But it works if I specify the initrd separately.

Maybe it's because of my kernel config?

Hi again,

kernel .config might be an issue but if it works with initrd specified I assume that the delta between the 2 .config will only concern the initramfs.

Can you make sure CONFIG_INITRAMFS_SOURCE= is correclty set in .config when it's shipped inside the kernel? It would make better sense if both cases were failing.

Hi, i think my problem is that i pointed --initramfs= to a non existing file. (see my log, i used x86 instead of x86_64).

I think kigen should give an error about that :)

Now I have a different problem, is that my initramfs doesn't see my devices. But I don't think that's related to kigen.

good eyes!!! indeed _x64 is missing... So the problem is between chair and keyboard ;o)

Will add a file check on the --initramfs= input

thanks for spotting, I was actually creating a luks/lvm profile to match your setup on kicktoo (still in progress) and was planning on copy/pasting your output. glad you found the issue.

I think my actual problem was that I created the kernel with embedded initramfs using:

# kigen k --clean --initramfs=xxxx

The --clean seems to break the actual merging of the initramfs. If I create the kernel separately first and then again using --initramfs= and without --clean it seems to work.

I wonder if you can reproduce it as well.

Then, running 'make clean' removes your entry CONFIG_INITRAMFS_SOURCE= in .config. Sucks.

do this grep CONFIG_INITRAMFS_SOURCE /usr/src/linux/.config cd /usr/src/linux make clean grep CONFIG_INITRAMFS_SOURCE /usr/src/linux/.config

except with --fixdotconfig, kigen does not touch .config at all. So I don't understand how it can have effect on your .config. Given this, it has to come from the actual kernel Makefile and the 'clean' target.

kernel.py

79 if (self.clean is True) or (self.clean == 'True' ): 80 if self.make_clean() is not zero: self.fail('clean')

121 if self.initramfs is not '': 126 self.import_user_initramfs(self.initramfs) 133 elif (self.kernel_conf['initramfs'] is not '') and (self.initramfs is ''): 136 self.import_user_initramfs(self.kernel_conf['initramfs'])

# grep CONFIG_INITRAMFS_SOURCE /usr/src/linux/.config
CONFIG_INITRAMFS_SOURCE=""

# cd /usr/src/linux
# make clean
  CLEAN   .
  CLEAN   arch/x86/kernel/cpu
  CLEAN   arch/x86/kernel
  CLEAN   arch/x86/realmode/rm
  CLEAN   arch/x86/vdso
  CLEAN   arch/x86/lib
  CLEAN   drivers/scsi/aic7xxx
  CLEAN   drivers/tty/vt
  CLEAN   drivers/video/logo
  CLEAN   firmware
  CLEAN   kernel
  CLEAN   lib
  CLEAN   usr
  CLEAN   arch/x86/boot/compressed
  CLEAN   arch/x86/boot
  CLEAN   arch/x86/tools
  CLEAN   .tmp_versions

# grep CONFIG_INITRAMFS_SOURCE /usr/src/linux/.config
CONFIG_INITRAMFS_SOURCE=""

looks ok to me

fair enough but kigen does not touch .dotconfig. So the cause is outside kigen and outside 'make clean'. Must be human then. Especially since it worked before. The code has not changed.

was just wondering if you could reproduce it with --clean

ok then I'll do it on a luks non lvm one. In this particular case it should not matter. Wait an hour or 2 I'll test it now

Ok thanks, I can even reproduce it with these 2 scripts:

$ cat test1.sh

mount /boot
kigen k --clean --dotconfig=~/linux-x86_64-3.8.6-hardened.config
kigen i --dynlibs --host-disklabel --host-luks --host-lvm2 --host-busybox --rename=/tmp/ramfs
kigen k --clean --initramfs=/tmp/ramfs --rename=/boot/kernel1.bzImage

$ cat test2.sh

mount /boot
kigen k --clean --dotconfig=~/linux-x86_64-3.8.6-hardened.config
kigen i --dynlibs --host-disklabel --host-luks --host-lvm2 --host-busybox --rename=/tmp/ramfs
kigen k --initramfs=/tmp/ramfs --rename=/boot/kernel2.bzImage

kernel1 panic's with block device error, kernel2 boots fine.

It looks like kernel1.bzImage does not include the initramfs:

# ls -lah /boot/kernel[12].bzImage
-rw-r--r-- 1 root root 9.4M May  8 21:33 kernel1.bzImage
-rw-r--r-- 1 root root  13M May  8 21:36 kernel2.bzImage

mmm, in both cases I end up with a kernel panic... strange.

ok, got it to boot using kigen k kigen i --dynlibs --host-luks --source-disklabel kigen k --initramfs=/boot/kernel-kigen-x86-3.8.6-hardened with kernel /boot/kernel-kigen-x86-3.8.6-hardened init=/linuxrc root=/dev/ram0 real_root=/dev/mapper/root crypt_root=/dev/sda3

However, if I boot using kernel /boot/kernel-kigen-x86-3.8.6-hardened init=/linuxrc real_root=/dev/mapper/root crypt_root=/dev/sda3 initramfs fails with: Could not find root block device in LABEL=root That may be caused because my root device is not labelled (I've never used LABEL myself).

So, I'll go forward using kernel /boot/kernel-kigen-x86-3.8.6-hardened init=/linuxrc root=/dev/ram0 real_root=/dev/mapper/root crypt_root=/dev/sda3 it doesn't look like a label issue anyway

Now, if I use kigen k --initramfs=/boot/kernel-kigen-x86-3.8.6-hardened --clean and the same boot config, I should have a kernel panic. Testing... indeed kernel panic. if I rerun kigen k --initramfs=/boot/kernel-kigen-x86-3.8.6-hardened it boots

so yes i can reproduce but i don't understand why. somehow --clean , hence the 'make clean' step, is responsible for this.

if I run kigen k --initramfs=/boot/kernel-kigen-x86-3.8.6-hardened cd /usr/src/linux make clean kigen k --initramfs=/boot/kernel-kigen-x86-3.8.6-hardened I get kernel panic. So the problem comes from 'make clean'.

Glad you are able to reproduce it as well. Did you check the size of the resulting kernel? I think the initramfs is not embedded when you do --clean.

On Thu, May 9, 2013 at 1:40 PM, r1k0 notifications@github.com wrote:

if I run kigen k --initramfs=/boot/kernel-kigen-x86-3.8.6-hardened cd /usr/src/linux make clean kigen k --initramfs=/boot/kernel-kigen-x86-3.8.6-hardened I get kernel panic. So the problem comes from 'make clean'.

— Reply to this email directly or view it on GitHubhttps://github.com/r1k0/kigen/issues/23#issuecomment-17660106 .

indeed the cleaned version has not the initramfs inside.

even though --initramfs appends the initramfs AFTER --clean is run, the initramfs is not part of the kernel.

if I understood the logic behind I could work around it (seems like --clean removes files that prevent shipping the initramfs BUT if rerun without --clean then it works)

looks like an upstream strangeness.

Im planning on throwing some time and effort to kigen when the mood is right so I'll dig this one. Out of context, there is a lot I don't remember and given what I read I'm tempted to point the upstream Makefile for this bug.

That's ok, thanks for looking into it. I have my workaround, so I can at least produce working kernels with embedded initramfs consistently.

On Thu, May 9, 2013 at 2:35 PM, r1k0 notifications@github.com wrote:

Im planning of throwing some time and effort to kigen when the mood is right so I'll dig this one. Out of context, there is a lot I don't remember and given what I read I'm tempted to point the upstream Makefile for this bug.

— Reply to this email directly or view it on GitHubhttps://github.com/r1k0/kigen/issues/23#issuecomment-17661983 .

r1k0 / kigen

embedded initramfs does not boot with kernel 3.8.3 #23

Links: