FATAL: kernel too old with linux kernel 5.9.2

Dauto98 commented 3 years ago

Hello cirosantilli2,

I am building the linux kernel by myself to use in Gem5 FS simulation by following the step I found in ./build --download-dependencies --arch aarch64 gem5-buildroot --dry-run. what I do is:

clone linux version 5.9.2: git clone --depth 1 --branch v5.9.2 git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
copy the config from linux_config/buildroot-aarch64 in your repo to .config in Linux repo
copy the config from linux_config/min and linux_config/default in your repo to config.min and config.def in Linux repo
run ./scripts/kconfig/merge_config.sh -m -O ./output .config config.min config.def
run make ARCH=arm64 mrproper, I have to do this step otherwise make will complain about the source tree is not clean.
run make -j 8 ARCH=arm64 CROSS_COMPILE=/usr/bin/aarch64-linux-gnu- CC=/usr/bin/aarch64-linux-gnu-gcc O=./output/ olddefconfig
run make -j 8 ARCH=arm64 CROSS_COMPILE=/usr/bin/aarch64-linux-gnu- CC=/usr/bin/aarch64-linux-gnu-gcc O=./output/

The kernel is in output/vmlinux.

The command I use to run Gem5

# in Gem5 repo folder
sudo ./build/ARM/gem5.opt configs/example/fs.py --machine-type=VExpress_GEM5_V1 --kernel=../linux/output/vmlinux --bootloader=system/arm/bootloader/arm64/boot.arm64 --disk-image=../buildroot-2021.02/output/images/rootfs.ext2 --command-line 'root=/dev/sda earlycon=pl011,0x1c090000 earlyprintk=pl011,0x1c090000 lpj=19988480 rw loglevel=8 mem=256MB console_msg_format=syslog nokaslr norandmaps panic=-1 printk.devkmsg=on printk.time=y rw console=ttyAMA0 - lkmc_home=/lkmc'

The error when I run Gem5:

FATAL: kernel too old
<0>[    0.312789] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00007f00
<4>[    0.312796] CPU: 0 PID: 1 Comm: init Not tainted 5.9.2 #1
<4>[    0.312800] Hardware name: V2P-CA15 (DT)
<4>[    0.312804] Call trace:
<4>[    0.312809]  dump_backtrace+0x0/0x210
<4>[    0.312813]  show_stack+0x2c/0x38
<4>[    0.312818]  dump_stack+0xe4/0x134
<4>[    0.312823]  panic+0x168/0x374
<4>[    0.312828]  do_exit+0x6b0/0x998
<4>[    0.312833]  do_group_exit+0x48/0xf0
<4>[    0.312838]  __arm64_sys_exit_group+0x24/0x28
<4>[    0.312844]  el0_svc_common.constprop.4+0x7c/0x190
<4>[    0.312850]  do_el0_svc+0x80/0xa0
<4>[    0.312855]  el0_sync_handler+0x160/0x1f0
<4>[    0.312859]  el0_sync+0x140/0x180
<0>[    0.312863] Kernel Offset: disabled
<0>[    0.312867] CPU features: 0x0040020,28402412
<0>[    0.312871] Memory Limit: 256 MB
<4>[    0.312875] Reboot failed -- System halted

The bootloader I built in Gem5 by running make -C system/arm/bootloader/arm64/. The disk-image I also mimic the step you did in the ./build script. However, the error is in the kernel, so I think the disk-image has nothing to do with this.

The error is kernel too old. However, I use v5.9.2, which is almost the latest version now (the latest is 5.9.9), and the version you use in the build script is 5.9.2. The config file from the Linux repo and your repo is almost the same, except for some debugging config I think, and of course the compiler.

Dauto98 commented 3 years ago

The compiler I am using is aarch64-linux-gnu-gcc

aarch64-linux-gnu-gcc (Ubuntu/Linaro 7.5.0-3ubuntu1~18.04) 7.5.0

just tested with aarch64-none-linux-gnu-gcc

aarch64-none-linux-gnu-gcc (GNU Toolchain for the A-profile Architecture 10.2-2020.11 (arm-10.16)) 10.2.1 20201103

taken from the ARM website, it still has the same problem.

cirosantilli2 commented 3 years ago

Hi Duc,

I'm not sure what's going on, so I'll propose the stupid debugging approach:

use QEMU for testing instead of gem5 as it boots in 5 seconds and this is purely a software problem so will be reproducible there as well
check that with the Linux image build with my scripts it works
if yes, diff out your image from my image. Notably: are the final generated Linux .config the same for both builds? And are the source versions also exactly the same?

Edit: saw you mentioned "The config file from the Linux repo and your repo is almost the same, except for some debugging config I think, and of course the compiler.". I'd try to understand exactly why they differ.

At start of boot the kernel shows its version:

<5>[    0.000000] Linux version 5.9.2

is that what you see?

Dauto98 commented 3 years ago

Hi,

I can't make it work with Qemu, all it showed is a black screen written compat_monitor0 console and nothing else happen. The command I use is

qemu-system-aarch64   
    -M virt   
    -cpu cortex-a57   
    -kernel ../linux-kernel-module-cheat/out.docker/linux/default/aarch64/vmlinux   
    -append "root=/dev/vda console=ttyAMA0"   
    -netdev user,id=eth0   
    -device virtio-net-device,netdev=eth0   
    -drive file=../linux-kernel-module-cheat/out.docker/buildroot/build/default/aarch64/images/rootfs.ext2,if=none,format=raw,id=hd0   
    -device virtio-blk-device,drive=hd0 ;

The kernel and drive I tried both the images from your docker output and from my linux and buildroot repo which I built separately. They didn't work.

However, booting Gem5 with your kernel image and my disk-image from my buildroot repo work. Changing the kernel to the one I built will make Gem5 fail.

At start of boot the kernel shows its version:
<5>[    0.000000] Linux version 5.9.2
is that what you see?

Yes, the full line is

Linux version 5.9.2 (dauto98@Dauto98-ROG-Strix-G531GD) (aarch64-none-linux-gnu-gcc (GNU Toolchain for the A-profile Architecture 10.2-2020.11 (arm-10.16)) 10.2.1 20201103, GNU ld (GNU Toolchain for the A-profile Architecture 10.2-2020.11 (arm-10.16)) 2.35.1.20201028) #3 SMP Tue Mar 30 22:57:30 +07 2021

And here is the comparison between your config and my config. Your config is on the left, mine is on the right.

Dauto98 commented 3 years ago

Hi,

Maybe the problem actually lies in the disk image, not the kernel.

I just rebuilt the kernel in your repo and tried with Gem5 again. Here is how I build the kernel

Remove the out/ folder
run ./build-buildroot --arch aarch64 so that we have the buildroot toolchain which is used to build the kernel and the disk image
run ./build-linux --arch aarch64 to build the kernel

Then, I ran Gem5 with the following command

./build/ARM/gem5.opt configs/example/fs.py 
    --machine-type=VExpress_GEM5_V1 
    --kernel=../linux-kernel-module-cheat/out/linux/default/aarch64/vmlinux # kernel built above
    --bootloader=system/arm/bootloader/arm64/boot.arm64 
    --disk-image=../buildroot-2021.02/output/images/rootfs.ext2 # disk-image I built in a separate buildroot repo
    --command-line 'root=/dev/sda earlycon=pl011,0x1c090000 earlyprintk=pl011,0x1c090000 lpj=19988480 rw loglevel=8 mem=256MB console_msg_format=syslog nokaslr norandmaps panic=-1 printk.devkmsg=on printk.time=y rw console=ttyAMA0 - lkmc_home=/lkmc'

The kernel is built above, and the disk image I built in a separate buildroot repo. I follow similar steps in your ./build-buildroot to build the disk image, but without your rootfs_overlay, benchmark, and kernel module files, and some config you append using the cat << EOF command.

The result is the FATAL: kernel too old error I got in the first comment.

However, the problem vanishes when I change to the image from your buildroot output in your repo

./build/ARM/gem5.opt configs/example/fs.py 
    --machine-type=VExpress_GEM5_V1 
    --kernel=../linux-kernel-module-cheat/out/linux/default/aarch64/vmlinux # same kernel as above
    --bootloader=system/arm/bootloader/arm64/boot.arm64 
    --disk-image=../linux-kernel-module-cheat/out/buildroot/build/default/aarch64/images/rootfs.ext2 # changed 
    --command-line 'root=/dev/sda earlycon=pl011,0x1c090000 earlyprintk=pl011,0x1c090000 lpj=19988480 rw loglevel=8 mem=256MB console_msg_format=syslog nokaslr norandmaps panic=-1 printk.devkmsg=on printk.time=y rw console=ttyAMA0 - lkmc_home=/lkmc'

even when I use my kernel built from a separated repo (the one from the first comment of this issue), it still works. Gem5 left me at the buildroot login prompt. It didn't look like an Ubuntu is installed on the image, well but the thing here is the Kernel too old error has gone.

Here is how I build the image in buildroot repo

make O=./output/ qemu_aarch64_virt_defconfig
copy linux-kernel-module-cheat/buildroot_config/default to .config in buildroot repo
make O=./output/ olddefconfig
make O=./output/ all

Yes I left out some of your configs and the rootfs_overlay and the kernel modules and maybe some other stuff I don't know so probably those cause the problem.

Dauto98 commented 3 years ago

I found out that it was the kernel header version config in buildroot. It was set to 5.10.x so indeed the 5.9.2 kernel version would fail. I change that config to 5.4.x (BR2_KERNEL_HEADERS_5_4=y) and it works..

I am using buildroot-2021.02 version btw.

cirosantilli2 / issues

FATAL: kernel too old with linux kernel 5.9.2 #18