search

fabianishere / kexec-mod

Kexec as loadable kernel module for Linux ARM64 kernels
GNU General Public License v2.0
16 stars 3 forks source link

Build error on Raspberry Pi 4 #1

Open xuancong84 opened 2 years ago

xuancong84 commented 2 years ago

This is a very useful tool because it is not easy to rebuild the kernel without kernel config parameters. All you need to do is to apt install Linux kernel header then build this module.

However, on Raspberry Pi 4, the build process fails with the following error:

root@raspberrypi:~/kexec-mod/kernel# make KDIR=/usr/src/linux-headers-5.10.63-v8+/
make -C /usr/src/linux-headers-5.10.63-v8+/ M=/root/kexec-mod/kernel modules
make[1]: Entering directory '/usr/src/linux-headers-5.10.63-v8+'
  CC [M]  /root/kexec-mod/kernel/arch/arm64/idmap.o
/root/kexec-mod/kernel/arch/arm64/idmap.c: In function ‘kexec_pa_symbol’:
/root/kexec-mod/kernel/arch/arm64/idmap.c:118:19: error: passing argument 1 of ‘pud_offset’ from incompatible pointer type [-Werror=incompatible-pointer-types]
  pud = pud_offset(pgd , va);
                   ^~~
In file included from ./arch/arm64/include/asm/pgtable-types.h:50,
                 from ./arch/arm64/include/asm/page.h:16,
                 from ./arch/arm64/include/asm/proc-fns.h:14,
                 from ./arch/arm64/include/asm/pgtable.h:9,
                 from /root/kexec-mod/kernel/arch/arm64/idmap.c:15:
./include/asm-generic/pgtable-nopud.h:42:40: note: expected ‘p4d_t *’ {aka ‘struct <anonymous> *’} but argument is of type ‘pgd_t *’ {aka ‘struct <anonymous> *’}
 static inline pud_t *pud_offset(p4d_t *p4d, unsigned long address)
                                 ~~~~~~~^~~
cc1: some warnings being treated as errors
make[3]: *** [scripts/Makefile.build:280: /root/kexec-mod/kernel/arch/arm64/idmap.o] Error 1
make[2]: *** [scripts/Makefile.build:497: /root/kexec-mod/kernel/arch/arm64] Error 2
make[1]: *** [Makefile:1825: /root/kexec-mod/kernel] Error 2
make[1]: Leaving directory '/usr/src/linux-headers-5.10.63-v8+'
make: *** [Makefile:15: module] Error 2
fabianishere commented 2 years ago

Hi,

I have only tested the module to work for Linux kernels up to 4.18. It seems that in more newer versions, the kernel developers have unified the page directory representation, requiring changes to my page walk implementation.

Could you check whether the following patch resolves the issue:

diff --git a/kernel/arch/arm64/idmap.c b/kernel/arch/arm64/idmap.c
index c644914..df87920 100644
--- a/kernel/arch/arm64/idmap.c
+++ b/kernel/arch/arm64/idmap.c
@@ -105,6 +105,7 @@ phys_addr_t kexec_pa_symbol(void *ptr)
    unsigned long va = (unsigned long) ptr;
    unsigned long page_offset;
    pgd_t *pgd;
+   p4d_t* p4d;
    pud_t *pud;
    pmd_t *pmd;
    pte_t *ptep, pte;
@@ -115,7 +116,12 @@ phys_addr_t kexec_pa_symbol(void *ptr)
        return 0;
    }

-   pud = pud_offset(pgd , va);
+   p4d = p4d_offset(pgd, vmpage);
+   if (p4d_none(*p4d) || p4d_bad(*p4d)) {
+       return 0;
+   }
+
+   pud = pud_offset(p4d , va);
    if (pud_none(*pud) || pud_bad(*pud)) {
        return 0;
    }

I recommend first testing the module in QEMU though, since it does some pretty hacky things and I cannot guarantee it still working correctly in Linux 5.10.

xuancong84 commented 2 years ago

Thanks for the quick reply! In addition to this error, there are still tones of other errors preventing a successful compilation:

  1. Deprecated symbols ENTRY() and ENDPROC()

    /home/pi/projects/raspberry-pi/kernel/kexec-mod/kernel/arch/arm64/cpu-reset.S: Assembler messages:
/home/pi/projects/raspberry-pi/kernel/kexec-mod/kernel/arch/arm64/cpu-reset.S:34: Error: unknown mnemonic `entry' -- `entry(__cpu_soft_restart)'
/home/pi/projects/raspberry-pi/kernel/kexec-mod/kernel/arch/arm64/cpu-reset.S:57: Error: unknown mnemonic `endproc' -- `endproc(__cpu_soft_restart)'
make[3]: *** [scripts/Makefile.build:361: /home/pi/projects/raspberry-pi/kernel/kexec-mod/kernel/arch/arm64/cpu-reset.o] Error 1
make[2]: *** [scripts/Makefile.build:497: /home/pi/projects/raspberry-pi/kernel/kexec-mod/kernel/arch/arm64] Error 2
make[1]: *** [Makefile:1825: /home/pi/projects/raspberry-pi/kernel/kexec-mod/kernel] Error 2

    This one is easy to resolve, just replace by SYM_FUNC_START and SYM_FUNC_END

  2. A few missing links

    /home/pi/projects/raspberry-pi/kernel/kexec-mod/kernel/arch/arm64/machine_kexec.c:2:10: fatal error: orig/machine_kexec.c: No such file or directory
/home/pi/projects/raspberry-pi/kernel/kexec-mod/kernel/kexec.c:8:10: fatal error: orig/kexec.c: No such file or directory

    This one also easy to solve, just download the full kernel source and make the correct soft link

  3. Tons of symbol not found

    ERROR: modpost: "crashk_res" [/home/pi/projects/raspberry-pi/kernel/kexec-mod/kernel/arch/arm64/kexec_mod_arm64.ko] undefined!
ERROR: modpost: "idmap_pg_dir" [/home/pi/projects/raspberry-pi/kernel/kexec-mod/kernel/arch/arm64/kexec_mod_arm64.ko] undefined!
ERROR: modpost: "reserved_pg_dir" [/home/pi/projects/raspberry-pi/kernel/kexec-mod/kernel/arch/arm64/kexec_mod_arm64.ko] undefined!
ERROR: modpost: "kallsyms_lookup_name" [/home/pi/projects/raspberry-pi/kernel/kexec-mod/kernel/arch/arm64/kexec_mod_arm64.ko] undefined!
ERROR: modpost: "swapper_pg_dir" [/home/pi/projects/raspberry-pi/kernel/kexec-mod/kernel/arch/arm64/kexec_mod_arm64.ko] undefined!
ERROR: modpost: "kimage_file_post_load_cleanup" [/home/pi/projects/raspberry-pi/kernel/kexec-mod/kernel/kexec_mod.ko] undefined!
ERROR: modpost: "kallsyms_lookup_name" [/home/pi/projects/raspberry-pi/kernel/kexec-mod/kernel/kexec_mod.ko] undefined!

  4. Not sure.

All these making the project hopeless for the new kernel -:(

xuancong84 commented 2 years ago 
diff --git a/kernel/arch/arm64/idmap.c b/kernel/arch/arm64/idmap.c
index c644914..df87920 100644
--- a/kernel/arch/arm64/idmap.c
+++ b/kernel/arch/arm64/idmap.c
@@ -105,6 +105,7 @@ phys_addr_t kexec_pa_symbol(void *ptr)
    unsigned long va = (unsigned long) ptr;
    unsigned long page_offset;
    pgd_t *pgd;
+   p4d_t* p4d;
    pud_t *pud;
    pmd_t *pmd;
    pte_t *ptep, pte;
@@ -115,7 +116,12 @@ phys_addr_t kexec_pa_symbol(void *ptr)
        return 0;
    }

-   pud = pud_offset(pgd , va);
+   p4d = p4d_offset(pgd, vmpage);
+   if (p4d_none(*p4d) || p4d_bad(*p4d)) {
+       return 0;
+   }
+
+   pud = pud_offset(p4d , va);
    if (pud_none(*pud) || pud_bad(*pud)) {
        return 0;
    }

Btw, your patch is also problematic because: kexec-mod/kernel/arch/arm64/idmap.c:119:24: error: ‘vmpage’ undeclared (first use in this function); did you mean ‘page’?

fabianishere commented 2 years ago

Some of these issues are indeed pretty easy to fix. However, this big issue is that kallsyms_lookup_name is not exported in newer kernels.

JoelLinn commented 2 years ago

Doesn't work with 5.4 indeed (not rpi but same issues)

make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- KDIR=~/Documents/dev/kernel-android-5.4 
make -C /home/joel/Documents/dev/kernel-android-5.4 M=/home/joel/Documents/dev/kexec-mod/kernel modules
make[1]: Entering directory '/home/joel/Documents/dev/kernel-android-5.4'
  Building modules, stage 2.
  MODPOST 2 modules
ERROR: "idmap_pg_dir" [/home/joel/Documents/dev/kexec-mod/kernel/arch/arm64/kexec_mod_arm64.ko] undefined!
ERROR: "reserved_pg_dir" [/home/joel/Documents/dev/kexec-mod/kernel/arch/arm64/kexec_mod_arm64.ko] undefined!
ERROR: "kallsyms_lookup_name" [/home/joel/Documents/dev/kexec-mod/kernel/arch/arm64/kexec_mod_arm64.ko] undefined!
ERROR: "swapper_pg_dir" [/home/joel/Documents/dev/kexec-mod/kernel/arch/arm64/kexec_mod_arm64.ko] undefined!
ERROR: "kallsyms_lookup_name" [/home/joel/Documents/dev/kexec-mod/kernel/kexec_mod.ko] undefined!
make[2]: *** [scripts/Makefile.modpost:114: __modpost] Error 1
make[1]: *** [Makefile:1729: modules] Error 2
make[1]: Leaving directory '/home/joel/Documents/dev/kernel-android-5.4'
make: *** [Makefile:15: module] Error 2
thesaadmemon commented 1 year ago

@fabianishere is it confirmed to work for 4.18? Thank you!

fabianishere commented 1 year ago

@thesaadmemon It has worked with 4.18 ARM64 kernels, but easiest would be to try building the module against your kernel configuration and checking if it runs.