Missing module.symvers

[Lekensteyn]

The module.symvers file is necessary to build out-of-tree kernel modules. Without this file, a full kernel build has to be performed.

Background: a user on RPI forums using XBian is unable to load a kernel module built for other kernels. The kernel module has to be recompiled for XBian, but that is not easily done as (1) module.symvers is missing (2) kernel headers are not contained in a single package.

This issue targets (1) by requesting inclusion of that file. As a workaround for (2), I will try to grab the headers from tag 3.6.11 and hope for the best. (note: it will be used for cross-compilation, the RPi is too slow for doing large module builds)

[mk01]

as for the headers, this is publicly available as part of wheezy distro - check "apt-cache show linux-headers-3.6-trunk-common"

as for the symvers file, we can include this in package.

[Lekensteyn]

Those headers are for 3.6.9. Usually there is not a big difference for minor versions, but it would still be nice to use the same kernel headers as used for the image.

Since the headers are to be used for building modules, the symvers file could be included within the linux-headers-xxx package. In the xbian-deb-sources repo, I cannot find a rules file for the kernel package, I wonder how you build your packages? The kernel/*/control points to a single package, usually kernel headers are also included with the control file.

[mk01]

since I use RPi I compiled the kernel on various machines as cross platform against locally installed header file version from 3.2 to 3.9 with no issues.

if you check the wheezy repo you will realize, that there is no even kernel in distro, no sources, no header files. kernel is provided as unpacked monolithic sources.

[mk01]

ok, stupid things am I saying, of course the local system kernel-headers does not matter because all is included inside the sources provided by rpi project. ;)

[Lekensteyn]

So, how do you package the kernel for xbian? The instructions on https://github.com/xbianonpi/xbian/wiki/Compile-Kernel looks untested....

sudo cd ..

[mk01]

compile from sources and build package from raw files

[mk01]

btw: if you install dkms, nothing more than again linux-headers-3.6-trunk is installed.

if there would be real demand for full environment (header files provided as well as binary and pre packaged sources) it's not a big deal to prepare it from latest git upstream. but beside you there is no demand and even debian is not providing this because specifics how the kernel is being developed.

[Lekensteyn]

I am not asking for packages containing the source (which is actually provided by Debian, see providers of linux-source), just the kernel headers (make headers_install or sth). Debian provides for each package the original source tarball plus Debian modifications (which includes a rules file). This still holds for kernel packages.

Most patches and minor versions should not affect the headers too much, I can take those from git. The most important file I am requesting here is module.symvers, you can put it on git instead of creating a package for it, that would also be fine.

My workflow here would be grabbing sources, then doing an OOT build with something like make CROSS_COMPILE=.. ARCH=arm -C /path/to/headers .... If you already have headers in a .deb file, I would copy it over and extract it with 7z or dpkg-deb, else I will fetch it from a tag in Linus' tree. If this fails, I will have a look at the exact differences between the RPi kernel tree and upstream Linux.

[mk01]

debian style packaged kernel appeared at wheezy repo only on Apr 06. and still, it's provided by rpi devels, not debian.

and it's not stable, not testing, not devel, it is experimental.

save time and don't try with linus headers. you will miss entire platform. that's the point. it's still not in official tree, that's why wheezy was long months without this and why all is pointing to raspberry team.

believe me, I was surprised as you when I firstly checked apt-cache, i'm surprised now that finally linux-image is available as expected from a linux distribution.

if this would be standard as of launch, why do you think someone would bother with sources? waste own hours doing useless job?

[Lekensteyn]

Okay, there seems to be a misunderstanding here. When I referred to the "packages containing the source as provided by Debian", I wanted to point out that Debian ships sources in their repos too.

You state that the kernel source of the RPi is provided by RPI devels, not Debian (which I understand). The RPi "firmware" also includes the kernel which is independent of the userspace. In that way, it makes sense that these files are not upgraded using apt (rpi-update is supposed to do that job)

Thanks for clarifying things. Let's get back on-topic, do you have the Module.symvers somewhere?

[mk01]

I just want to say that:

1) standard linux-image files, linux-header and relevant what you see at the wheezy repo is "new". according the discussions at rpi forums devels completed and uploaded relevant packages first on april 06. 2) before that wheezy for RPI was without this part. 3) kernel img, modules and relevant files to boot and load wheezy as distro was provided solely by rpi 4) if you don't tell me today ,that indeed is provided by Debian (understand wheezy public repo) I would still tell everybody it's not part of distro - talking strictly about RPI platform - of course linux distro provides everything including the sources and source packages

I hope we are on the same wave now - it's really not relevant to your question, I just wanted to explain why things evolved to current model of distributing rasp and xbian. - and answer your second question how / why you are not following usual workflows within distro.

5) not back to topic, I'm going to update packaging, from next update there will be headers package (including symvers) available. ok ? 6) I don't have access to apt repo, I will send you direct links.

[mk01]

it was not 1 hour, but still

https://github.com/xbianonpi/linux-image

can you look at it if is it sufficient ?

[Lekensteyn]

Unfortunately no, Makefiles are missing:

make[1]: Entering directory `/tmp/linux-image/xbian-package-headers/usr/src/xbian-package-kernel-3.8.12-1+'
make[1]: *** No rule to make target `modules'.  Stop.
make[1]: Leaving directory `/tmp/linux-image/xbian-package-headers/usr/src/xbian-package-kernel-3.8.12-1+'
make: *** [modules] Error 2

Adding Makefile and arch/arm/Makefile is still unsufficient:

make ARCH=arm CROSS_COMPILE=/tmp/x-tools/arm-unknown-linux-gnueabi/bin/arm-unknown-linux-gnueabi- -C /tmp/linux-image/xbian-package-headers/usr/src/xbian-package-kernel-3.8.12-1+ M=/tmp/rtl8188eu  modules
make[1]: Entering directory `/tmp/linux-image/xbian-package-headers/usr/src/xbian-package-kernel-3.8.12-1+'
Makefile:323: /tmp/linux-image/xbian-package-headers/usr/src/xbian-package-kernel-3.8.12-1+/scripts/Kbuild.include: No such file or directory
/bin/bash: /tmp/linux-image/xbian-package-headers/usr/src/xbian-package-kernel-3.8.12-1+/scripts/gcc-goto.sh: No such file or directory

  ERROR: Kernel configuration is invalid.
         include/generated/autoconf.h or include/config/auto.conf are missing.
         Run 'make oldconfig && make prepare' on kernel src to fix it.

make: Entering an unknown directory
make: *** empty variable name.  Stop.
make: Leaving an unknown directory
make[1]: *** [_module_/tmp/rtl8188eu] Error 2
make[1]: Leaving directory `/tmp/linux-image/xbian-package-headers/usr/src/xbian-package-kernel-3.8.12-1+'
make: *** [modules] Error 2

Ultimately I decided to add the RPi kernel tree as remote and checkout the rpi-3.6.y branch, wget the config from https://raw.github.com/xbianonpi/xbian-patches/master/kernel/.config, run yes | make config && make prepare && make scripts (where make is make ARCH=arm CROSS_COMPILE=/tmp/x-tools/arm-unknown-linux-gnueabi/bin/arm-unknown-linux-gnueabi- -j4). Unfortunately, this still gave me a "8188eu: disagrees about version of symbol module_layout" error. This happens when the kernel config/Module.symvers does not match the loaded kernel. It turns out that the kernel you passed to me is a 3.8.12 kernel... which I have only noticed when booting into it :o

So, end of the story, make prepare && make scripts (with rpi-3.8.y branch; .config from /proc/config.gz and Module.symvers as provided by you; and make as make ARCH=... ...), the module can get loaded. This is still not sufficient for the existing 3.6.11 kernel though.

To-do items:

• Find out what files (besides Makefile) has to be copied to the kernel-headers package. Consider looking at scrips/package/builddeb file.

[mk01]

to be honest just follow your hint with headers_install. this are just includes/*.h. you need more.

I will grab source package for linux-headers and go through the debian package scripts

mk

[mk01]

@Lekensteyn

https://github.com/xbianonpi/xbian-package-kernel-headers

currently compiling SPL & ZFS as external modules. can you test & confirm?

[Lekensteyn]

I have tried the new package, but it still fails because make scripts has either not run before or isn't included.

$ make ARCH=arm KSRC=/tmp/xbian-package-kernel-headers/content/usr/src/linux-headers-3.8.13 CROSS_COMPILE=/tmp/x-tools/arm-unknown-linux-gnueabi/bin/arm-unknown-linux-gnueabi-
make ARCH=arm CROSS_COMPILE=/tmp/x-tools/arm-unknown-linux-gnueabi/bin/arm-unknown-linux-gnueabi- -C /tmp/xbian-package-kernel-headers/content/usr/src/linux-headers-3.8.13 M=/tmp/rtl8188eu  modules
make[1]: Entering directory `/tmp/xbian-package-kernel-headers/content/usr/src/linux-headers-3.8.13'
  CC [M]  /tmp/rtl8188eu/core/rtw_cmd.o
/bin/sh: 1: scripts/basic/fixdep: not found
make[2]: *** [/tmp/rtl8188eu/core/rtw_cmd.o] Error 127
make[1]: *** [_module_/tmp/rtl8188eu] Error 2
make[1]: Leaving directory `/tmp/xbian-package-kernel-headers/content/usr/src/linux-headers-3.8.13'
make: *** [modules] Error 2

[mk01]

you are cross compiling ??? (is obvious, but to be honest I didn't pay attention to it last time) what is your BUILD ARCH?

the headers deployed are for native compiling.

fixdep is generated for ARCH arm, is ELF binary and can't be started on your host (/bin/sh: 1: scripts/basic/fixdep: not found). you can check, that scripts/basic/fixdep is indeed included with proper Makefile.

can provide for build arch amd64 but not more

[Lekensteyn]

Yes, compiling on the raspberry pi is too slow. You are not building the kernel on the pi right...?

Anyway, I tried to build on my x86_64 host which failed because of a missing scripts/basic/fixdep (there is a .gitignore script excluding the fixdep binary!). Even after making that file, I ultimately get an error because scripts/mod/modpost cannot be found.

So I decided to try it on the RPI (running Arch Linux) which was very slow as expected. It once again complains of a missing scripts/basic/fixdep program. Therefore I ran make scripts (which did not finish completely) and tried to build the module again. Regardless of the make scripts error, the module building process did actually complete successfully.

arch-linux$ make scripts
  HOSTCC  scripts/basic/fixdep
  HOSTCC  scripts/kconfig/conf.o
  SHIPPED scripts/kconfig/zconf.tab.c
  SHIPPED scripts/kconfig/zconf.lex.c
  SHIPPED scripts/kconfig/zconf.hash.c
  HOSTCC  scripts/kconfig/zconf.tab.o
  HOSTLD  scripts/kconfig/conf
scripts/kconfig/conf --silentoldconfig Kconfig
  HOSTCC  scripts/dtc/checks.o
  HOSTCC  scripts/dtc/data.o
  SHIPPED scripts/dtc/dtc-lexer.lex.c
  SHIPPED scripts/dtc/dtc-parser.tab.h
  HOSTCC  scripts/dtc/dtc-lexer.lex.o
  SHIPPED scripts/dtc/dtc-parser.tab.c
  HOSTCC  scripts/dtc/dtc-parser.tab.o
  HOSTCC  scripts/dtc/dtc.o
  HOSTCC  scripts/dtc/flattree.o
  HOSTCC  scripts/dtc/fstree.o
  HOSTCC  scripts/dtc/livetree.o
  HOSTCC  scripts/dtc/srcpos.o
  HOSTCC  scripts/dtc/treesource.o
  HOSTCC  scripts/dtc/util.o
  HOSTLD  scripts/dtc/dtc
  HOSTCC  scripts/genksyms/genksyms.o
  SHIPPED scripts/genksyms/lex.lex.c
  SHIPPED scripts/genksyms/keywords.hash.c
  SHIPPED scripts/genksyms/parse.tab.h
  HOSTCC  scripts/genksyms/lex.lex.o
  SHIPPED scripts/genksyms/parse.tab.c
  HOSTCC  scripts/genksyms/parse.tab.o
  HOSTLD  scripts/genksyms/genksyms
  CC      scripts/mod/empty.o
  HOSTCC  scripts/mod/mk_elfconfig
  MKELF   scripts/mod/elfconfig.h
  HOSTCC  scripts/mod/file2alias.o
  HOSTCC  scripts/mod/modpost.o
  HOSTCC  scripts/mod/sumversion.o
  HOSTLD  scripts/mod/modpost
  HOSTCC  scripts/kallsyms
  HOSTCC  scripts/conmakehash
  HOSTCC  scripts/bin2c
  HOSTCC  scripts/sortextable
scripts/sortextable.c:31:32: fatal error: tools/be_byteshift.h: No such file or directory
compilation terminated.
make[1]: *** [scripts/sortextable] Error 1
make: *** [scripts] Error 
$ make KSRC=/tmp/xbian-package-kernel-headers/content/usr/src/linux-headers-3.8.13 ARCH=arm
make ARCH=arm CROSS_COMPILE= -C /tmp/xbian-package-kernel-headers/content/usr/src/linux-headers-3.8.13 M=/tmp/rtl8188eu  modules
make[1]: Entering directory `/tmp/xbian-package-kernel-headers/content/usr/src/linux-headers-3.8.13'
  CC [M]  /tmp/rtl8188eu/core/rtw_cmd.o
  CC [M]  /tmp/rtl8188eu/core/rtw_security.o
  CC [M]  /tmp/rtl8188eu/core/rtw_debug.o
  CC [M]  /tmp/rtl8188eu/core/rtw_io.o
  CC [M]  /tmp/rtl8188eu/core/rtw_ioctl_query.o
  CC [M]  /tmp/rtl8188eu/core/rtw_ioctl_set.o
  CC [M]  /tmp/rtl8188eu/core/rtw_ieee80211.o
  CC [M]  /tmp/rtl8188eu/core/rtw_mlme.o
  CC [M]  /tmp/rtl8188eu/core/rtw_mlme_ext.o
  CC [M]  /tmp/rtl8188eu/core/rtw_wlan_util.o
  CC [M]  /tmp/rtl8188eu/core/rtw_pwrctrl.o
  CC [M]  /tmp/rtl8188eu/core/rtw_rf.o
  CC [M]  /tmp/rtl8188eu/core/rtw_recv.o
  CC [M]  /tmp/rtl8188eu/core/rtw_sta_mgt.o
  CC [M]  /tmp/rtl8188eu/core/rtw_ap.o
  CC [M]  /tmp/rtl8188eu/core/rtw_xmit.o
  CC [M]  /tmp/rtl8188eu/core/rtw_p2p.o
  CC [M]  /tmp/rtl8188eu/core/rtw_tdls.o
  CC [M]  /tmp/rtl8188eu/core/rtw_br_ext.o
  CC [M]  /tmp/rtl8188eu/core/rtw_iol.o
  CC [M]  /tmp/rtl8188eu/core/rtw_led.o
  CC [M]  /tmp/rtl8188eu/core/rtw_sreset.o
  CC [M]  /tmp/rtl8188eu/core/rtw_efuse.o
  CC [M]  /tmp/rtl8188eu/hal/hal_intf.o
  CC [M]  /tmp/rtl8188eu/hal/hal_com.o
  CC [M]  /tmp/rtl8188eu/hal/rtl8188e_hal_init.o
  CC [M]  /tmp/rtl8188eu/hal/rtl8188e_phycfg.o
  CC [M]  /tmp/rtl8188eu/hal/rtl8188e_rf6052.o
  CC [M]  /tmp/rtl8188eu/hal/rtl8188e_dm.o
  CC [M]  /tmp/rtl8188eu/hal/rtl8188e_rxdesc.o
  CC [M]  /tmp/rtl8188eu/hal/rtl8188e_cmd.o
  CC [M]  /tmp/rtl8188eu/hal/usb_halinit.o
  CC [M]  /tmp/rtl8188eu/hal/rtl8188eu_led.o
  CC [M]  /tmp/rtl8188eu/hal/rtl8188eu_xmit.o
  CC [M]  /tmp/rtl8188eu/hal/rtl8188eu_recv.o
  CC [M]  /tmp/rtl8188eu/hal/usb_ops_linux.o
  CC [M]  /tmp/rtl8188eu/hal/rtl8188e_mp.o
  CC [M]  /tmp/rtl8188eu/hal/rtl8188e_xmit.o
  CC [M]  /tmp/rtl8188eu/hal/rtl8188e_sreset.o
  CC [M]  /tmp/rtl8188eu/hal/odm_debug.o
  CC [M]  /tmp/rtl8188eu/hal/odm_interface.o
  CC [M]  /tmp/rtl8188eu/hal/odm_HWConfig.o
  CC [M]  /tmp/rtl8188eu/hal/odm.o
  CC [M]  /tmp/rtl8188eu/hal/HalPhyRf.o
  CC [M]  /tmp/rtl8188eu/hal/HalHWImg8188E_MAC.o
  CC [M]  /tmp/rtl8188eu/hal/HalHWImg8188E_BB.o
  CC [M]  /tmp/rtl8188eu/hal/HalHWImg8188E_RF.o
  CC [M]  /tmp/rtl8188eu/hal/Hal8188EFWImg_CE.o
  CC [M]  /tmp/rtl8188eu/hal/HalPhyRf_8188e.o
  CC [M]  /tmp/rtl8188eu/hal/odm_RegConfig8188E.o
  CC [M]  /tmp/rtl8188eu/hal/Hal8188ERateAdaptive.o
  CC [M]  /tmp/rtl8188eu/hal/odm_RTL8188E.o
  CC [M]  /tmp/rtl8188eu/hal/HalPwrSeqCmd.o
  CC [M]  /tmp/rtl8188eu/hal/Hal8188EPwrSeq.o
  CC [M]  /tmp/rtl8188eu/os_dep/osdep_service.o
  CC [M]  /tmp/rtl8188eu/os_dep/os_intfs.o
  CC [M]  /tmp/rtl8188eu/os_dep/usb_intf.o
  CC [M]  /tmp/rtl8188eu/os_dep/usb_ops_linux.o
  CC [M]  /tmp/rtl8188eu/os_dep/ioctl_linux.o
  CC [M]  /tmp/rtl8188eu/os_dep/xmit_linux.o
  CC [M]  /tmp/rtl8188eu/os_dep/mlme_linux.o
  CC [M]  /tmp/rtl8188eu/os_dep/recv_linux.o
  CC [M]  /tmp/rtl8188eu/os_dep/ioctl_cfg80211.o
  CC [M]  /tmp/rtl8188eu/os_dep/rtw_android.o
  CC [M]  /tmp/rtl8188eu/core/rtw_mp.o
  CC [M]  /tmp/rtl8188eu/core/rtw_mp_ioctl.o
  LD [M]  /tmp/rtl8188eu/8188eu.o
  Building modules, stage 2.
  MODPOST 1 modules
  CC      /tmp/rtl8188eu/8188eu.mod.o
  LD [M]  /tmp/rtl8188eu/8188eu.ko
make[1]: Leaving directory `/tmp/xbian-package-kernel-headers/content/usr/src/linux-headers-3.8.13'

Anyway, I think I will be fine with the Module.symvers and .config file. As I will be cross-compiling anyway and have the sources already, I do not really need this package to get fixed for x86_64.

[mk01]

inspired by your last post:

1) .gtignore file is ok 2) run "make headers_check; make headers_install ; make scripts". it will correctly regenerate platform dependent scripts and binaries under scripts directory 3) if you want to cross_compile, do this on your BUILD machine 4) if compile on RPI, just run the same command on RPI

just as info, the sortextable looks like a platform x86 specific. when natively prepared headers, file is missing.

[mk01]

as addition to xbian-package-kernel there is now xbian-package-kernel-headers. this provides the dev environment.