This issue is going to be used to keep a track of kernel module headers or source releases. A few changes happened over time that need to be consolidated a bit before investing further into refining this area.

Rough Timeline/braindump:

pre-thud: We would ship only kernel_modules_headers.tar.gz This was generated using https://github.com/balena-os/module-headers

Some issues with this script is:

This is our own one and we have to keep fixing issues by importing them from upstream. e.g. #1302 was painful to find/fix.
This script compiles the kernel tools and ship those as binaries (binaries run on target device. not for cross compiling kernel modules)

around pre-thud:

We wanted to switch to shipping source. And decided to ship full kernel source as kernel_source.tar.gz.
Based on kernel-devsrc.bb recipe upstream https://github.com/balena-os/meta-balena/pull/1328

thud/post-thud:

Upstream kernel-devsrc changed from shipping full kernel source to shipping just kernel headers as source.
so kernel_source.tar.gz was full kernel source for some devices. And suddenly became kernel module headers source for some devices. Depending on the yocto version from bsp

now:

A full kernel source is larger(~100megs) than just headers(~10megs).
Size is important for builds especially on-device.
We ditched the full kernel source in favour of upstream kernel-devsrc behaviour. Pulled in kernel-devsrc.bb from zeus. https://github.com/balena-os/meta-balena/pull/1605/commits/4df407b394e2210f59a54f90874ccfa68909e7f0
Added automated Dockerfile hello-world testing during build time to catch unnecessary niggles during build. https://github.com/balena-os/meta-balena/pull/1605
Pulled in the external gen_mod_headers script into meta-balena as well. Having to bump an external repo was annoying. https://github.com/balena-os/meta-balena/commit/55733f7af683d5fa7a1fb2cb545283e21c435383#diff-54195e40cffeed7a145dced739200108
We tried to follow up with upstream about this breaking change between pre-thud and post-thud for the kernel-devsrc behaviour. I with we had caught this sooner and raised a bit of noise to prevent breakage. https://www.yoctoproject.org/pipermail/yocto/2019-May/045371.html

Next steps/ideas: At the moment we have 2 artefacts.

kernel_modules_headers.tar.gz. Generated using gen_mod_headers. The tools in this tarball are prebuilt binaries for target device.
kernel_source.tar.gz: Generated using kernel-devsrc upstream. This was full kernel source for some releases. But is a minimal set of headers for module compilation without any pre-built binaries now.

Todo:

use upstream kernel-devsrc to generate kernel_modules_headers if possible so that we can at least inherit upstream headers work and do away with gen_mod_headers
If possible, somehow patch kernel_source.tar.gz with a Makefile hack so that we can seamlessly replace the prebuilt headers with the source version (in the same url to prevent application breakage).
- If not possible, make a breaking change at balenaOS 3 or by informing all users using the modules and make the switch to kernel_source. They only need to add a make modules_prepare before compiling the kernel modules
There is upstream work to recreate the functionality of the old kernel-devsrc recipe which packaged the full kernel source.
- https://bugzilla.yoctoproject.org/show_bug.cgi?id=13360
- Contribute upstream/pull in recipe
- Ship full kernel source as well in preparation for allowing customers to tweak kernel configuration options for the OS.

Other notes: We provide this example project to customer as a reference to when needing to build kernel modules https://github.com/balenalabs/kernel-module-build

This project tries to build kernel modules for multiple os versions to allow the whole fleet.
This project could use some tlc.

Other issues that are similar being consolidated here:

balena-os / meta-balena

Kernel Module Headers or full Kernel Source Tracker #1822

1503

1519