search

longsleep / build-pine64-image

Pine64 Linux build scripts, tools and instructions
MIT License
235 stars 126 forks source link

Is there an easy way to install kernel modules? #42

Closed lampi87 closed 7 years ago

lampi87 commented 7 years ago

Hi!

Is there an easy way to install kernel modules beside clone and compile of whole kernel?

Thx, Alexander

longsleep commented 7 years ago

Sure, Kernel headers are provided in my releases and thus any out-of-tree module compilation does work.

lampi87 commented 7 years ago

But my Ubuntu claims, that classmap.h is missing for scripts/selinux/mdp/mdp.c when I try to build modules in /usr/src/linux-headers... Based on my research this must be already included in source code?!

longsleep commented 7 years ago

Well not every Kernel module does compile out of tree / with Kernel headers only.

renoirb commented 7 years ago

Sorry to re-open this one.

If you were to make possible to add kernel modules and/or allow compiling from a running kernel (note: /usr/local/sbin/pine64_update_uboot.sh; /usr/local/sbin/pine64_update_kernel.sh; reboot still works).

What would you do? I've compiled kernel, and configured modules back in 2007 and it's far.

Would you mind sharing a few notes on how you would do it, and I'll see if I can help.

renoirb commented 7 years ago

My use-case is rather simple. I'm setting up Kubernetes and I miss modules.

I'd like to get all green lights from lxc-checkconfig.

Most likely modules required for Kubernetes are merged AFTER the point from your fork (at 3.10)

Here are a few pointers on how I added modules.

I expect i'll have to get back into digging into kernel configuration/compilation. A few pointers will be of great help!

Use-case notes:

  1. List modules

    find /lib/modules/$(uname -r) -type f -name \*.ko > ~/lib-modules.txt

  2. Load one module and check it out

    modprobe overlayfs
modinfo overlayfs
filename:       /lib/modules/3.10.105-0-pine64-longsleep/kernel/fs/overlayfs/overlayfs.ko
alias:          fs-overlayfs
license:        GPL
description:    Overlay filesystem
author:         Miklos Szeredi <miklos@szeredi.hu>
srcversion:     37BB89DFFC4ADEA3F38CFC7
depends:
intree:         Y
vermagic:       3.10.105-0-pine64-longsleep SMP preempt mod_unload modversions aarch64

  3. Add modules (dynamically) at startup

    cat /etc/modules-load.d/example.conf
ebt_vlan
overlayfs
nf_nat_ipv4
nf_nat_ipv6
xt_nat
nf_nat

  4. Add modules missing ... ?

    ???

longsleep commented 7 years ago

All green for me:

root@pi3:~# uname -a
Linux pi3 3.10.105-0-pine64-longsleep #3 SMP PREEMPT Sat Mar 11 16:05:53 CET 2017 aarch64 aarch64 aarch64 GNU/Linux

root@pi3:~# sh lxc-checkconfig.sh 
--- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
Network namespace: enabled
Multiple /dev/pts instances: enabled

--- Control groups ---
Cgroup: enabled
Cgroup clone_children flag: enabled
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled

--- Misc ---
Veth pair device: enabled
Macvlan: enabled
Vlan: enabled
File capabilities: enabled

Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config lxc-checkconfig.sh
renoirb commented 7 years ago

Oh, that's a different run output.

Which filesystem configuration you have with Docker? aufs or overlay ? Thing is that there's 3 versions of "overlay". (more below)

I’ve mistakenly given you a different script version. The original I used was from this article from Hypriot

uname -a
Linux node4 3.10.105-0-pine64-longsleep #3 SMP PREEMPT Sat Mar 11 16:05:53 CET 2017 aarch64 aarch64 aarch64 GNU/Linux

wget https://github.com/docker/docker/raw/master/contrib/check-config.sh
sudo bash check-config.sh
info: reading kernel config from /proc/config.gz ...

Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
- apparmor: enabled and tools installed
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled
- CONFIG_BRIDGE: enabled (as module)
- CONFIG_BRIDGE_NETFILTER: enabled
- CONFIG_NF_NAT_IPV4: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONFIG_IP_NF_NAT: missing
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_NF_NAT_NEEDED: enabled
- CONFIG_POSIX_MQUEUE: enabled
- CONFIG_DEVPTS_MULTIPLE_INSTANCES: enabled

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_CGROUP_PIDS: missing
- CONFIG_MEMCG_SWAP: enabled
- CONFIG_MEMCG_SWAP_ENABLED: enabled
    (cgroup swap accounting is currently enabled)
- CONFIG_MEMCG_KMEM: enabled
- CONFIG_RESOURCE_COUNTERS: enabled
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_IOSCHED_CFQ: enabled
- CONFIG_CFQ_GROUP_IOSCHED: enabled
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: enabled
- CONFIG_NET_CLS_CGROUP: enabled (as module)
- CONFIG_NETPRIO_CGROUP: enabled (as module)
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: enabled
- CONFIG_IP_VS: enabled (as module)
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_RR: enabled (as module)
- CONFIG_EXT3_FS: missing
- CONFIG_EXT3_FS_XATTR: missing
- CONFIG_EXT3_FS_POSIX_ACL: missing
- CONFIG_EXT3_FS_SECURITY: missing
    (enable these ext3 configs if you are using ext3 as backing filesystem)
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: enabled (as module)
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled
      - CONFIG_CRYPTO_GCM: enabled (as module)
      - CONFIG_CRYPTO_SEQIV: enabled (as module)
      - CONFIG_CRYPTO_GHASH: enabled (as module)
      - CONFIG_XFRM: enabled
      - CONFIG_XFRM_USER: enabled
      - CONFIG_XFRM_ALGO: enabled
      - CONFIG_INET_ESP: missing
      - CONFIG_INET_XFRM_MODE_TRANSPORT: enabled
  - "ipvlan":
    - CONFIG_IPVLAN: missing
  - "macvlan":
    - CONFIG_MACVLAN: enabled
    - CONFIG_DUMMY: enabled
  - "ftp,tftp client in container":
    - CONFIG_NF_NAT_FTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_FTP: enabled (as module)
    - CONFIG_NF_NAT_TFTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_TFTP: enabled (as module)
- Storage Drivers:
  - "aufs":
    - CONFIG_AUFS_FS: enabled (as module)
  - "btrfs":
    - CONFIG_BTRFS_FS: enabled (as module)
    - CONFIG_BTRFS_FS_POSIX_ACL: enabled
  - "devicemapper":
    - CONFIG_BLK_DEV_DM: enabled (as module)
    - CONFIG_DM_THIN_PROVISIONING: enabled (as module)
  - "overlay":
    - CONFIG_OVERLAY_FS: missing
  - "zfs":
    - /dev/zfs: missing
    - zfs command: missing
    - zpool command: missing

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

The above is the result after:

And I'm kinda blocked, let's say I want to test with zfs and other missing modules above I can't use. Only solution is to compile kernel.

Context I'm taking the following use-case as an excuse to find and script something to re-compile, and package a kernel with different options.

So, I'm using Docker overlay kernel module as something we may want to add to a kernel build.

But I can't make the kernel load successfully (any) overlay kernel module on the node.

Unless I misinterpreted the info in this issue, we may have a different kernel module version in your fork.

I've compared Linus’s at 4.x, yours at 3.10.105), versions and modules are different —obvious, I know :/

Which makes me wonder

Maybe we could add a script to build the kernel from within the board (not just update it).

I thought of looking at how you build a kernel, and add modules.

That's where I'm lost, I'm not sure where to start.

Current state

Today here's what I have

Versions

kubectl version
Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1", GitCommit:"b0b7a323cc5a4a2019b2e9520c21c7830b7f708e", GitTreeState:"clean", BuildDate:"2017-04-03T20:44:38Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/arm64"}

docker -v
Docker version 1.12.6, build 78d1802

docker info
Containers: 4
 Running: 2
 Paused: 0
 Stopped: 2
Images: 6
Server Version: 1.12.6
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 22
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: overlay bridge null host
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor seccomp
Kernel Version: 3.10.105-0-pine64-longsleep
Operating System: Ubuntu 16.04.2 LTS
OSType: linux
Architecture: aarch64
CPUs: 4
Total Memory: 1.942 GiB
Name: node4
ID: ...
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Insecure Registries: ...
longsleep commented 7 years ago

Well we have Kernel 3.10 and most of the stuff missing is not available. As backporting is a lot of work and a waste of time i suggest you either look into mainline Kernel or fix your use case to work with the Docker support possible with Kernel 3.10.

gkkpch commented 7 years ago

correct, Docker needs an overlayfs version from => 3.18 (think is was v22). I know Hardkernel backported it for kernel 3.14

gkkpch commented 7 years ago

Not seen anything older than that...

umiddelb commented 7 years ago

If you don't rely on HDMI support, audio playback, etc., you may try z2d. It will create a µSD card with mainline U-Boot and 4.9.22 mainline kernel (based on this kernel source tree). The hardware support is OK for headless usage (and of course for Docker and K8s), you can even use the lower USB receptacle.

renoirb commented 7 years ago

A bit of a self-promoting plug, @umiddelb, still an appropriate intervention :) Your solution seems a nice way to help build using a more recent kernel.

I should give it another shot, although I missed the nice CPU heat check from @longsleep.

I'll keep you posted. Maybe there's something we can do to have best of both projects (longsleep's build-pine64-image, and yours) AND the ability to tweak kernel modules we want to get.

umiddelb commented 7 years ago

Maybe I should have cited

As backporting is a lot of work and a waste of time i suggest you either look into mainline Kernel ...

at the beginning. Simon(longsleep)'s work simply addresses a different audience. If you want to use your Pine64 non-headless with a desktop GUI you have to take the BSP u-boot and BSP kernel. There is no other option.

On the other hand, if you prefer to run a mainline kernel, longsleeps work won't help you (it never did, it never intended to do so). People like apritzel and Icenowy have been working on this issue, but they don't publish ready to use images.

renoirb commented 7 years ago

Got it @umiddelb. And keep up the good work, your scripts works well, i'm already at my second node now

pfeerick commented 7 years ago

@umiddelb Quick question if I may... is there any major advantage/difference between using z2d and a nightly build of Armbian as that is on 4.10 now. Is it better suited for docker use?

renoirb commented 7 years ago

In relation to Pine64. I am seeing one difference @pfeerick just now about kernel modules if you want to use Mali and sunxi specific modules described in pine64.org CPU/GPU Architecture features. You can't probe the temperature and such that's part of what's called BSP. See also linux-sunxi Linux Maintaining effort. Other things may raise up.

umiddelb commented 7 years ago

Quick question if I may... is there any major advantage/difference between using z2d and a nightly build of Armbian as that is on 4.10 now. Is it better suited for docker use?

I'm using apritzel's 4.9 kernel (subsequently patched to 4.9.22) which has been used in Armbian 5.24. Later on the Armbian people people decided to abandon the regular mainline release and switched to Icenowy's kernel for the nightly builds. This kernel is still work in progress (e.g. boots up with kernel oops).

z2d installs plain Ubuntu/Debian/CentOS, you won't find any of the enhancements shipped with Armbian.

You can of course build and install the 4.9 kernel on your Pine64 running Armbian.

From the Docker perspective both should run well since Igor pays very much attention shipping Docker-ready kernels.

pfeerick commented 7 years ago

Many thanks @renoirb and @umiddelb for the quick and informative replies ;) No enhancements are fine... less deciphering how things have been changed from the distro norms :) Sounds like I'll have to give x2d a try... don't like the sound of those kernel oopses! I'd like my mainline pine64 mostly stable ;)

renoirb commented 7 years ago

After digging about Linux Kernel installation from both @longsleep and and @umiddleb. I noticed, in both case, that the kernel is downloaded as a compressed file amd extracted.

Is there a technical reason of why it is not a deb/rpm? I guess it's a matter if being easy to use the same code on many distributions (CentOs, Fedora, Debian, Ubuntu, etc). But yet, there is at least one tool to help package a deb/rpm (Jordan Sissel's FPM), and an HTTP directory with a metadata file can work for all.

Any other reason for not being a package? (Curiosity)

longsleep commented 7 years ago

@renoirb - packaging is work and packaging the kernel and boot loader is not completely trivial. I was not willing to spend time on this.

And yes, I also wanted my kernel and boot loader binary builds to be easily usable independent from distributions.

Also a distribution package is nothing more than a compressed archive following certain distro specific rules and dependencies.

renoirb commented 7 years ago

Fair enough @longeep. I thought there could be other motives, but that one is a good one enough, and is indeed not a trivial thing to do for all platforms, and done simply. Nothing's simpler than a compressed archive.

Anyway, the way it's done doesn't block ones who wants to build kernel.

What about yout, @umiddelb, your work looks more Devoper oriented. Would you like help on on board kernel compilation and packaging?

umiddelb commented 7 years ago

What about yout, @umiddelb, your work looks more Devoper oriented. Would you like help on on board kernel compilation and packaging?

Indeed the kernel for the Pine64 has been built on the Pine64 (and some other Aarch64 boxes using distcc), you can find some further help compiling your own kernel here

Concerning you request for packaging, the kernel build system itself contains some make targets for deb and rpm packages, e.g. make rpm-pkg (see make help for further details).