search

ARM-software / lisa

Linux Integrated System Analysis
Apache License 2.0
200 stars 119 forks source link

kmodules building failed #2174

Closed Jiaming97 closed 8 months ago

Jiaming97 commented 8 months ago

Describe the bug I met with the bug when I tried to build kmodules using alpine.

To Reproduce The version of the LISA is v3.1.0.

I have followed the LISA guide to create the target_conf.yml. Enabling a module Here is my target_conf.yml. The environment of the target is buildroot. 

target-conf:
  kind: linux
  name: tc
  host: localhost
  port: 8023
  username: root
  password: ""
  strict-host-check: false

  kernel:
    src: /home/jiaguo01/workzone/TC2/buildroot/output/buildroot/tmp_build/linux

    modules:
      make-variables:
        CC: clang
        LLVM: -16
      build-env: alpine

  wait-boot:
    enable: false

  devlib:
    file-xfer: scp
    max-async: 1

When I tried to build and load the kmodules. The error log is as follows.

[LISAShell lisa] \> lisa-load-kmod --conf target_conf.yml -- echo hello world
[2024-01-17 16:10:05,597][lisa.target.Target] INFO  Target configuration:
+- devlib:
    |- excluded-modules from target_conf.yml (list): []
    |- file-xfer from target_conf.yml (str): scp
    |- max-async from target_conf.yml (int): 1
    +- platform:
        |- class from target_conf.yml (str): devlib.platform.Platform
|- host from target_conf.yml (str): localhost
+- kernel:
    +- modules:
        |- build-env from target_conf.yml (str): alpine
        |- make-variables from target_conf.yml (dict): {'CC': 'clang', 'LLVM': -16}
    |- src from target_conf.yml (str): /home/jiaguo01/workzone/TC2/buildroot/output/buildroot/tmp_build/linux
|- kind from target_conf.yml (str): linux
|- lazy-platinfo from target_conf.yml (bool): False
|- name from target_conf.yml (str): tc
|- password from target_conf.yml (str): <password>
|- port from target_conf.yml (int): 8023
|- strict-host-check from target_conf.yml (bool): False
|- tools from target_conf.yml (list): []
|- username from target_conf.yml (str): root
+- wait-boot:
    |- enable from target_conf.yml (bool): False
    |- timeout from target_conf.yml (int): 10
[2024-01-17 16:10:05,598][lisa.target.Target] INFO  Creating result directory: /workzone/jiaguo01/LISA/lisa/results/Target-tc-20240117_161005.598085
[2024-01-17 16:10:28,367][lisa.target.Target] INFO  Connected to target tc
[2024-01-17 16:10:29,292][lisa.energy_model.EnergyModel.from_target] INFO  Attempting to load EM using LinuxEnergyModel
[2024-01-17 16:10:37,811][lisa.target.Target] INFO  Loading target devlib module cpuidle
[2024-01-17 16:10:47,174][lisa.target.Target] INFO  Loading target devlib module cpufreq
[2024-01-17 16:10:53,988][lisa.target.Target] INFO  Loading target devlib module sched
[2024-01-17 16:11:00,920][sched] INFO  Scheduler sched_domain procfs entries found
[2024-01-17 16:11:00,920][sched] INFO  Detected kernel compiled with SCHED_DEBUG=y
[2024-01-17 16:11:00,920][sched] INFO  CPU capacity sysfs entries found
[2024-01-17 16:11:11,402][lisa.target.Target] INFO  Effective platform information:
|- abi from target (str): arm64
+- cpu-capacities:
    |- orig from target (dict): {0: 286, 1: 286, 2: 286, 3: 286, 4: 793, 5: 793, 6: 793, 7: 1024}
    |- writeable from target (bool): False
    |- rtapp from target(platform-info/rtapp/calib),target(platform-info/cpu-capacities/orig),target(platform-info/cpu-capacities/writeable) (str): <depends on lazy keys: platform-info/rtapp/calib>
|- cpus-count from target (int): 8
|- freq-domains from target (list): [[0, 1, 2, 3], [4, 5, 6], [7]]
|- freqs from target (dict): {0: [768000, 1153000, 1537000, 1844000, 2152000], 1: [768000, 1153000, 1537000, 1844000, 2152000], 2: [768000, 1153000, 1537000, 1844000, 2152000], 3: [768000, 1153000, 1537000, 1844000, 2152000], 4: [946000, 1419000, 1893000, 2271000, 2650000], 5: [946000, 1419000, 1893000, 2271000, 2650000], 6: [946000, 1419000, 1893000, 2271000, 2650000], 7: [1088000, 1632000, 2176000, 2612000, 3047000]}
+- kernel:
    |- config from target (TypedKernelConfig): <kernel config>
    |- symbols-address from target (DeferredValue): <symbols address>
    |- version from target (KernelVersion): 5.15.41-g7385306ee901-dirty 1 SMP PREEMPT Wed Jan 17 14:26:11 CST 2024
|- name from target-conf (str): tc
|- nrg-model from target (LinuxEnergyModel): <lisa.energy_model.LinuxEnergyModel object at 0x7fdc90b6e370>
|- numa-nodes-count from target (int): 1
|- os from target (str): linux
+- rtapp:
    |- calib from target (DeferredValue): <lazy value of RTA.get_cpu_calibrations>
|- capacity-classes from target(platform-info/cpu-capacities/orig) (list): [[0, 1, 2, 3], [4, 5, 6], [7]]
[2024-01-17 16:11:11,402][lisa._kmod.KernelTree] INFO  Detected CROSS_COMPILE=aarch64-linux-gnu- and CC=clang
[2024-01-17 16:11:55,882][lisa._kmod.KernelTree] INFO  Detected CROSS_COMPILE=aarch64-linux-gnu- and CC=clang
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
[2024-01-17 16:11:55,907][lisa._kmod.alpine_chroot] INFO  Setting up Alpine chroot from https://dl-cdn.alpinelinux.org/alpine/v3.17/releases/x86_64/alpine-minirootfs-3.17.3-x86_64.tar.gz -> /workzone/jiaguo01/LISA/lisa/cache/git-5315fabab791fa313e9007c31c3777c7b7d2dce8-dirty-d03c28ea4c554840c813073a22a39c1020119305/alpine_chroot/tmp4m8o48ww/tmphqssvyyp
[2024-01-17 16:12:08,154][lisa._kmod.KernelTree] INFO  Preparing kernel tree for modules
[2024-01-17 16:12:08,218][lisa._kmod.KernelTree] INFO  Detected CROSS_COMPILE=aarch64-linux-gnu- and CC=clang
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
[2024-01-17 16:12:08,250][lisa._kmod.KernelTree] INFO  Preparing kernel tree for modules
[2024-01-17 16:12:08,302][lisa._kmod.KernelTree] INFO  Detected CROSS_COMPILE=aarch64-linux-gnu- and CC=clang
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
[2024-01-17 16:12:08,340][lisa._kmod.KernelTree] INFO  Preparing kernel tree for modules
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/workzone/jiaguo01/LISA/lisa/lisa/_kmod.py", line 1866, in _do_compile
    key = get_key(kernel_tree)
  File "/workzone/jiaguo01/LISA/lisa/lisa/_kmod.py", line 1828, in get_key
    raise ValueError('Kernel tree has no checksum')
ValueError: Kernel tree has no checksum

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/workzone/jiaguo01/LISA/lisa/lisa/_unshare.py", line 110, in _unshare_wrapper
    return f(*args, **kwargs)
  File "/workzone/jiaguo01/LISA/lisa/lisa/_kmod.py", line 1868, in _do_compile
    with kernel_tree as kernel_tree:
  File "/workzone/jiaguo01/LISA/lisa/lisa/_kmod.py", line 714, in __enter__
    spec = cm.__enter__()
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/workzone/jiaguo01/LISA/lisa/lisa/_kmod.py", line 1305, in try_loaders
    raise ValueError(f'Could not load kernel trees:\n{excep_str}')
ValueError: Could not load kernel trees:
from_installed_headers: ValueError: Building from /lib/modules is not supported with the Alpine build environment as /lib/modules might not be self contained (i.e. symlinks pointing outside)
from_sysfs_headers: CalledProcessError: Command '['chroot', PosixPath('/tmp/tmpla06phsw/overlaid'), 'sh', '-c', 'source /etc/profile && exec make -j36 -C /tmp/tmpxup6g0gx/overlaid -- ARCH=arm64 CC=clang CROSS_COMPILE=aarch64-linux-gnu- KBUILD_MODPOST_WARN=1 LLVM=1 mrproper']' returned non-zero exit status 2.:
make: Entering directory '/tmp/tmpxup6g0gx/overlaid'
Makefile:2: /workzone/jiaguo01/TC2/buildroot/src/linux/Makefile: No such file or directory
make: *** No rule to make target '/workzone/jiaguo01/TC2/buildroot/src/linux/Makefile'.  Stop.
make: Leaving directory '/tmp/tmpxup6g0gx/overlaid'

from_proc_config: CalledProcessError: Command '['chroot', PosixPath('/tmp/tmpz32klin8/overlaid'), 'sh', '-c', 'source /etc/profile && exec make -j36 -C /tmp/tmpf2gx9rhu/overlaid -- ARCH=arm64 CC=clang CROSS_COMPILE=aarch64-linux-gnu- KBUILD_MODPOST_WARN=1 LLVM=1 mrproper']' returned non-zero exit status 2.:
make: Entering directory '/tmp/tmpf2gx9rhu/overlaid'
Makefile:2: /workzone/jiaguo01/TC2/buildroot/src/linux/Makefile: No such file or directory
make: *** No rule to make target '/workzone/jiaguo01/TC2/buildroot/src/linux/Makefile'.  Stop.
make: Leaving directory '/tmp/tmpf2gx9rhu/overlaid'

from_user_tree: CalledProcessError: Command '['chroot', PosixPath('/tmp/tmpadj20lo8/overlaid'), 'sh', '-c', 'source /etc/profile && exec make -j36 -C /tmp/tmpse8x466y/overlaid -- ARCH=arm64 CC=clang CROSS_COMPILE=aarch64-linux-gnu- KBUILD_MODPOST_WARN=1 LLVM=1 mrproper']' returned non-zero exit status 2.:
make: Entering directory '/tmp/tmpse8x466y/overlaid'
Makefile:2: /workzone/jiaguo01/TC2/buildroot/src/linux/Makefile: No such file or directory
make: *** No rule to make target '/workzone/jiaguo01/TC2/buildroot/src/linux/Makefile'.  Stop.
make: Leaving directory '/tmp/tmpse8x466y/overlaid'

"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/workzone/jiaguo01/LISA/lisa/.lisa-venv-3.8/bin/lisa-load-kmod", line 33, in <module>
    sys.exit(load_entry_point('lisa-linux', 'console_scripts', 'lisa-load-kmod')())
  File "/workzone/jiaguo01/LISA/lisa/lisa/_cli_tools/lisa_load_kmod.py", line 111, in main
    with dmesg_cm(), kmod_cm:
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/workzone/jiaguo01/LISA/lisa/lisa/_cli_tools/lisa_load_kmod.py", line 69, in cm
    with _kmod_cm:
  File "/workzone/jiaguo01/LISA/lisa/lisa/utils.py", line 2840, in __enter__
    return cm.__enter__()
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/workzone/jiaguo01/LISA/lisa/lisa/utils.py", line 2806, in _wrap_gen
    res = gen.send(None)
  File "/workzone/jiaguo01/LISA/lisa/lisa/_kmod.py", line 1946, in run
    x = self.install(**kwargs)
  File "/workzone/jiaguo01/LISA/lisa/lisa/_kmod.py", line 1966, in install
    return super().install(*args, **kwargs)
  File "/workzone/jiaguo01/LISA/lisa/lisa/_kmod.py", line 1906, in install
    content = self._compile()
  File "/workzone/jiaguo01/LISA/lisa/lisa/utils.py", line 586, in wrapper
    raise e
  File "/workzone/jiaguo01/LISA/lisa/lisa/utils.py", line 572, in catch
    x = f(*args, **kwargs)
  File "/workzone/jiaguo01/LISA/lisa/lisa/_kmod.py", line 1812, in _compile
    bin_, spec = compile_(self)
  File "/workzone/jiaguo01/LISA/lisa/lisa/_unshare.py", line 223, in wrapper
    return _with_unshare(f, args=args, kwargs=kwargs)
  File "/workzone/jiaguo01/LISA/lisa/lisa/_unshare.py", line 194, in _with_unshare
    return pool.apply(
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 357, in apply
    return self.apply_async(func, args, kwds).get()
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
ValueError: Could not load kernel trees:
**from_installed_headers: ValueError: Building from /lib/modules is not supported with the Alpine build environment as /lib/modules might not be self contained (i.e. symlinks pointing outside)**
from_sysfs_headers: CalledProcessError: Command '['chroot', PosixPath('/tmp/tmpla06phsw/overlaid'), 'sh', '-c', 'source /etc/profile && exec make -j36 -C /tmp/tmpxup6g0gx/overlaid -- ARCH=arm64 CC=clang CROSS_COMPILE=aarch64-linux-gnu- KBUILD_MODPOST_WARN=1 LLVM=1 mrproper']' returned non-zero exit status 2.:
make: Entering directory '/tmp/tmpxup6g0gx/overlaid'
Makefile:2: /workzone/jiaguo01/TC2/buildroot/src/linux/Makefile: No such file or directory
**make: *** No rule to make target '/workzone/jiaguo01/TC2/buildroot/src/linux/Makefile'.  Stop.**
make: Leaving directory '/tmp/tmpxup6g0gx/overlaid'

from_proc_config: CalledProcessError: Command '['chroot', PosixPath('/tmp/tmpz32klin8/overlaid'), 'sh', '-c', 'source /etc/profile && exec make -j36 -C /tmp/tmpf2gx9rhu/overlaid -- ARCH=arm64 CC=clang CROSS_COMPILE=aarch64-linux-gnu- KBUILD_MODPOST_WARN=1 LLVM=1 mrproper']' returned non-zero exit status 2.:
make: Entering directory '/tmp/tmpf2gx9rhu/overlaid'
Makefile:2: /workzone/jiaguo01/TC2/buildroot/src/linux/Makefile: No such file or directory
make: *** No rule to make target '/workzone/jiaguo01/TC2/buildroot/src/linux/Makefile'.  Stop.
make: Leaving directory '/tmp/tmpf2gx9rhu/overlaid'

from_user_tree: CalledProcessError: Command '['chroot', PosixPath('/tmp/tmpadj20lo8/overlaid'), 'sh', '-c', 'source /etc/profile && exec make -j36 -C /tmp/tmpse8x466y/overlaid -- ARCH=arm64 CC=clang CROSS_COMPILE=aarch64-linux-gnu- KBUILD_MODPOST_WARN=1 LLVM=1 mrproper']' returned non-zero exit status 2.:
make: Entering directory '/tmp/tmpse8x466y/overlaid'
Makefile:2: /workzone/jiaguo01/TC2/buildroot/src/linux/Makefile: No such file or directory
make: *** No rule to make target '/workzone/jiaguo01/TC2/buildroot/src/linux/Makefile'.  Stop.
make: Leaving directory '/tmp/tmpse8x466y/overlaid'

I have checked the path, the file /workzone/jiaguo01/TC2/buildroot/src/linux/Makefile exists.

Acutally, when I tried to build without the following settings in the target_conf.yml. The modules can be built successfully.

    modules:
      make-variables:
        CC: clang
        LLVM: -16
      build-env: alpine

So, I think the problem may due to the alpine. Could you please check with it? Thanks!

douglas-raillard-arm commented 8 months ago

Hi @Jiaming97 , are you using LISA from PyPI or installed from the git repo ? We had a large number of changes since the last PyPI release, so you should try that first.

Then if you still have issues, we will need the logging at debug level to see the output of the build command (LISA is using the logging module from the std lib, if you use lisa.utils.setup_logging() you can pass level=logging.DEBUG to get that).

douglas-raillard-arm commented 8 months ago

Also if it was working for you before and has broken on very recent kernels, you might be affected by what this PR is fixing (type change of util_est member of some structs): https://github.com/ARM-software/lisa/pull/2161

EDIT: looking again at your error it's pretty unlikely, I'll need the debug log to know what's happening

douglas-raillard-arm commented 8 months ago

I can see a discrepancy between the config:

    src: /home/jiaguo01/workzone/TC2/buildroot/output/buildroot/tmp_build/linux

and the error:

Makefile:2: /workzone/jiaguo01/TC2/buildroot/src/linux/Makefile: No such file or directory

Are you using symlinks anywhere ? When using the alpine build env, LISA creates a separate mount namespace (to mount an overlayfs without actually being root) and enters a chroot. The only thing available inside that chroot is the path that was specified as src in the conf (using a bind mount). If that folder contains symlinks pointing outside of it, things will break as the symlink targets will not be valid inside that chroot.

If that is a symlink issue, I can't think of an easy fix beyond removing symlinks, or just not using Alpine build env.

Jiaming97 commented 8 months ago

Hi @douglas-raillard-arm ,

Thank you for your help. The issue may not be the symlink. I think the problem may be the configurations in target_conf.yml .

Since it is the kernel module building, I thought the prebuilt kernel should be provided in the kernel/src/. So I use the following configuration.

  kernel:
    src: /home/jiaguo01/workzone/TC2/buildroot/output/buildroot/tmp_build/linux

The path is the output of building the linux kernel, which contains the configuration and header files used in the build. It failed. It seems that when I change this to the source code. It can build successfully.

douglas-raillard-arm commented 8 months ago

Yes, kernel/src needs to point at the source tree: https://lisa-linux-integrated-system-analysis.readthedocs.io/en/latest/target.html#lisa.target.TargetConf

As a side note, you can probably drop this from your conf btw:

      make-variables:
        CC: clang
        LLVM: -16

The current code will detect it needs clang and the version from the live target (/proc/config.gz). In alpine build env it will find the package for clang-16 on its own, if you use host env you will need to have clang-16 available. Otherwise it will fallback to the closest version it finds, with a priority to higher versions.

douglas-raillard-arm commented 8 months ago

Closing as the issue seems resolved