Attempting to cross-compile linux kernel, getting stuck on inline assembly `.incbin` file not found error

[EvanKrall]

I'm working on a Raspberry Pi project, and I often need to recompile the kernel for it. I've been cross-compiling from my (fairly old; i7-6500U) intel box, so llama seems like it should save me a bunch of time.

I've got this sorta working. I've got a directory _toolchain with two scripts in it:

_toolchain/activate (which you source like you would to activate a python virtualenv):

_OLDPATH="${_OLDPATH:-$PATH}"
_OLDPS1="${_OLDPS1:-$PS1}"
export PATH="$(dirname $(readlink -f "${BASH_SOURCE[0]}")):$HOME/go/bin:$PATH"
export ARCH=arm64
export CROSS_COMPILE=aarch64-linux-gnu-
export LLAMACC_FUNCTION=gcc-aarch64-linux-gnu
export LLAMACC_LOCAL_CC=/usr/bin/aarch64-linux-gnu-gcc
export LLAMACC_LOCAL_CXX=/usr/bin/aarch64-linux-gnu-g++
export KBUILD_BUILD_TIMESTAMP='ccache'  # probably not relevant to llama

PS1="${_OLDPS1}\[\033[0m\](llamacc arm64) "

function deactivate() {
    export PATH="$_OLDPATH"
    PS1="$_OLDPS1"
    unset ARCH
    unset CROSS_COMPILE
    unset KBUILD_BUILD_TIMESTAMP
}
# ... and some other functions helpful for my own purposes

_toolchain/aarch64-linux-gnu-gcc:

#!/bin/bash
exec llamacc "$@"

As well as this Dockerfile at images/gcc-aarch64-linux-gnu-jammy/Dockerfile:

FROM ghcr.io/nelhage/llama as llama
FROM ubuntu:jammy
RUN apt-get update && apt-get -y install ca-certificates gcc-11-aarch64-linux-gnu gcc-aarch64-linux-gnu && apt-get clean

RUN cd /usr/bin/; for exe in aarch64-linux-gnu-*; do ln --force -s "/usr/bin/$exe" "${exe##aarch64-linux-gnu-}"; done
RUN cd /usr/bin/; ln --force -s /usr/bin/aarch64-linux-gnu-gcc cc
COPY --from=llama /llama_runtime /llama_runtime
WORKDIR /
ENTRYPOINT ["/llama_runtime"]

(and, of course, gcc-aarch64-linux-gnu installed locally on my local box, which is also running jammy.)

Much of the build seems to work, but I'm getting stuck building the configs module:

$ (llamacc arm64) make -j100 Image modules dtbs
  CALL    scripts/atomic/check-atomics.sh
  CALL    scripts/checksyscalls.sh
  CHK     include/generated/compile.h
  CC [M]  kernel/configs.o
/tmp/cczL3rE5.s: Assembler messages:
/tmp/cczL3rE5.s:9: Error: file not found: kernel/config_data.gz
Running llamacc: invoke: exit 1
make[1]: *** [scripts/Makefile.build:288: kernel/configs.o] Error 1
make: *** [Makefile:1868: kernel] Error 2

LLAMACC_REMOTE_ASSEMBLE is unset, so I'm not sure why it's trying to assemble remotely. I guess because kernel/configs.c has an inline assembly block:

/*
 * "IKCFG_ST" and "IKCFG_ED" are used to extract the config data from
 * a binary kernel image or a module. See scripts/extract-ikconfig.
 */
asm (
"   .pushsection .rodata, \"a\"     \n"
"   .ascii \"IKCFG_ST\"         \n"
"   .global kernel_config_data      \n"
"kernel_config_data:                \n"
"   .incbin \"kernel/config_data.gz\"   \n"
"   .global kernel_config_data_end      \n"
"kernel_config_data_end:            \n"
"   .ascii \"IKCFG_ED\"         \n"
"   .popsection             \n"
);

With LLAMACC_VERBOSE=1: gist because github doesn't like comments over 65k

Notably, I don't see anything about kernel/config_data.gz in the LLAMACC_VERBOSE output. The file does exist locally.

Calling LLAMACC_LOCAL=1 make kernel/configs.o succeeds, and seems to allow the rest of the build to continue. :tada:

[nelhage]

Oof. The basic situation here is:

• In order to build a source file in the cloud, llama has to ensure that any other files it depends on are also present in the Lambda instance.
• Llama uses the C preprocessor (via some command-line options exposed by gcc and clang) to discover those dependencies.
• Unfortunately, the assembler also has an include mechanism. This creates a dependency on an external file, which we cannot "see" through the preprocessor.
• That's the primary reason LLAMACC_REMOTE_ASSEMBLE is not enabled by default. That option only affects assembly source files (.S); we still assemble the output of the C compiler remotely, because the C compiler does not use assembly-level include mechanisms.
• But in this case, the kernel developers are writing C code that causes the resulting assembly to have such a mechanism, and so we break.

The obvious avenues to fix are:

• Somehow add support for sending assembly .include / .incbin dependencies along. I'm not sure how hard that is.
• Somehow detect this situation and fall back to a local build.
  • We could either do that before sending the file to the cloud, or we could detect this particular class of error and have the cloud send back a "rerun locally" error
  • Least satisfyingly, but maybe useful, we could have a general mechanism for the user to opt specific paths or specific command lines out of remote building, so that you could do the previous step by hand.

[q-flexai]

First things first, llama is awesome, thanks @nelhage!

I'm running into the same issue while trying to rebuild a Ubuntu x86 kernel from their Ubuntu-6.5.0-25.25 tag, no cross-compilation involved. It fails with the following:

/tmp/ccplWhxP.s: Assembler messages:
/tmp/ccplWhxP.s:10: Error: file not found: kernel/kheaders_data.tar.xz

And it's the same problem, really:

 $ git grep kheaders_data.tar.xz
kernel/kheaders.c:23:"  .incbin \"kernel/kheaders_data.tar.xz\" \n"

Tempted to grep for .incbin in llamacc and have it do the local build if it finds it...

[q-flexai]

I've added the workaround you were suggesting, but taking a big hammer approach of always doing the local build if it fails, keyed off by default under a new environment variable LLAMACC_LOCAL_FALLBACK=1. Tested both with/without and it did for my build.