Improve kbuild

[nuclearcat]

1. [ ] Implement very basic fragment handling if set in kernel parameters, using fragments from legacy system This will unblock team to proceed on LAVA tests templates
2. [ ] Make sure proper artifacts generated and node model correctly updated Related issue This will unblock regression team to work on data model usability, improvements
3. [ ] Implement new fragments format with automatic conversion from legacy system with proper fragments merging If LAVA tests enabled at this moment, we might start tracking real regressions
4. [ ] Split kbuild job to multiple stages, to remove need of custom python modules and kernelci tools from compiler docker images. Discussion in progress

[nuclearcat]

api new kbuild implementation proposal

k8s/docker/shell steps:

STEP 1

container: kernelci/kernelci (or just need kernelci tools if it is a shell) commands:

kci build --yaml-config=/config.yaml --settings=/kernelci.toml --output=/scratch 65379184aee23a022ea7fb1f kbuild-gcc-10-x86-chromeos or kci build --yaml-config=/config.yaml --settings=/kernelci.toml --output=/scratch node.json kbuild-gcc-10-x86-chromeos

Tool will read the config.yaml and kernelci.toml files from the container, use API key to retrieve the node.json file from the kernelci.org server OR use the node.json, and then generate steps to build the kernel.

STEP 2

/scratch/compile.sh runs in cross-compiler specific container or just shell with (cross-)compiler installed contains for example:

#!/bin/sh

SRC=/scratch/linux.tgz

sh /scratch/firmware/step.sh 1>>/scratch/firmware/step.log 2>&1
if [ $? -ne 0 ]; then
    echo "Firmware fetch failed" >>/scratch/firmware/error
    # Doesn't make sense to continue if the firmware build failed
    exit 1
fi

# kci build will fetch and output fragments to /scratch/config in series to merge
# like 0.fragment, 1.fragment, 2.fragment, etc.
sh /scratch/config/step.sh 1>>/scratch/config/step.log 2>&1
if [ $? -ne 0 ]; then
    echo "Kernel config failed" >>/scratch/config/error
    # Doesn't make sense to continue if the config failed
    exit 1
fi

# building image
sh /scratch/build/step.sh 1>>/scratch/build/step.log 2>&1
if [ $? -ne 0 ]; then
    echo "Kernel image build failed" >>/scratch/build/error
    # Doesn't make sense to continue if the build failed
    exit 1
fi

# building and packing modules
sh /scratch/modules/step.sh 1>>/scratch/modules/step.log 2>&1
if [ $? -ne 0 ]; then
    echo "Kernel modules build failed" >>/scratch/modules/error
    # Doesn't make sense to continue if failed
    exit 1
fi

# building and packing dtbs
...
# kselftest (optional, so it wont exit if failed)
...

STEP 3

Runs in container kernelci/kernelci or shell with kernelci tools

kci build --yaml-config=/config.yaml --settings=/kernelci.toml --output=/scratch 65379184aee23a022ea7fb1f kbuild-gcc-10-x86-chromeos --submit

Will collect/parse logs, artifacts, and submit to kernelci, upload artifacts to storage, submit all relevant data to API node.

What is nice, that artifacts will include scripts to reproduce the build, just using generic cross-compiler container. So we can just pack everything for developer if we detect regression, without need to introduce to him kernelci specifics.