Add a rudimentary build system script

[joergroedel]

Add a Python script which loads build recipes from JSON files, builds COCONUT and packages it into an IGVM file.

This script is the foundation for building user-space components and package them up into a file-system image during the build process.

Update: With the latest version the build script supports generating multiple IGVM files with one recipe. The use-case is mostly compile-testing, as per-target firmware files are not supported.

Closes: #65

[AdamCDunlap]

Is this able to compile the self-test binaries? i.e. the ones currently named things like coconut-test-qemu.igvm? Currently they're built with cargo test rather than cargo build

[joergroedel]

Is this able to compile the self-test binaries? i.e. the ones currently named things like coconut-test-qemu.igvm? Currently they're built with cargo test rather than cargo build

No, the test binaries can not yet be built with this script, I plan to add this in the future.

[AdamCDunlap]

I tested the vanadium-target.json target and it looks good.

I do have a bit of feedback.

% ./build asdf
Error: File not found: asdf
Building igvmbuilder...
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.08s
Building igvmmeasure...
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.10s
Traceback (most recent call last):
  File "/usr/local/google/home/acdunlap/svsm/svsm/./build", line 564, in <module>
    build(parse_arguments())
  File "/usr/local/google/home/acdunlap/svsm/svsm/./build", line 515, in build
    k_recipe = kernel_recipe(config)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/google/home/acdunlap/svsm/svsm/./build", line 148, in kernel_recipe
    kernel_json = config.get("kernel", {})
                  ^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'get'

I think this file not found error should be fatal, as currently it decides to build igvmbuilder and igvmmeasure before failing which can cause the real error to scroll off screen.

---

% FW_FILE=~/svsm/ovmf/ovmf_x64_csm_debug.fd ./build configs/vanadium-target.json
Building igvmbuilder...
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.07s
Building igvmmeasure...
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.10s
Building tdx-stage1...
./utils/gen_meta bin/meta.bin
cc    -c -o stage1/reset.o stage1/reset.S
cc -o bin/stage1-trampoline stage1/stage1-trampoline.o stage1/reset.o -nostdlib -Wl,--build-id=none -Wl,-Tstage1/stage1.lds -no-pie
Building stage2...
   Compiling svsm v0.1.0 (/usr/local/google/home/acdunlap/svsm/svsm/kernel)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 2.21s
Building svsm...
   Compiling svsm v0.1.0 (/usr/local/google/home/acdunlap/svsm/svsm/kernel)
error: expected one of `!` or `::`, found `error`
 --> kernel/src/svsm.rs:7:8
  |
7 | syntax error
  |        ^^^^^ expected one of `!` or `::`

error: could not compile `svsm` (bin "svsm") due to 1 previous error
Traceback (most recent call last):
  File "/usr/local/google/home/acdunlap/svsm/svsm/./build", line 564, in <module>
    build(parse_arguments())
  File "/usr/local/google/home/acdunlap/svsm/svsm/./build", line 520, in build
    kernel_parts = build_kernel_parts(k_recipe, args)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/google/home/acdunlap/svsm/svsm/./build", line 381, in build_kernel_parts
    binaries = recipe_build(k_recipe, get_svsm_kernel_target(), args)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/google/home/acdunlap/svsm/svsm/./build", line 340, in recipe_build
    binary = cargo_build(package, config, target, args)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/google/home/acdunlap/svsm/svsm/./build", line 295, in cargo_build
    runner.execute()
  File "/usr/local/google/home/acdunlap/svsm/svsm/./build", line 101, in execute
    subprocess.run(command, check=True)
  File "/usr/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['cargo', 'build', '--bin', 'svsm', '--features', 'vtpm', '--target', 'x86_64-unknown-none']' returned non-zero exit status 101.

Here I added a syntax error in svsm.rs. Building with no changes takes over 2 seconds to find the error since it compiles stage2 first. I know it'd be faster to find errors by running cargo check manually or using rust-analyzer, so this might not be a big problem.

I would like to see the python stack traces suppressed when it's erroring on a cargo build since the cargo error message makes it clear what's failing and the stack trace just adds noise.

[joergroedel]

Some build-time numbers from my machine:

build (all targets):

real    0m36.861s
user    1m38.291s
sys     0m25.873s

build (only qemu):

real    0m29.645s
user    1m33.087s
sys     0m23.509s

make:

real    0m30.903s
user    1m34.701s
sys     0m23.531s

[joergroedel]

I tested the vanadium-target.json target and it looks good.

Thanks for testing!

I think this file not found error should be fatal, as currently it decides to build igvmbuilder and igvmmeasure before failing which can cause the real error to scroll off screen.

This is fixed now, I changed the script to catch all these exceptions at the top-level function and exit the script after printing a message.

Here I added a syntax error in svsm.rs. Building with no changes takes over 2 seconds to find the error since it compiles stage2 first. I know it'd be faster to find errors by running cargo check manually or using rust-analyzer, so this might not be a big problem.

Yeah, that order comes from the definitions in the recipe file. If svsm is listed first, then it tries to build the svsm first, I will change the ordering in the recipes to always build svsm first.

I would like to see the python stack traces suppressed when it's erroring on a cargo build since the cargo error message makes it clear what's failing and the stack trace just adds noise.

Fixed now, thanks for your testing.

[stefano-garzarella]

@joergroedel this PR should close #65 , right?

[joergroedel]

I run this on the new config files:

python -mjson.tool $JSON > /tmp/json && mv /tmp/json $JSON

Thanks, formatted all JSON files with the json.tool.

[joergroedel]

@joergroedel this PR should close #65 , right?

Right, mentioned it in the PR description.