puzzleos / stubby

UEFI bootloader stub
GNU Lesser General Public License v2.1
12 stars 8 forks source link

Run tests on Ubuntu 22.04 #35

Closed smoser closed 1 year ago

smoser commented 1 year ago

I think that we originally used 20.04 due to a bug in 22.04. https://bugs.launchpad.net/bugs/1986692

That is fixed so we should be able to move to 22.04.

smoser commented 1 year ago

closing this for now because 22.04 runs significantly slower. Its not just 1 test, I saw this on multipe that I ran with it.

I collected the results below from the ''Boot test' output for this pr vs current tip

test 20.04 22.04 diff
mode=nvram
ib-noshim-allowed 42.08 74.14 32.06
sb-noshim-allowed 43.44 74.89 31.45
ib-shim-allowed 46.36 78.76 32.40
sb-shim-allowed 47.41 79.66 32.25
ib-shim-denied 44.22 76.79 32.57
sb-shim-denied 45.93 78.80 32.87
sb-noshim-marker1-allowed 41.99 74.84 32.85
sb-shim-marker1-allowed 45.56 77.14 31.58
mode=efi-shell
ib-noshim-allowed 33.01 50.27 17.26
sb-noshim-allowed 34.52 51.80 17.28
ib-shim-allowed 36.79 53.34 16.55
sb-shim-allowed 39.45 56.32 16.87
sb-shim-denied 32.10 46.70 14.60
ib-shim-denied 34.84 51.30 16.46
sb-noshim-marker1-allowed 31.81 48.32 16.51
sb-shim-marker1-allowed 35.37 52.32 16.95

The first thing I would dig on is if there is something in the ovmf firmware from 22.04 vs 20.04 that is causing the slowdown. That can be tested pretty easily by just publishing the 20.04 ovmf versions and downloading them rather than using the ones from the system (collect-firmwares does that).

I'm uploading the current result of 'collect-firmwares' running in a focal container here. focal-firmwares.tar.gz

smoser commented 1 year ago

This is pretty clearly the Jammy OVMF in one way or another. I'm running jammy locally and running the shell script below with 'true' and 'false' values for KVM.

Results (wall clock/"real") of the combined set of 8 tests in 8 threads look like:

kvm mode focal jammy
true efi-shell 16.495 24.291
true nvram 31.886 37.588
false efi-shell 48.175 82.063
false nvram 74.986 121.952
#!/bin/bash
b=${1:-results}
KVM=${KVM:-true}
for m in efi-shell nvram; do 
    for r in focal jammy; do 
        res=$b/results/$r/$m
        wkd=$b/workdir/$r/$m
        log=$b/$r-$m.log
        rm -Rf $res $wkd
        mkdir -p $res $wkd
        echo "# KVM=$KVM rel=$r mode=$m -> $log" 1>&2
        s=$SECONDS
        (
        echo "# KVM=$KVM rel=$r mode=$m"
        bash -c 'time "$@"' -- ./test/harness run \
            --boot-mode=$m --inputs=./inputs/$r \
            --results=$res --sbat=sbat.csv \
            --stubby=stubby.efi --work-dir=$wkd \
            test/tests.yaml > $log 2>&1
        ) > "$log" 2>&1
        echo "$((SECONDS-s))s $?"
    done
done