search

Dasharo / open-source-firmware-validation

OSFV infrastructure with automated tests and scripts for managing test results
Apache License 2.0
6 stars 1 forks source link

PC Engines UEFI regression tests #237

Closed mkopec closed 2 months ago

macpijan commented 3 months ago

@pkubaj @mkopec please make sure to stay in sync if you are trying to run these tests on APUs at the same time

pkubaj commented 3 months ago

You mentioned that it's systemd that needs to timeout. Do you have configured only the NIC's that are actually up? If all the NIC's are set up to DHCP, but only one is connected, systemd will try to bring them all up, which may cause this timeout.

mkopec commented 3 months ago

I've left it at the default settings after installation, I didn't change any network settings at all

pkubaj commented 3 months ago

That may be the reason why it fails, I don't need extended timeout.

pkubaj commented 3 months ago

All the VBO tests fail with:

Verified-Boot                                                         | FAIL |
Suite setup failed:
'Unable to find image 'ghcr.io/dasharo/dasharo-sdk:v1.1.2' locally
v1.1.2: Pulling from dasharo/dasharo-sdk
531fc43e70ac: Already exists
5cb93b946394: Already exists
5223cbc1d9dc: Already exists
fa2252fed426: Already exists
4d45c1ad94a7: Pull complete
efb517a41838: Pull complete
8b8c3a2b8ed9: Pull complete
6c3011019bb9: Pull complete
7553ae6720d9: Pull complete
ef55e7fd811e: Pull complete
Digest: sha256:5011c33dd43bf2b648731c44be91d4b8339d194035e67d98450f2e7872b94e8f
Status: Downloaded newer image for ghcr.io/dasharo/dasharo-sdk:v1.1.2
create_new_keys.sh: ERROR: AP RO root key directory "/home/ubuntu/ApRoV1Signing-PreMP" not found. Run make_arv_root.sh to create it or specify --arv-root-path.
Failed to generate keys : (1)
' does not contain 'The Verified Boot keys were generated into following directory'
mkopec commented 3 months ago

Huh, that's a new error. I did not see it while testing. Might be related to the bump in SDK version

macpijan commented 3 months ago

Huh, that's a new error. I did not see it while testing. Might be related to the bump in SDK version

"Fix" for that error is: https://github.com/Dasharo/dasharo-tools/pull/7

Results so far:

DEVICE_IP=192.168.10.36 FW_FILE=pcengines_apu2_v0.9.0.rom CONFIG=pcengines-apu2 RTE_IP=192.168.10.172 ./scripts/run.sh dasharo-security/verified-boot.robot

==============================================================================
Verified-Boot                                                                 
==============================================================================
VBO006.002 Check whether the verstage was run :: Check whether the... | PASS |
------------------------------------------------------------------------------
VBO007.002 Boot from RW when correctly signed firmware is flashed ... | PASS |
------------------------------------------------------------------------------
VBO009.001 Recovery boot popup is displayed when incorrectly signe... | PASS |
------------------------------------------------------------------------------
VBO010.001 Recovery boot popup can be skipped :: Check whether the... | PASS |
------------------------------------------------------------------------------
VBO011.001 Recovery popup is not displayed when correctly signed f... | PASS |
------------------------------------------------------------------------------
VBO012.001 Self-signed binary is bootable without errors :: Check ... ...ls: cannot access '/home/ubuntu/test-firmware_resigned.rom': No such file or directory
VBO012.001 Self-signed binary is bootable without errors :: Check ... | FAIL |
Image size doesn't match the flash chip's size!
------------------------------------------------------------------------------
Verified-Boot                                                         | FAIL |
6 tests, 5 passed, 1 failed
==============================================================================

DEVICE_IP=192.168.10.72 FW_FILE=pcengines_apu3_v0.9.0.rom CONFIG=pcengines-apu3 RTE_IP=192.168.10.173 ./scripts/run.sh dasharo-security/verified-boot.robot

==============================================================================
Verified-Boot                                                                 
==============================================================================
VBO006.002 Check whether the verstage was run :: Check whether the... | PASS |
------------------------------------------------------------------------------
VBO007.002 Boot from RW when correctly signed firmware is flashed ... | PASS |
------------------------------------------------------------------------------
VBO009.001 Recovery boot popup is displayed when incorrectly signe... | FAIL |
No match found for 'login:' in 5 minutes. Output:
EFI stub: WARNING: Failed to obtain seed from EFI_RNG_PROTOCOL
------------------------------------------------------------------------------
VBO010.001 Recovery boot popup can be skipped :: Check whether the... | PASS |
------------------------------------------------------------------------------
VBO011.001 Recovery popup is not displayed when correctly signed f... | PASS |
------------------------------------------------------------------------------
VBO012.001 Self-signed binary is bootable without errors :: Check ... ...ls: cannot access '/home/ubuntu/test-firmware_resigned.rom': No such file or directory
VBO012.001 Self-signed binary is bootable without errors :: Check ... | FAIL |
Image size doesn't match the flash chip's size!
------------------------------------------------------------------------------
Verified-Boot                                                         | FAIL |
6 tests, 4 passed, 2 failed
==============================================================================

VBO009.001 FAIL on APU3 is rather random boot problem? VBO012.001 fail is consistent on both, something is probably still not right in the tests.

macpijan commented 3 months ago

It is reproducible:

robot -L TRACE -t "VBO012*" -v config:pcengines-apu2 -v rte_ip:192.168.10.172 -v device_ip:192.168.10.36 -v fw_file:pcengines_apu2_v0.9.0.rom dasharo-security/verified-boot.robot
==============================================================================
Verified-Boot                                                                 
==============================================================================
VBO012.001 Self-signed binary is bootable without errors :: Check ... ...ls: cannot access '/home/ubuntu/test-firmware_resigned.rom': No such file or directory
VBO012.001 Self-signed binary is bootable without errors :: Check ... | FAIL |
Image size doesn't match the flash chip's size!
------------------------------------------------------------------------------
Verified-Boot                                                         | FAIL |
1 test, 0 passed, 1 failed
==============================================================================
miczyg1 commented 3 months ago

make_arv_root.sh

Yes, new vboot needs some new keys...

This is how I generate the keys for our Enterprise firmware on laptops:

    # vboot keys
    /vboot/scripts/keygeneration/make_arv_root.sh "$VBOOT_KEY_DIR/arv_root"

    errorCheck "Failed to generate vboot ARV root key"

    /vboot/scripts/keygeneration/create_new_keys.sh \
        --8k-root --8k-recovery \
        --output "$VBOOT_KEY_DIR/vboot" --arv-root-path "$VBOOT_KEY_DIR/arv_root"

    errorCheck "Failed to generate vboot keys"
macpijan commented 3 months ago

@miczyg1

This is how I generate the keys for our Enterprise firmware on laptops:

Is there any reason you are not using procedures we have in the tools, docs, tests? I would appreciate contribution to them if necessary.

Created PR with your suggestions:

https://github.com/Dasharo/dasharo-tools/pull/9

mkopec commented 3 months ago

@macpijan rebased and resolved all comments