Open Source Firmware Remote Test Environment

Warning

!!! WARNING !!! This repository is in the process of migration and multiple major reworks. If you do not know what you are doing, consider not using it until at least v0.5.0 is released. When this is scheduled, link to such a milestone will appear here. !!! WARNING !!!

The following repository contains set of tests and other features to conduct Dasharo firmware validation procedures.

Test environment overview

Dasharo OSFV consists of following modules:

• dasharo-compatibility,
• dasharo-security,
• dasharo-performance,
• dasharo-stability.

In addition, keep in mind that due to the approach to generating release files, for the raptor-CS talos2 platform dedicated mechanism for testing environment and running tests have been implemented.

Supported platforms

Getting started

Initializing environment

• Clone repository and setup virtualenv:

git clone https://github.com/Dasharo/open-source-firmware-validation
cd open-source-firmware-validation
git submodule update --init --checkout
python3 -m virtualenv venv
source venv/bin/activate

• Install modules (in case of Raptor Talos II platform):

pip install -U -r requirements-openbmc.txt

• Install modules (in case of other platforms):

pip install -r requirements.txt

• If you initialize the environment and try to run the environment again you just need to use only this command:

source venv/bin/activate

NOTE: keywords.robot requires osfv_cli to be installed on the host system. Go through these steps to configure the scripts

Running tests

When running tests on Dasharo platforms use the following commands:

• For running a single test case:

robot -L TRACE -v rte_ip:$RTE_IP -v config:$CONFIG -v device_ip:$DEVICE_IP \
-t $TEST_CASE_ID $TEST_MODULE/$TEST_SUITE

• For running a single test suite:

robot -L TRACE -v rte_ip:$RTE_IP -v config:$CONFIG -v device_ip:$DEVICE_IP \
$TEST_MODULE/$TEST_SUITE

• For running a single test module:

robot -L TRACE -v rte_ip:$RTE_IP -v config:$CONFIG -v device_ip:$DEVICE_IP \
$TEST_MODULE

Parameters should be defined as follows:

• $DEVICE_IP - IP address of the DUT. Required only when there is no serial input enabled for the device, or tests are executed over SSH. Currently, this is the case for NovaCustom and MSI devices.
• $RTE_IP - IP address of the RTE. Required only if RTE is used on a given test stand.
• $FW_FILE - path to and name of the coreboot firmware file. This is usually not required when running single tests or suites, where flashing is not necessary.
• $CONFIG - platform config - see the platform-configs directory for available configurations.
• $TEST_MODULE - name of the test module (i.e. dasharo-compatibility),
• $TEST_SUITE - name of the test suite (i.e. uefi-shell.robot),
• $TEST_CASE_ID - ID of the requested to run test case (i.e. CBP001.001*). Note that after test case ID asterisk should be added, if you do not wish to provide the full test name here.

You can also run tests with -v snipeit:no in order to skip checking whether the platform is available on snipeit. By default, this is enabled.

When running tests on Talos2 platform use the following commands:

WARNING The support state of this platform in the main branch may vary. We should have a single documentation for all platforms. This effort is tracked in this issue.

• For running a single test case:

robot -L TRACE -v device_ip:$DEVICE_IP -v config:raptor-cs_talos2 -v fw_file:$FW_FILE \
-v bootblock_file:$BOOTBLOCK_FILE -v zImage_file:$ZIMAGE_FILE -v pnor_file:$PNOR_FILE \
-t $TEST_CASE_ID $TEST_MODULE/$TEST_SUITE

• For running single test suite:

robot -L TRACE -v device_ip:$DEVICE_IP -v config:raptor-cs_talos2 -v fw_file:$FW_FILE \
-v bootblock_file:$BOOTBLOCK_FILE -v zImage_file:$ZIMAGE_FILE -v pnor_file:$PNOR_FILE \
$TEST_MODULE/$TEST_SUITE

• For running single test module:

robot -L TRACE -v device_ip:$DEVICE_IP -v config:raptor-cs_talos2 -v fw_file:$FW_FILE \
-v bootblock_file:$BOOTBLOCK_FILE -v zImage_file:$ZIMAGE_FILE -v pnor_file:$PNOR_FILE \
./$TEST_MODULE

Parameters should be defined as follows:

• $DEVICE_IP - OBMC IP address (currently 192.168.20.9),
• $FW_FILE - path to and name of the coreboot firmware file,
• $BOOTBLOCK_FILE - path to and name of the bootblock file,
• $ZIMAGE_FILE - path to and name of the zImage file,
• $PNOR_FILE - path to and name of the pnor file,
• $TEST_MODULE - name of the test module (i.e. dasharo-compatibility),
• $TEST_SUITE - name of the test suite (i.e. coreboot-base-port),
• $TEST_CASE_ID - ID of the requested to run test case (i.e. CBP001.001). Note that after test case ID asterisk should be added. This is necessary due to the construction of the flag -t (or --test)

You can also run tests with -v snipeit:no in order to skip checking whether the platform is available on snipeit.

QEMU workflow

Make sure to proceed with Getting started section first.

Many of the test and keywords can be tested in emulation environment. This can greatly increase the development speed:

• there is no need to acquire hardware,
• there is no need to flash hardware, or resolve other hardware-related problems,
• the boot time (and responsivness in general) is much faster.

Booting

Following script assume that you have OVMF_CODE.fd and OVMF_VARS.fd in you current working directory. If those binaries will not be found script will download latest release of Dasharo (UEFI) for QEMU Q35.

If you want to use script in development workflow, most likely you have already built Dasharo (UEFI) for QEMU Q35 according to this instruction. In that case you would like to provide directory with Dasharo (UEFI) binaries as environment variable (DIR).

You may also decide to not use graphics user interface for QEMU. In that case choose mode nographic. If you run QEMU on a remote machine you may consider to use mode vnc with default port for graphical output being 5900.

Dasharo (UEFI) in QEMU can be started with:

./scripts/ci/qemu-run.sh graphic firmware

In this mode, a graphical QEMU window would popup, so you can observe the test flow, or control it manually. The actual testing will happen over serial, which is exposed via telnet. For more modes and options, please refer to the script's help text.

You may also build customized Dasharo firmware for QEMU (e.g. with some Dasharo options enabled or disabled). In such a case, please refer to:

• Building Manual in Dasharo for QEMU documentation
• Development section in Dasharo for QEMU documentation

Refer to the latest releases to see which test have been proven to work on QEMU so far.

You may also refer to the ./scripts/ci/qemu-self-test.sh, where we aim to keep testing common keywords, to ensure of their correct operation.

Contributing

• Install pre-commit hooks after cloning repository:

pre-commit install

Guidelines

A list of guidelines we shall follow during transition to improve the quality of this repository. We start with getting rid of duplicated keywords, reducing the size of keywords.robot file, and improving their overall quality.

There are other areas of interest that we will look into in the next steps and add as guidelines:

• variables (use Python/YAML, not robot syntax),
• platform-configs (get rid of duplication, and unused data),
• separate test for different OS into different suites,
• prepare the OS for running test suite via some dedicated tools (e.g. Ansible), rather than implementing keywords for that from scratch,
• reduce the number of unnecessary power events, so tests can finish quicker,
• improve overall code quality by enabling back more robocop checks we cannot pass right now,
• To Be Continued.

Pre-commit and CI checks

1. Make sure to use pre-commit locally. All pre-commit and other CI checks must pass of course, prior requesting for review. Please check the status of checks in your PR. If the failure is questionable, provide your arguments for that, rather than silently ignoring this fact.

Code style

1. It is automatically handled by robotidy. The current rules can be found here.

Keywords

1. No new keywords in keywords.robot will be accepted
   • new keywords must be placed in a logically divided modules, under lib/ directory
   • see openbmc-test-automation as a reference
   • if you need to modify something in keywords.robot, you should create a new module under lib/
   • if you add new keyword module, you should review the keywords.module and move related keywords there as well, if suitable
2. If keyword from keywords.robot can be reused or improved, do that instead of creating a new one
   • keyword duplication will not be accepted,
   • you will be asked to use/improve existing keywords instead.
3. You are encouraged to use Python for more sophisticaed or complex keywords (e.g. more convoluted data parsing and processing). We are not forced to use RF for all keywords. Especially when it is simply easier to use Python.
4. For reading from terminal (no matter if it is Telnet, or SSH), following keywords must be used:
   • Read From Terminal Until Prompt
   • Read From Terminal Until
   • Read From Terminal Usage of other keywords is prohibited. Whenever you modify a test/keyword, you should rework it to use one of the above.
5. For writing into terminal, following keywords must be used:
   • Execute Command In Terminal
   • Write Into Terminal
   • Write Bare Into Terminal Usage of other keywords is prohibited. Whenever you modify a test/keyword, you should rework it to use one of the above. You should use Execute Command In Terminal unless you have a very good reason not to. Thanks to that, your keyword will not leave floating output in buffer to be received by another keywords, not expecting that.

Documentation

• Each new (or modified) file, test, keyword, must have a [Documentation] section.

Useful refactoring tools

• sherlock
  • can detect unused keywords, and much more
• Renaming keywords
• Renaming Test Cases
• Renaming Variables