search

ARMmbed / mbed-os-tf-m-regression-tests

An Mbed OS application that runs the TF-M regression tests
2 stars 12 forks source link

Reduce test timeout to 90 seconds #98

Closed LDong-Arm closed 3 years ago

LDong-Arm commented 3 years ago

The timeout set in the application is sent to the Greentea test host, becoming the maximum number of seconds the host will wait before getting expected test results.

Currently TF-M Regression and PSA Compliance tests take less than 20 seconds each, so there is no point waiting 600 seconds. The reduced timeout is still sufficient and speeds up retries (if enabled).

Note: The intention is use --retry-count in CI as storage on Musca S1 is sometimes unstable. It's not added to this PR because it may not be needed on local boards.

LDong-Arm commented 3 years ago

There's an issue with Travis connection. Will try to restart it. (Done)

Patater commented 3 years ago

@adbridge Could we run Nightly CI on this PR please?

Patater commented 3 years ago

@adbridge Could we run Nightly CI on this PR please?

Nevermind, this isn't going to affect mbed-os nightly.

LDong-Arm commented 3 years ago

I've run all tests using the Greentea test framework:

python3 test_psa_target.py -t GNUARM -m ARM_MUSCA_S1

and all tests passed as usual:

| target                      | platform_name | test suite               | result | elapsed_time (sec) | copy_method |                                                                                            
|-----------------------------|---------------|--------------------------|--------|--------------------|-------------|                                                                                            
| ARM_MUSCA_S1-GCC_ARM-115200 | ARM_MUSCA_S1  | initial_attestation      | OK     | 106.24             | default     |                                                                                            
| ARM_MUSCA_S1-GCC_ARM-115200 | ARM_MUSCA_S1  | internal_trusted_storage | OK     | 106.08             | default     |                                                                                            
| ARM_MUSCA_S1-GCC_ARM-115200 | ARM_MUSCA_S1  | protected_storage        | OK     | 109.6              | default     |                                                                                            
| ARM_MUSCA_S1-GCC_ARM-115200 | ARM_MUSCA_S1  | regression               | OK     | 112.32             | default     |                                                                                            
| ARM_MUSCA_S1-GCC_ARM-115200 | ARM_MUSCA_S1  | storage                  | OK     | 111.33             | default     |