Running Regression + PSA Compliance tests suite without ITS erased

[jainvikas8]

When using the Mbed OS + TF-M v1.1 for ARM_MUSCA_B1 or using test_psa_target.py we see some ITS test failures.

The way to reproduce is as follows: Step 1: Clear ITS area with pyocd pyocd erase --help ITS - 0xA1F0000 to 0xA1FC000 for ARM_MUSCA_B1 Step 2: Compile and run TF-M 1.1v regression tests. (PASSED) - Re-run, without re-flashing, it will still pass. Step 3: Compile and run PSA Compliance - ITS suite (failed for one test)

TEST: 402 | DESCRIPTION: Write once error check
 [Info] Executing tests from non-secure
 [Info] Executing ITS tests
 [Info] UID 1 was created with PSA_STORAGE_FLAG_WRITE_ONCE previously
 [Check 1] Retrieve metadata for UID 1 and check content
 Failed at Checkpoint: 10
 Actual: 36
 Expected: 4
 TEST RESULT: FAILED (Error Code=0x00000001)

The problem with this flow of execution is:

• The UID was set with different data length - https://git.trustedfirmware.org/TF-M/trusted-firmware-m.git/tree/test/suites/its/its_tests_common.c#n70 as opposed to what https://github.com/ARM-software/psa-arch-tests/blob/master/api-tests/dev_apis/storage/test_s002/test_s002.c#L82 is checking.
• There is no consistency of data or pattern, as this is a test. The flag is WRITE_ONCE_UID, so this can have any data/size set when used. It could be the same test or another test suite.

Now let's change the flow Step 1: Clear ITS area Step 2: Compile and run PSA Compliance - ITS suite (PASSED) -- Re-run, without re-flashing, it will still pass. Step 3: Compile and run TF-M 1.1v regression tests. (Number of failed tests: 8 of 19)

Running Test Suite PSA internal trusted storage NS interface tests (TFM_ITS_TEST_1XXX)...
 > Executing 'TFM_ITS_TEST_1001'
 Description: 'Set interface'
 Set should not fail with valid UID (Failed at /mnt/c/Work/Workspace/tf-m/trusted-firmware-m/test/suites/its/its_tests_common.c:33)
 TEST FAILED!
 > Executing 'TFM_ITS_TEST_1004'
 Description: 'Set interface with write once UID'
 Write once data should not have changed (Failed at /mnt/c/Work/Workspace/tf-m/trusted-firmware-m/test/suites/its/its_tests_common.c:159)
 TEST FAILED!
 > Executing 'TFM_ITS_TEST_1007'
 Description: 'Get interface with invalid UIDs'
 Get succeeded with non-existant UID (Failed at /mnt/c/Work/Workspace/tf-m/trusted-firmware-m/test/suites/its/its_tests_common.c:375)
 TEST FAILED!
 > Executing 'TFM_ITS_TEST_1010'
 Description: 'Get info interface with write once UID'
 Size incorrect for write once UID (Failed at /mnt/c/Work/Workspace/tf-m/trusted-firmware-m/test/suites/its/its_tests_common.c:585)
 TEST FAILED!
 > Executing 'TFM_ITS_TEST_1011'
 Description: 'Get info interface with valid UID'
 Set should not fail (Failed at /mnt/c/Work/Workspace/tf-m/trusted-firmware-m/test/suites/its/its_tests_common.c:613)
 TEST FAILED!
 > Executing 'TFM_ITS_TEST_1013'
 Description: 'Remove interface with valid UID'
 Set should not fail (Failed at /mnt/c/Work/Workspace/tf-m/trusted-firmware-m/test/suites/its/its_tests_common.c:709)
 TEST FAILED!
 > Executing 'TFM_ITS_TEST_1017'
 Description: 'Multiple partial gets'
 Set should not fail (Failed at /mnt/c/Work/Workspace/tf-m/trusted-firmware-m/test/suites/its/its_tests_common.c:861)
 TEST FAILED!
 > Executing 'TFM_ITS_TEST_1019'
 Description: 'Set, get and remove interface with different asset sizes'
 Set should not fail with valid UID (Failed at /mnt/c/Work/Workspace/tf-m/trusted-firmware-m/test/suites/its/its_tests_common.c:996)
 TEST FAILED!
 TESTSUITE FAILED!Number of failed tests: 8 of 19

The problem with this flow of execution is:

• The PSA Compliance test - uses 2 WRITE_ONCE_UID instead of 1 used by Regression - https://github.com/ARM-software/psa-arch-tests/blob/master/api-tests/dev_apis/storage/test_s002/test_s002.c#L24 , https://git.trustedfirmware.org/TF-M/trusted-firmware-m.git/tree/test/suites/its/its_tests_common.h#n18
  • The regression test suite for TFM_ITS_TEST_1001 - doesn't have any error handling - https://git.trustedfirmware.org/TF-M/trusted-firmware-m.git/tree/test/suites/its/its_tests_common.c#n32 if the ID was already used as WRITE_ONCE.

One may argue to run these test suites we need to start clean (erase ITS), but that reduces the lifetime of flash.

Questions:

• Do we need to have synchronization between difference test suites (Regression & PSA Compliance storage test suites)?
• Do we need to have better handling of errors for WRITE_ONCE_UID flags? To reduce ITS erase for each test start? Does it reduce security implications? - https://github.com/ARM-software/psa-arch-tests/tree/master/api-tests/dev_apis#security-implication Or
• Do we just document the flow stating the suites need to be executed in a order to achieve desired results to avoid unexpected behavior?

[Patater]

This isn't a problem that can be solved by Mbed OS and its tests. TF-M and PSA Compliance tests will need to either be compatible with each other (and tested that they remain compatible with each other), or we need to erase between test runs. Until the tests are compatible with each other, which doesn't seem like anything valuable to do for either TF-M or PSA Compliance tests, we should erase between runs.

[jainvikas8]

This isn't a problem that can be solved by Mbed OS and its tests. TF-M and PSA Compliance tests will need to either be compatible with each other (and tested that they remain compatible with each other), or we need to erase between test runs. Until the tests are compatible with each other, which doesn't seem like anything valuable to do for either TF-M or PSA Compliance tests, we should erase between runs.

This may involve further changes to scripts that build the target (PR for Mbed OS), but how do we differentiate whether its a test suite run or release? (User would not be happy if the release build erased it's previous ITS storage)

[Patater]

We should test what we release. If this involves using a physical target and erasing its flash, that's what it'll take.

We don't ship the regression test config binaries, though. We should probably invest some time in determining how best to test the release-config binaries. It's a bit annoying that TF-M regression tests can't be used with release binaries. This is a separate issue, though.