test: drivers.flash.common.default: fails on frdm and ite.

Describe the bug The tests/drivers/flash/common/drivers.flash.common.default fails on frdm and ite board.

To Reproduce Steps to reproduce the behavior:

cd zephyr workspace
source zephyr-env.sh

twister --device-testing --hardware-map ./map_frdm.yml -p frmd_k64f -T tests/drivers/flash/common/drivers.flash.common.default -vv

twister --device-testing --hardware-map ./map_ite.yml -p it8xxx2_evb -T tests/drivers/flash/common/drivers.flash.common.default -vv

Impact annoyance

Logs and console output <path_to>/twister-out/frdm_k64f/tests/drivers/flash/common/drivers.flash.common.default/handler.log

*** Booting Zephyr OS build v3.7.0-1436-g4c16ec1fba56 ***
Running TESTSUITE flash_driver
===================================================================
Test will run on device flash-controller@40020000
START - test_flash_erase
PASS - test_flash_erase in 0.012 seconds
===================================================================
START - test_flash_fill
PASS - test_flash_fill in 0.048 seconds
===================================================================
START - test_flash_flatten
PASS - test_flash_flatten in 0.050 seconds
===================================================================
START - test_flash_page_layout
start_offset=0xe2000    size=4096       index=226
page_count=256
PASS - test_flash_page_layout in 0.006 seconds
===================================================================
START - test_negative_flash_erase
TEST_FLASH_START = 0x0
TEST_FLASH_SIZE = 0x100000
PASS - test_negative_flash_erase in 0.005 seconds
===================================================================
START - test_negative_flash_fill
E: Negative offset not allowed
E: Incorrect size or offset alignment, expected 8
PASS - test_negative_flash_fill in 0.008 seconds
===================================================================
START - test_negative_flash_flatten
PASS - test_negative_flash_flatten in 0.001 seconds
===================================================================
START - test_negative_flash_read
E: ***** BUS FAULT *****
E:   Precise data bus error
E:   BFAR Address: 0xffffffff
E: r0/a1:  0x200009f0  r1/a2:  0xffffffff  r2/a3:  0x000001ff
E: r3/a4:  0x200009ef r12/ip:  0x00005b21 r14/lr:  0x00005b33
E:  xpsr:  0xa1000000
E: Faulting instruction address (r15/pc): 0x000063ba
E: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0
E: Current thread: 0x20000058 (test_negative_flash_read)
E: Halting system

<path_to>/twister-out/it8xxx2_evb/tests/drivers/flash/common/drivers.flash.common.default/handler.log

***** delaying boot 500ms (per build configuration) *****
*** Booting Zephyr OS build v3.7.0-1394-g0a7908588c75 (delayed boot 500ms) ***
Running TESTSUITE flash_driver
===================================================================
Test will run on device flash-controller@f01000
START - test_flash_erase
PASS - test_flash_erase in 0.068 seconds
===================================================================
START - test_flash_fill
PASS - test_flash_fill in 0.110 seconds
===================================================================
START - test_flash_flatten
PASS - test_flash_flatten in 0.123 seconds
===================================================================
START - test_flash_page_layout
start_offset=0x60000    size=4096   index=96
page_count=256
PASS - test_flash_page_layout in 0.006 seconds
===================================================================
START - test_negative_flash_erase
TEST_FLASH_START = 0x80000000
TEST_FLASH_SIZE = 0x100000
PASS - test_negative_flash_erase in 0.006 seconds
===================================================================
START - test_negative_flash_fill
E: Incorrect size or offset alignment, expected 4
E: Negative offset not allowed
E: Incorrect size or offset alignment, expected 4
PASS - test_negative_flash_fill in 0.013 seconds
===================================================================
START - test_negative_flash_flatten
PASS - test_negative_flash_flatten in 0.001 seconds
===================================================================
START - test_negative_flash_read
Assertion failed at WEST_TOPDIR/<path_to>tests/drivers/flash/common/src/main.c:427: flash_driver_test_negative_flash_read: (rc < 0 is false)
Invalid use of flash_read returned 0
FAIL - test_negative_flash_read in 0.021 seconds
===================================================================
START - test_negative_flash_write
ASSERTION FAIL [!terminate] @ <path_to>/kernel/sched.c:493
aborted _current back from dead
E:      a0: 00000004    t0: 00000000
E:      a1: 000001ed    t1: 00000008
E:      a2: 00000000    t2: aaaaaaaa
E:      a3: 80100958    t3: 00000010
E:      a4: 00000000    t4: 00000003
E:      a5: 80100e80    t5: 00000000
E:      a6: 00000000    t6: 00000010
E:      a7: 0000002d
E:      sp: 80101840
E:      ra: 8000636e
E:    mepc: 80001e2c
E: mstatus: 00001880
E: 
E:      s0: 80100e80    s6: 00000000
E:      s1: 80100000    s7: 00000000
E:      s2: 00001888    s8: 00000000
E:      s3: 00000001    s9: 00000000
E:      s4: 80100864   s10: 00000000
E:      s5: 80100000   s11: 00000000
E: 
E: call trace:
E: 
E: >>> ZEPHYR FATAL ERROR 4: Kernel panic on CPU 0
E: Current thread: 0x80100958 (main)
E: Halting system

Environment:

OS: Linux
Toolchain Zephyr_sdk: zephyr-sdk-0.16.8
Zephyr version: v3.7.0-1394-g0a7908588c75

Result of bisect:

1a48a0174fd519f08703c1e1c0658c545bab3372 is the first bad commit
commit 1a48a0174fd519f08703c1e1c0658c545bab3372
Author: Sebastian Głąb <sebastian.glab@nordicsemi.no>
Date:   Wed Jul 3 09:04:49 2024 +0200

    tests: drivers: flash: common: Increase test coverage

    Add functional test for flash_erase().
    Add negative tests for flash_read(), flas_write(), flash_erase(),
    flash_fill() and flash_flatten().
    Add functional test for flash page layout.

    Signed-off-by: Sebastian Głąb <sebastian.glab@nordicsemi.no>

 tests/drivers/flash/common/src/main.c    | 282 +++++++++++++++++++++++++++++++
 tests/drivers/flash/common/testcase.yaml |   4 +-
 2 files changed, 285 insertions(+), 1 deletion(-)

@nordic-segl can you take a look ?

test_negative_flash_read checks return value from flash_read() when:

memory address is (TEST_FLASH_START - 1);
memory address is (TEST_FLASH_START + TEST_FLASH_SIZE);
length of data to be read is bigger than flash size;
length of data to be read is 0;

I see that on frdm_k64f: TEST_FLASH_START = 0x0 TEST_FLASH_SIZE = 0x100000 There is BUS ERROR when flash_read() tries to read from (TEST_FLASH_START - 1) = 0xffffffff. It's not trivial for 'generic' driver to know flash boundaries for the target selected for compilation. I can add skip in this test when flash starts at 0x0. For example:

    if (TEST_FLASH_START == 0) {
        /* Skip this test because some drivers are unable to detect that address 0xffffffff is invalid. */
        ztest_test_skip();
    }

However, I don't know what will happen in next three checks (see points 2. - 4. above).

I see that on it8xxx2_evb: TEST_FLASH_START = 0x80000000 TEST_FLASH_SIZE = 0x100000 test_negative_flash_read is FAILED because reading from 0x7fffffff was successful. test_negative_flash_write resulted in Kernel panic when flash_write() was called with invalid memory address.

On both targets flash_erase(), flash_fill(), flash_flatten() were able to detect invalid addresses / sizes. While, issue occurs for flash_read() and flash_write() only.

Issue is observed at run time. Since, I don't have these boards on my desk, it will be difficult for me to investigate.

It seems that driver lacks check for memory boundary in read/write functions. However, there is no clear statement that such check must be done. API says that return value is 0 on success, negative errno code on fail.

Let me know your opinion. Can this be handled by adding more checks in the driver? Or, negative tests shall be removed as they require checking input values by the read/write functions. Which is NOT clearly stated by API as mandatory.

Hi I'm also seeing failures on mr_canhubk3 board.

test_negative_flash_erase pass, but the parameter errors detected by the driver are actually different than the one the test was attempting to check (address out of range). And I believe most of the NOR flash drivers will present a similar behavior. A similar situation happens with test_negative_flash_flatten.

START - test_negative_flash_erase
TEST_FLASH_START = 0x400000
TEST_FLASH_SIZE = 0x3f4000
E: addr 4194303 is not sector-aligned          -> /* Check error returned when erasing memory at wrong address (too low) */
E: size 512 is not a multiple of sectors       -> /* Check error returned when erasing memory at wrong address (too high) */
E: size 4145153 is not a multiple of sectors   -> /* Check error returned when erasing to large chunk of memory */
 PASS - test_negative_flash_erase in 0.017 seconds

 START - test_negative_flash_flatten
E: addr 4194303 is not sector-aligned           -> /* Check error returned when flatten memory at wrong address (too low) */
E: size 512 is not a multiple of sectors        -> /* Check error returned when flatten memory at wrong address (too high) */
E: size 4145153 is not a multiple of sectors    -> /* Check error returned when flatten to large chunk of memory */
 PASS - test_negative_flash_flatten in 0.012 seconds

test_negative_flash_fill fails because the offset=0x003F_FFFF / size=32 of the flash_write() being called by flash_fill(), falls within the memory device which has size of 0x0080_0000. The flash driver checks the boundary of the memory access for all read/write/erase operations, but the problem is that the negative tests are taking the TEST_FLASH_START/TEST_FLASH_SIZE from zephyr,flash chosen, which in in my target (and I believe many others) is the internal SoC flash, whereas the flash device under test containing the TEST_AREA is a different external flash. The same applies for test_negative_flash_read and test_negative_flash_write results.

START - test_negative_flash_fill

    Assertion failed at WEST_TOPDIR/zephyr/tests/drivers/flash/common/src/main.c:369: flash_driver_test_negative_flash_fill: (rc < 0 is false)
Invalid use of flash_fill returned 0
 FAIL - test_negative_flash_fill in 0.018 seconds
===================================================================
START - test_negative_flash_read

    Assertion failed at WEST_TOPDIR/zephyr/tests/drivers/flash/common/src/main.c:427: flash_driver_test_negative_flash_read: (rc < 0 is false)
Invalid use of flash_read returned 0
 FAIL - test_negative_flash_read in 0.017 seconds
===================================================================
START - test_negative_flash_write

    Assertion failed at WEST_TOPDIR/zephyr/tests/drivers/flash/common/src/main.c:456: flash_driver_test_negative_flash_write: (rc < 0 is false)
Invalid use of flash_write returned 0
 FAIL - test_negative_flash_write in 0.017 seconds

In order to make all tests pass I need to do the following modifications:

set #define TEST_FLASH_START 0x0 and #define TEST_FLASH_SIZE 0x800000 to match the actual size of the flash device where the storage_partition is

make the erase size of the negative erase test aligned to page size as most of NOR flash will expect


diff --git a/tests/drivers/flash/common/src/main.c b/tests/drivers/flash/common/src/main.c
index 7889e09e8f5..8e8ed3ece8f 100644
--- a/tests/drivers/flash/common/src/main.c
+++ b/tests/drivers/flash/common/src/main.c
@@ -324,6 +324,8 @@ ZTEST(flash_driver, test_flash_erase)
ZTEST(flash_driver, test_negative_flash_erase)
{
int rc;

size_t erase_size = (page_info.size * ((EXPECTED_SIZE + page_info.size - 1) /
page_info.size));

if !defined(TEST_AREA)

/* Flash memory boundaries are correctly calculated @@ -336,11 +338,11 @@ ZTEST(flash_driver, test_negative_flash_erase) TC_PRINT("TEST_FLASH_SIZE = 0x%lx\n", (unsigned long)TEST_FLASH_SIZE);

/ Check error returned when erasing memory at wrong address (too low) /
rc = flash_erase(flash_dev, (TEST_FLASH_START - 1), EXPECTED_SIZE);
rc = flash_erase(flash_dev, (TEST_FLASH_START - 1), erase_size); zassert_true(rc < 0, "Invalid use of flash_erase returned %d", rc);

/ Check error returned when erasing memory at wrong address (too high) /
rc = flash_erase(flash_dev, (TEST_FLASH_START + TEST_FLASH_SIZE), EXPECTED_SIZE);
rc = flash_erase(flash_dev, (TEST_FLASH_START + TEST_FLASH_SIZE), erase_size); zassert_true(rc < 0, "Invalid use of flash_erase returned %d", rc);

/ Check error returned when erasing to large chunk of memory /

@nordic-segl I'm assigning you to not leave this unassigned, but feel free to re-assign it if there's a better fit. I can provide further testing at least on this board if needed. Thanks!

I will move negative tests to a separate test and add filter for platforms. Many vendors are affected. Taking Flash start address and it's size from zephyr,flash doesn't work for all targets.

I will move negative tests to a separate test and add filter for platforms. Many vendors are affected.

ok, that makes sense! Thanks

First attempt to fix this issue is here: https://github.com/zephyrproject-rtos/zephyr/pull/77380

Issue seen on STM32 as well

Fix is ready: https://github.com/zephyrproject-rtos/zephyr/pull/77380

Please, do review. "At least 2 approving reviews are required by reviewers with write access."

Fix is ready: #77380

Please, do review. "At least 2 approving reviews are required by reviewers with write access."

Let me verify this fix form frdm and ite perspective.

PS: frdm_k64f twister --device-testing --hardware-map ../../frdm_map.yaml -T tests/drivers/flash/common/ -vv twister throws. FAILED RunID mismatch.

INFO    - Using Ninja..
INFO    - Zephyr version: v3.7.0-1474-gaf09fcff7d19
DEBUG   - Running cmake script <path_to>/zephyrproject/zephyr/cmake/verify-toolchain.cmake
DEBUG   - Calling cmake: /usr/bin/cmake -DFORMAT=json -P <path_to>/zephyrproject/zephyr/cmake/verify-toolchain.cmake
DEBUG   - Finished running <path_to>/zephyrproject/zephyr/cmake/verify-toolchain.cmake
INFO    - Using 'zephyr' toolchain.
DEBUG   - Reading test case configuration files under <path_to>/zephyrproject/zephyr/tests/drivers/flash/common...
DEBUG   - Found possible testsuite in<path_to>/zephyrproject/zephyr/tests/drivers/flash/common
DEBUG   - No duplicates found.
DEBUG   - Reading platform configuration files under ['<path_to>/zephyrproject/zephyr/boards', '<path_to>/zephyrproject/zephyr/subsys/testsuite/boards']...
DEBUG   - platform filter: ['frdm_k64f']
DEBUG   -   vendor filter: []
DEBUG   -     arch_filter: None
DEBUG   -      tag_filter: None
DEBUG   -     exclude_tag: None
INFO    - Building initial testsuite list...
DEBUG   - frdm_k64f                 tests/drivers/flash/common/drivers.flash.common.nrf_qspi_nor SKIPPED: Not in testsuite platform allow list
DEBUG   - frdm_k64f                 tests/drivers/flash/common/drivers.flash.common.nrf_qspi_nor.size_in_bytes SKIPPED: Not in testsuite platform allow list
DEBUG   - frdm_k64f                 tests/drivers/flash/common/drivers.flash.common.nrf_qspi_nor_4B_addr SKIPPED: Not in testsuite platform allow list
DEBUG   - frdm_k64f                 tests/drivers/flash/common/drivers.flash.common.soc_flash_nrf SKIPPED: Not in testsuite platform allow list
DEBUG   - frdm_k64f                 tests/drivers/flash/common/drivers.flash.common.no_explicit_erase SKIPPED: Not in testsuite platform allow list
DEBUG   - frdm_k64f                 tests/drivers/flash/common/drivers.flash.common.tfm_ns SKIPPED: Not runnable on device
DEBUG   - frdm_k64f                 tests/drivers/flash/common/drivers.flash.common.stm32 SKIPPED: Not in testsuite platform allow list
DEBUG   - frdm_k64f                 tests/drivers/flash/common/drivers.flash.common.mx25r_high_perf SKIPPED: Not in testsuite platform allow list
DEBUG   - frdm_k64f                 tests/drivers/flash/common/drivers.flash.common.mr_canhubk3_sfdp_runtime SKIPPED: Not in testsuite platform allow list
DEBUG   - frdm_k64f                 tests/drivers/flash/common/drivers.flash.common.spi_nor SKIPPED: Not in testsuite platform allow list
DEBUG   - frdm_k64f                 tests/drivers/flash/common/drivers.flash.common.spi_nor_wp_hold SKIPPED: Not in testsuite platform allow list
DEBUG   - frdm_k64f                 tests/drivers/flash/common/drivers.flash.common.sam0 SKIPPED: Not in testsuite platform allow list
DEBUG   - frdm_k64f                 tests/drivers/flash/common/drivers.flash.common.gd32 SKIPPED: Not in testsuite platform allow list
INFO    - Writing JSON report <path_to>/zephyrproject/zephyr/twister-out/testplan.json

Device testing on:

| Platform   | ID                                               | Serial device   |
|------------|--------------------------------------------------|-----------------|
| frdm_k64f  | 02400b0147784e4500029004d7450023ddb1000097969900 | /dev/ttyACM0    |

INFO    - JOBS: 8
INFO    - Adding tasks to the queue...
DEBUG   - adding frdm_k64f/tests/drivers/flash/common/drivers.flash.common.default
INFO    - Added initial list of jobs to queue
DEBUG   - Running cmake on <path_to>/zephyrproject/zephyr/tests/drivers/flash/common for frdm_k64f
DEBUG   - Calling cmake: /usr/bin/cmake -B<path_to>/zephyrproject/zephyr/twister-out/frdm_k64f/tests/drivers/flash/common/drivers.flash.common.default -DTC_RUNID=78a9c8383f24bf2eaf670dbb87abf08d -DCONFIG_COMPILER_WARNINGS_AS_ERRORS=y -DEXTRA_GEN_DEFINES_ARGS=--edtlib-Werror -GNinja -S<path_to>/zephyrproject/zephyr/tests/drivers/flash/common -DBOARD=frdm_k64f -DMODULES=dts,kconfig -P<path_to>/zephyrproject/zephyr/cmake/package_helper.cmake
DEBUG   - Launched 8 jobs
DEBUG   - Finished running cmake <path_to>/zephyrproject/zephyr/tests/drivers/flash/common for frdm_k64f in 2.77 seconds
DEBUG   - Running cmake on <path_to>/zephyrproject/zephyr/tests/drivers/flash/common for frdm_k64f
DEBUG   - Calling cmake: /usr/bin/cmake -B/home/zephyr/zephyrproject/zephyr/twister-out/frdm_k64f/tests/drivers/flash/common/drivers.flash.common.default -DTC_RUNID=78a9c8383f24bf2eaf670dbb87abf08d -DCONFIG_COMPILER_WARNINGS_AS_ERRORS=y -DEXTRA_GEN_DEFINES_ARGS=--edtlib-Werror -GNinja -S<path_to>/zephyrproject/zephyr/tests/drivers/flash/common -DBOARD=frdm_k64f
DEBUG   - Finished running cmake <path_to>/zephyrproject/zephyr/tests/drivers/flash/common for frdm_k64f in 3.21 seconds
DEBUG   - build test: frdm_k64f/tests/drivers/flash/common/drivers.flash.common.default
DEBUG   - Building <path_to>/zephyrproject/zephyr/tests/drivers/flash/common for frdm_k64f
DEBUG   - Running /usr/bin/cmake --build <path_to>/zephyrproject/zephyr/twister-out/frdm_k64f/tests/drivers/flash/common/drivers.flash.common.default
DEBUG   - Finished building <path_to>/zephyrproject/zephyr/tests/drivers/flash/common for frdm_k64f in 2.34 seconds
DEBUG   - Determine test cases for test instance: frdm_k64f/tests/drivers/flash/common/drivers.flash.common.default
DEBUG   - Determine test cases for test suite: drivers.flash.common.default
DEBUG   - Test instance frdm_k64f/tests/drivers/flash/common/drivers.flash.common.default already has 5 cases.
DEBUG   - drivers.flash.common.default.flash_page_layout, drivers.flash.common.default.flash_erase, drivers.flash.common.default.flash_flatten, drivers.flash.common.default.flash_fill, drivers.flash.common.default.read_unaligned_address in <path_to>/zephyrproject/zephyr/twister-out/frdm_k64f/tests/drivers/flash/common/drivers.flash.common.default/zephyr/zephyr.elf
DEBUG   - run test: frdm_k64f/tests/drivers/flash/common/drivers.flash.common.default
DEBUG   - Reset instance status from 'passed' to None before run.
DEBUG   - Retain DUT:frdm_k64f, Id:02400b0147784e4500029004d7450023ddb1000097969900, counter:1, failures:0
DEBUG   - Using serial device /dev/ttyACM0 @ 115200 baud
DEBUG   - Flash command: ['west', 'flash', '--skip-rebuild', '-d', '<path_to>/zephyrproject/zephyr/twister-out/frdm_k64f/tests/drivers/flash/common/drivers.flash.common.default', '--runner', 'pyocd', '--', '--dev-id', '02400b0147784e4500029004d7450023ddb1000097969900']
DEBUG   - DEVICE: 047 seconds
DEBUG   - DEVICE: - PASS - [flash_driver.test_flash_flatten] duration = 0.049 seconds
DEBUG   - DEVICE: - PASS - [flash_driver.test_flash_page_layout] duration = 0.006 seconds
DEBUG   - DEVICE: - PASS - [flash_driver.test_read_unaligned_address] duration = 0.016 seconds
DEBUG   - DEVICE:
DEBUG   - DEVICE: ------ TESTSUITE SUMMARY END ------
DEBUG   - DEVICE:
DEBUG   - DEVICE: ===================================================================
DEBUG   - DEVICE: RunID: 3a609a6bb41743f92519b53d0fcd0bd7
DEBUG   - DEVICE: PROJECT EXECUTION SUCCESSFUL
DEBUG   - -- west flash: using runner pyocd
-- runners.pyocd: Flashing file: <path_to>/zephyrproject/zephyr/twister-out/frdm_k64f/tests/drivers/flash/common/drivers.flash.common.default/zephyr/zephyr.bin
[---|---|---|---|---|---|---|---|---|----]
[========================================]

DEBUG   - Expected suite names:['flash_driver']
DEBUG   - Detected suite names:['flash_driver', 'flash_driver', 'flash_driver']
DEBUG   - Release DUT:frdm_k64f, Id:02400b0147784e4500029004d7450023ddb1000097969900, counter:1, failures:1
DEBUG   - run status: frdm_k64f/tests/drivers/flash/common/drivers.flash.common.default failed
INFO    - 14/14 frdm_k64f                 tests/drivers/flash/common/drivers.flash.common.default  FAILED RunID mismatch (device: 02400b0147784e4500029004d7450023ddb1000097969900, 2.366s)
ERROR   - see: <path_to>/zephyrproject/zephyr/twister-out/frdm_k64f/tests/drivers/flash/common/drivers.flash.common.default/handler.log

INFO    - 14 test scenarios (14 test instances) selected, 13 configurations skipped (13 by static filter, 0 at runtime).
--------------------------------
Total test suites: 14
Total test cases: 70
Executed test cases: 5
Skipped test cases: 65
Completed test suites: 14
Passing test suites: 0
Failing test suites: 1
Skipped test suites: 13
Skipped test suites (runtime): 0
Skipped test suites (filter): 13
Errors: 0
--------------------------------
INFO    - 0 of 14 test configurations passed (0.00%), 1 failed, 0 errored, 13 skipped with 0 warnings in 11.19 seconds
INFO    - In total 5 test cases were executed, 65 skipped on 1 out of total 1 platforms (100.00%)
INFO    - 1 test configurations executed on platforms, 0 test configurations were only built.

Hardware distribution summary:

| Board     | ID                                               |   Counter |   Failures |
|-----------|--------------------------------------------------|-----------|------------|
| frdm_k64f | 02400b0147784e4500029004d7450023ddb1000097969900 |         1 |          1 |
INFO    - Saving reports...
INFO    - Writing JSON report <path_to>/zephyrproject/zephyr/twister-out/twister.json
INFO    - Writing xunit report <path_to>/zephyrproject/zephyr/twister-out/twister.xml...
INFO    - Writing xunit report <path_to>/zephyrproject/zephyr/twister-out/twister_report.xml...
INFO    - -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
INFO    - The following issues were found (showing the top 10 items):
INFO    - 1) tests/drivers/flash/common/drivers.flash.common.default on frdm_k64f failed (RunID mismatch)
INFO    -
INFO    - To rerun the tests, call twister using the following commandline:
INFO    - west twister -p <PLATFORM> -s <TEST ID>, for example:
INFO    -
INFO    - west twister -p frdm_k64f -s tests/drivers/flash/common/drivers.flash.common.default
INFO    - or with west:
INFO    - west build -p -b frdm_k64f tests/drivers/flash/common -T drivers.flash.common.default
INFO    - -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
INFO    - Run completed

ITE passes with this fix (https://github.com/zephyrproject-rtos/zephyr/pull/77380)

twister --device-testing --hardware-map <path_to>/ite.yaml -T tests/drivers/flash/common/ -v

INFO    - Using Ninja..
INFO    - Zephyr version: v3.6.0-9205-gaf09fcff7d19
INFO    - Using 'zephyr' toolchain.
INFO    - Building initial testsuite list...
INFO    - Writing JSON report <path_to>/zephyrproject/zephyr/twister-out/testplan.json

Device testing on:

| Platform    | ID       | Serial device   |
|-------------|----------|-----------------|
| it8xxx2_evb | A50285BI | /dev/ttyUSB0    |

INFO    - JOBS: 4
INFO    - Adding tasks to the queue...
INFO    - Added initial list of jobs to queue
INFO    - 14/14 it8xxx2_evb               tests/drivers/flash/common/drivers.flash.common.default PASSED (device: A50285BI, 7.187s)

INFO    - 14 test scenarios (14 test instances) selected, 13 configurations skipped (13 by static filter, 0 at runtime).
INFO    - 1 of 14 test configurations passed (100.00%), 0 failed, 0 errored, 13 skipped with 0 warnings in 25.95 seconds
INFO    - In total 5 test cases were executed, 65 skipped on 1 out of total 1 platforms (100.00%)
INFO    - 1 test configurations executed on platforms, 0 test configurations were only built.

Hardware distribution summary:

| Board       | ID       |   Counter |   Failures |
|-------------|----------|-----------|------------|
| it8xxx2_evb | A50285BI |         1 |          0 |
INFO    - Saving reports...
INFO    - Writing JSON report <path_to>zephyrproject/zephyr/twister-out/twister.json
INFO    - Writing xunit report <path_to>/zephyrproject/zephyr/twister-out/twister.xml...
INFO    - Writing xunit re

Regarding:

frdm_k64f
twister --device-testing --hardware-map ../../frdm_map.yaml -T tests/drivers/flash/common/ -vv
twister throws. FAILED RunID mismatch.

It looks like a Twister issue. Does it happen every time?

I see that test was compiled with following RUNID=78a9c8383f24bf2eaf670dbb87abf08d

DEBUG   - Calling cmake: /usr/bin/cmake -B<path_to>/zephyrproject/zephyr/twister-out/frdm_k64f/tests/drivers/flash/common/drivers.flash.common.default -DTC_RUNID=78a9c8383f24bf2eaf670dbb87abf08d -DCONFIG_COMPILER_WARNINGS_AS_ERRORS=y -DEXTRA_GEN_DEFINES_ARGS=--edtlib-Werror -GNinja -S<path_to>/zephyrproject/zephyr/tests/drivers/flash/common -DBOARD=frdm_k64f -DMODULES=dts,kconfig -P<path_to>/zephyrproject/zephyr/cmake/package_helper.cmake

while console output is from RUNID=3a609a6bb41743f92519b53d0fcd0bd7

DEBUG   - DEVICE: ===================================================================
DEBUG   - DEVICE: RunID: 3a609a6bb41743f92519b53d0fcd0bd7
DEBUG   - DEVICE: PROJECT EXECUTION SUCCESSFUL

As if Twister parsed output from before device was flashed.

You're right, My setup issue. I've tested it on two different boards (frdms) and it's working properly.


DEBUG   - DEVICE: START - test_read_unaligned_address
DEBUG   - DEVICE: PASS - test_read_unaligned_address in 0.015 seconds
DEBUG   - DEVICE: ===================================================================
DEBUG   - DEVICE: TESTSUITE flash_driver succeeded
DEBUG   - DEVICE: 
DEBUG   - DEVICE: ------ TESTSUITE SUMMARY START ------
DEBUG   - DEVICE: 
DEBUG   - DEVICE: SUITE PASS - 100.00% [flash_driver]: pass = 5, fail = 0, skip = 0, total = 5 duration = 0.131 seconds
DEBUG   - DEVICE: - PASS - [flash_driver.test_flash_erase] duration = 0.013 seconds
DEBUG   - DEVICE: - PASS - [flash_driver.test_flash_fill] duration = 0.047 seconds
DEBUG   - DEVICE: - PASS - [flash_driver.test_flash_flatten] duration = 0.050 seconds
DEBUG   - DEVICE: - PASS - [flash_driver.test_flash_page_layout] duration = 0.006 seconds
DEBUG   - DEVICE: - PASS - [flash_driver.test_read_unaligned_address] duration = 0.015 seconds
DEBUG   - DEVICE: 
DEBUG   - DEVICE: ------ TESTSUITE SUMMARY END ------
DEBUG   - DEVICE: 
DEBUG   - DEVICE: ===================================================================
DEBUG   - DEVICE: RunID: 0d9c445fa65192ceedf1c3a6669fee91
DEBUG   - DEVICE: PROJECT EXECUTION SUCCESSFUL
DEBUG   - Expected suite names:['flash_driver']
DEBUG   - Detected suite names:['flash_driver', 'flash_driver', 'flash_driver', 'flash_driver', 'flash_driver', 'flash_driver']
DEBUG   - Release DUT:frdm_k64f, Id:02400b0177384e45002070016f01001ac401000197969900, counter:1, failures:0
DEBUG   - run status: frdm_k64f/tests/drivers/flash/common/drivers.flash.common.default passed
INFO    - 14/14 frdm_k64f                 tests/drivers/flash/common/drivers.flash.common.default PASSED (device: 02400b0177384e45002070016f01001ac401000197969900, 4.631s)

Fix was merged. Please verify whether issue is still observed.

No reproductive on ITE and FRDM. I close it.

zephyrproject-rtos / zephyr

test: drivers.flash.common.default: fails on frdm and ite. #77331

if !defined(TEST_AREA)