Enable per-test console verification in twister

[yperess]

Introduction

This idea builds on top of https://github.com/zephyrproject-rtos/zephyr/pull/46030. Once console verification is enabled for ztest, it would be beneficial to support specifying the harness config per test (since tests could be shuffled and test order is never guaranteed).

Problem description

Some ztest tests will want to verify what gets printed over UART. For example, console command might be printing critical user information after modifying state. A ztest might want to assert on the state but also verify that the right information was printed to the console.

Proposed change

Enable a per suite::test or suite::* harness configuration. This would allow us to assert on the console output when using ztest (even if the tests are shuffled).

Detailed RFC

Add support for the following under testcase.yaml:

tests:
  my.test:
    harness_configs:
      - type: one_line
        for_tests:
          - my_suite0::test0
          - my_suite1::*
        regex:
          - "Current value is: \d+"

Proposed change (Detailed)

Append to the ztest twister harness to parse the new harness_configs under each test. For each entry, generate a matcher that will be run on the logs, if the matcher fails, the test run should be marked as failed and the twister details for the failure appended to the test's logs

Alternatives

(1) I've considered adding this directly to ztest. The overhead of compiling a regex into the test binary and the additional buffers for forking the UART are too great.

(2) I've also considered keeping this inside the test (this is my close second choice. In this model, we'd add a new API to ztest that will print a specific message that will be handled by twister (similar to how we tell twister that a test is starting/stopping passed/failed/skipped). Twister would parse that line and generate the same matcher as above for the current test (if running in the test or before) or the suite (if running in the setup).

While this approach is nice in that it keeps the log check logic in the test it's messy and depends on twister to parse UART output even more. I believe that the proposed solution above is better.

[henrikbrixandersen]

Why not use the dummy shell directly within ztest test functions and verify the output produced? https://github.com/zephyrproject-rtos/zephyr/blob/main/include/zephyr/shell/shell_dummy.h

[yperess]

Why not use the dummy shell directly within ztest test functions and verify the output produced? https://github.com/zephyrproject-rtos/zephyr/blob/main/include/zephyr/shell/shell_dummy.h

Two reasons I kinda listed in the alternatives (through I could have done a better job looking at it now):

1. I don't want to compile the regex into the binary, it'll work for native_posix, but it's not a portable solution
2. For portability too, I don't want to add the extra buffer