Improvements for handling tests with "harness" in Zephyr testsuite

[pfalcon]

We have a number of tests in zephyr-upstream which should not be submitted to LAVA [currently], as they require special board setup/contraptions. This ticket is intended to be a set of notes how to improve detection of such tests based on Zephyr (sanitycheck) metadata. In particular, it would be expected that such tests should have suitable value of "harness" in sample.yaml/test.yaml.

[pfalcon]

The first culprit is tests/drivers/spi/spi_loopback. That systematically fails on all boards we have in LAVA. Let's look at its sample.yaml:

tests:
  drivers.spi:
    depends_on: spi
    tags: drivers spi
    harness: loopback

harness: loopback, d'oh. What kind of loopback - network loopback, gpio loopback, SPI loopback? How testing infrastructure would know that? So, harness: loopback isn't specific enough (at least on its own).

For example, let's consider how we could run such a test in LAVA. We would have one of boards to have a loopback connection on SPI, another board - on GPIO, etc. We'd have corresponding tags on these boards, e.g. boards with SPI loopback would have spi-loopback, GPIO - gpio-loopback, etc. (Again, the point is that these may be different boards). Then we'd need to map sanitycheck harness to such board tags. And currently, sanitycheck harness is not specific enough to perfmorm such mapping. Of course, we could try to make leaps of faith, e.g. if a test has depends_on: spi and harness: loopback, then it means that it needs SPI loopback, but I think such "artificial intelligence" will only lead to confusion and errors - everything should be clear and unambiguous in the first place.

In the meantime, we apparently just need to skip tests with harness: loopback, given that we don't have loopback contraptions for any of our test boards anyway.

I however grepped other test.yaml's for "loopback", to see how pervasive the problem is, and found just:

tests/drivers/gpio/gpio_basic_api:

tests:
  drivers.gpio.2pin:
    tags: drivers gpio
    depends_on: gpio
    min_flash: 34
    filter: dt_compat_enabled("test,gpio_basic_api")
    harness: ztest
    harness_config:
      fixture: gpio_loopback

2 obvious points based on just these 2 cases:

1. The 2nd test is indeed provides suitable level of specificity with gpio_loopback.
2. But the way to 2 tests specified is rather inconsistent:

    harness: loopback

vs

    harness: ztest
    harness_config:
      fixture: gpio_loopback

harness: ztest?? What would that mean?

[pfalcon]

@vanti: cc: you on this ticket too.

[vanti]

Currently we are excluding tests that require special harnesses via the "excluded_tests" list in zephyr-upstream/submit_for_testing.py. So one way would be to just add any such test to the list, as I previously did in https://review.linaro.org/c/ci/job/configs/+/35151. But it is likely the list is incomplete and we will encounter more of these over time as new tests are added, etc. So having a way to deal with harnesses would indeed be helpful.

I have noticed if I run sanitycheck today and have it directly run the tests on cc1352r1_launchxl, the built tests requiring harnesses (other than console) are automatically skipped. So it can be a stop-gap solution if we were to do the same in the test script, just to match up with sanitycheck's behavior.

Beyond that is as you said, there would need to be a way to specify which harnesses are supported on each board. Maybe this can be done by providing a set of tags in the board file and LAVA can determine whether a board is correspondingly tagged before dispatching the job to it? I have seen a similar feature in another test framework but haven't come across the equivalent in the LAVA dispatcher.

All that to say having proper "harness" values in Zephyr would be a good start, if we are thinking of going down this path.

[pfalcon]

@vanti: Thanks for comments, sorry for getting back to this only now. (I'm also aware that you may no longer work on these matters, in which case I, well, just record the ideas/notes, as the work on this goes in round-robin fashion with other tasks, so it's easy to forget the details).

Currently we are excluding tests that require special harnesses via the "excluded_tests" list in zephyr-upstream/submit_for_testing.py. So one way would be to just add any such test to the list, as I previously did in https://review.linaro.org/c/ci/job/configs/+/35151.

I see, I actually thought that the work you did on parsing test.yaml also included dealing with harnesses somehow.

I have noticed if I run sanitycheck today and have it directly run the tests on cc1352r1_launchxl, the built tests requiring harnesses (other than console) are automatically skipped.

Yep, I verified the same with frdm_k64f:

$ ./SANITYCHECK.sh -v -v --testcase-root tests/drivers/spi/spi_loopback/ --device-testing --device-serial /dev/ttyACM0 -p frdm_k64f
...
DEBUG   - frdm_k64f                 tests/drivers/spi/spi_loopback/drivers.spi         SKIPPED: Not runnable on device
INFO    - 0 test configurations selected, 1 configurations discarded due to filters.
...

So it can be a stop-gap solution if we were to do the same in the test script, just to match up with sanitycheck's behavior.

Yes, indeed, as we don't have boards with special harnesses in the lab, then to skip tests requiring such is the right way. And I would look into that, because adding such tests to manual blacklist indeed doesn't scale.