[RFC] Enable executability of upstream tests on boards that don't have required functionality through usage of custom shields

[PerMac]

Introduction

The executability of a test can depend on certain features that have to be available on a target. It is a common practice to expand board functionality by physically attaching a shield. Information about such shields often has to be propagated to the build-system and/or twister. This RFC proposes a generic solution for twister that would enable to expand on-target test setups with custom shields. The most important point is that such shields won't have to be defined in the upstream. Instead, users will be able to have the code for their shields implemented locally/downstream, and twister will be able to properly pick it up and use it if a shield-provided feature is required by a test (based on matching "depends_on/supported" keys).

Problem description

There is no clear way how a user can execute upstream tests on their targets if a test requires a feature that is not available by default on such target. A user could attach a shield to their target. However, it becomes problematic if it also requires applying an overlay (e.g. with -DSHIELD) and such shield is not available in the upstream. E.g. of such a conundrum can be seen here https://github.com/zephyrproject-rtos/zephyr/pull/71997

Proposed change

A new section will be available in a user-defined hardware map yaml file: shields. This section will be a simple dictionary in yaml. It will describe shields used in a user's setup and the functionality they support. A user will also mark in their hardware map if a device has a shield connected. This can be done in a similar way fixtures are added. E.g:

- connected: true
  id: 000683078806
  platform: nrf52840dk/nrf52840
  product: J-Link
  runner: jlink
  serial: /dev/ttyACM0
  shield: my_shield

- shields:
  - name: my_shield
    supports:
      - spi
      - i2c
      - i3c
      - can

(current status) When evaluating if a given test can be executed on a user's hardware twister checks if depends_on entries from test yaml match supported entries from board yaml. (proposal) If a board has a shield connected its twister's representation will also have a shield_supported list with entries from a given shield description (from the user's hardware map). If the required feature is found among shield_supportedan extra -DSHIELD=<shield_name> will be added to a build for this test+board configuration (hence only used when needed). Since SHIELDs can be defined downstream, multiple users will be able to run the same test with their shields, as long as "depends_on/supported" keywords match.

Detailed RFC

It seems the required changes are simple enough that I could prepare a PR with such implementation in the same amount of time as putting all the details of the implementation in this RFC. Also, I don't see anything more than already described. If only the above description sounds reasonable I will prepare a PR with the implementation.

Proposed change (Detailed)

Ditto

Dependencies

Such twister extension could also be in line with #68400. Having configurations for a generally available shield is a good idea, however, such configuration won't serve everyone. Some users/vendors might prefer/need to use their own shields (e.g. since they have such shields already available and in use or a board doesn't have an arduino pin layout compatibility)

Concerns and Unresolved Questions

"Depends_on" and "supported" functionality in twister is based purely on string matching. There are no underlying definitions or requirements for used keywords. There is also no validation of their usage other than a reviewer's keen eye. However, I don't see any alternative that would allow us to go forward. Filtration based on cmake (dt and kconfig) requires precompiling sources and then it is already too late to decide if a SHIELD should be applied. "Depends_on"- "supported" matching mechanism is already in use and I propose only an extension to it.

Alternatives

Unknown

[PerMac]

CC @erwango @carlescufi @nashif @teburd @nordic-segl @gmarull @hakehuang @henrikbrixandersen

[henrikbrixandersen]

I tried to add a similar feature a while back, but got stranded - the issue, I encountered, was the Twister builds one binary per test platform and uses that on any platform of that type available.

My initial work, including a comment on where I got stuck, is available here: https://github.com/vestas-wind-systems/zephyr/tree/twister_hwmap_shields

[hakehuang]

also consider the current fixtures #20340, we may need to use the same approach

[henrikbrixandersen]

also consider the fcurrent ixtures #20340, we may need to use the same approach

Please elaborate a bit on this comment?

[hakehuang]

also consider the current fixtures #20340, we may need to use the same approach

Please elaborate a bit on this comment?

sorry, updated.