Open gopiotr opened 1 year ago
Hi @nashif,
This issue, marked as an Enhancement, was opened a while ago and did not get any traction. It was just assigned to you based on the labels. If you don't consider yourself the right person to address this issue, please re-assing it to the right person.
Please take a moment to review if the issue is still relevant to the project. If it is, please provide feedback and direction on how to move forward. If it is not, has already been addressed, is a duplicate, or is no longer relevant, please close it with a short comment explaining the reason.
@gopiotr you are also encouraged to help moving this issue forward by providing additional information and confirming this request/issue is still relevant to you.
Thanks!
At this moment Twister supports splitting test execution into two separate phases - "build only" and "test only". Thanks to this it is possible to build applications on first computer/server and run (flash) those applications on second one. For example:
On first computer (without connected
nrf
hardware) run command (--build-only
):Move all output to second computer (with connected
nrf
hardware) run command (--test-only
):--subset and --test-only
Above approach doesn't work when we divide our building process several subsets, like this:
After apply those commands Twister will generate two several output folders with separate
twister.json
reports, what makes it impossible to use next Twister command with--test-only
option, because it requires one output folder with onetwister.json
report after building phase. Now to solve this issue it is necessary to merge those two output folders and reports manually, to obtain one finaltwister.json
report, ready to use by--test-only
.Dividing building into subsets is very significant, because it allow to simple parallelize building process by run it on several machines and thus get the "ready to flash" images faster, especially that building process doesn't require connected hardware setup, so it can be done in computing cloud.
And here I come with open question - is this reasonable to implement those merging output folders from subsets in Twister? If yes how it should be done? I think about adding argument (
--source-dir
) which could point the source folders, when--test-only
option is used, like:But with this solution some problems can arise:
Test filtration and --test-only
Now when Twister is called with
--test-only
option it is impossible to add some test filtration - it is expected, that all tests from "build only" phase will be executed. So, for example it is impossible to do something like this:or:
or:
This filtration applied during run Twister with
--test-only
is very useful when Twister is used in distributed test system. For example, when we have one machine dedicated for building thousands of tests, and then we would like to run those tests on different machines with connected different hardware setups. In this case to do it effectively, it is necessary to limit somehow the the test scope dedicated for particular "test only" machines.At this moment I don't have clear and simple proposition to meet above requirements. Some of them (like using tag filtration with --test-only) can be achieved with this PR: https://github.com/zephyrproject-rtos/zephyr/pull/49689 (thanks to this change: https://github.com/zephyrproject-rtos/zephyr/pull/49689/files#diff-0a0df38c6d70056ac2d7f06a7a16f8a148d1013d87fae6222d6a64323b6702cbL165-R166)