Additional support for separate execution of Twister's "build only" and "test only"

[gopiotr]

At this moment Twister supports splitting test execution into two separate phases - "build only" and "test only". Thanks to this it is possible to build applications on first computer/server and run (flash) those applications on second one. For example:

On first computer (without connected nrf hardware) run command (--build-only):

./scripts/twister -p nrf52840dk_nrf52840 -T tests/kernel/queue -T tests/kernel/poll --build-only

Move all output to second computer (with connected nrf hardware) run command (--test-only):

./scripts/twister -p nrf52840dk_nrf52840 -T tests/kernel/queue -T tests/kernel/poll --device-testing --hardware-map map.yml --test-only

--subset and --test-only

Above approach doesn't work when we divide our building process several subsets, like this:

./scripts/twister -p nrf52840dk_nrf52840 -T tests/kernel/queue -T tests/kernel/poll --build-only --subset 1/2
./scripts/twister -p nrf52840dk_nrf52840 -T tests/kernel/queue -T tests/kernel/poll --build-only --subset 2/2

After apply those commands Twister will generate two several output folders with separate twister.json reports, what makes it impossible to use next Twister command with --test-only option, because it requires one output folder with one twister.json report after building phase. Now to solve this issue it is necessary to merge those two output folders and reports manually, to obtain one final twister.json report, ready to use by --test-only.

Dividing building into subsets is very significant, because it allow to simple parallelize building process by run it on several machines and thus get the "ready to flash" images faster, especially that building process doesn't require connected hardware setup, so it can be done in computing cloud.

And here I come with open question - is this reasonable to implement those merging output folders from subsets in Twister? If yes how it should be done? I think about adding argument (--source-dir) which could point the source folders, when --test-only option is used, like:

./scripts/twister -p nrf52840dk_nrf52840 -T tests/kernel/queue -T tests/kernel/poll --device-testing --hardware-map map.yml --test-only --source-dir output_from_subset_1 --source-dir output_from_subset_2

But with this solution some problems can arise:

1. What to do with built applications from source directories? Should they be all moved into common current "outdir"? If yes, it can be problematic when the amount of built images is huge - it can last long time and requires new disk space. If not it can be problematic to point Twister that images are out of current "outdir".
2. Where should be stored logs from devices - in source directories, or in new one where current "outdir" is created?

Test filtration and --test-only

Now when Twister is called with --test-only option it is impossible to add some test filtration - it is expected, that all tests from "build only" phase will be executed. So, for example it is impossible to do something like this:

# Build test for two platforms (`nrf52832` and `nrf52840`):
./scripts/twister -p nrf52dk_nrf52832 -p nrf52840dk_nrf52840 -T tests/kernel/queue --build-only
# Run test on only one platform (`nrf52840`):
./scripts/twister -p nrf52840dk_nrf52840 -T tests/kernel/queue --device-testing --hardware-map map.yml --test-only
# Despite indication only one platform test will be executed on both platforms (platform filtration will be ignored).

or:

# Build two tests (`queue` and `poll`):
./scripts/twister -p nrf52840dk_nrf52840 -T tests/kernel/queue -T tests/kernel/poll --build-only
# Run only one test (`queue`):
./scripts/twister -p nrf52840dk_nrf52840 -T tests/kernel/queue --device-testing --hardware-map map.yml --test-only
# Twister will rise error, because it will expect that tests `pool` tests is also in scope.

or:

# Build big number of tests (`kernel`):
./scripts/twister -p nrf52840dk_nrf52840 -T tests/kernel --build-only
# Run only several tests filtered by tag (`fpu`):
./scripts/twister -p nrf52840dk_nrf52840 -T tests/kernel --tag `fpu` --hardware-map map.yml --test-only
# Despite indication filtration by tag, all tests build in previous step will be executed (tag filtration will be ignored).

This filtration applied during run Twister with --test-only is very useful when Twister is used in distributed test system. For example, when we have one machine dedicated for building thousands of tests, and then we would like to run those tests on different machines with connected different hardware setups. In this case to do it effectively, it is necessary to limit somehow the the test scope dedicated for particular "test only" machines.

At this moment I don't have clear and simple proposition to meet above requirements. Some of them (like using tag filtration with --test-only) can be achieved with this PR: https://github.com/zephyrproject-rtos/zephyr/pull/49689 (thanks to this change: https://github.com/zephyrproject-rtos/zephyr/pull/49689/files#diff-0a0df38c6d70056ac2d7f06a7a16f8a148d1013d87fae6222d6a64323b6702cbL165-R166)

[zephyrbot]

Hi @nashif,

This issue, marked as an Enhancement, was opened a while ago and did not get any traction. It was just assigned to you based on the labels. If you don't consider yourself the right person to address this issue, please re-assing it to the right person.

Please take a moment to review if the issue is still relevant to the project. If it is, please provide feedback and direction on how to move forward. If it is not, has already been addressed, is a duplicate, or is no longer relevant, please close it with a short comment explaining the reason.

@gopiotr you are also encouraged to help moving this issue forward by providing additional information and confirming this request/issue is still relevant to you.

Thanks!