golioth / golioth-zephyr-sdk

Golioth SDK For Zephyr
https://www.golioth.io
Apache License 2.0
67 stars 19 forks source link

samples: add runtime twister test support #287

Closed mniestroj closed 1 year ago

mniestroj commented 1 year ago

This adds support for running twister to build and runtime test samples on real hardware. Both firmware builds as well as pytest scripts executed by twister are using environment variables:

github-actions[bot] commented 1 year ago

Visit the preview URL for this PR (updated for commit 033fa17):

https://golioth-zephyr-sdk-doxygen-dev--pr287-samples-with-gol-gkepu35k.web.app

(expires Mon, 26 Sep 2022 20:32:31 GMT)

🔥 via Firebase Hosting GitHub Action 🌎

mniestroj commented 1 year ago

For the future:

There's a lot of overlap with the existing hardware in the loop tests which utilize GitHub Actions and samples/test. Ideally, there'd be just a single solution for hardware-based testing, and it seems like this twister solution is further along in terms of the sample test coverage.

I agree about the overlap. However each solution has some advantages and disadvantages compared to the other. For example, this pytest+twister approach utilizes samples that are user/developer oriented. I mean, they are written in a way to show developer APIs and features in the SDK. On the other hand samples/test/ allows to split building from testing. Additionally it has a basic interaction with the board over serial port, which is not possible with the pytest harness in twister, at least now.

It will cut down on maintenance if we can stick with one solution.

Good point. At least now, I have not integrated twister tests with GitLab or GitHub CI (apart from a one-shot try with qemu_x86). So keeping samples/test still makes sense. Regarding "one solution" I think that for now we can have those two solutions for some time and learn which looks better. In general I think that going with twister would be the way to go, but one major milestone that would make it closer to that is allowing those pytest scripts to interact over shell. That way we could cover samples/dfu/ without disabling "settings" backend and also automatically test samples/settings/.

Along with this, I think we'll need some kind of documentation for how to setup GitLab and environment similar to what you have, so others can run the tests if needed.

Currently this PR description lists all environment variables that need to be configured to run twister. Then, running twister is just issuing:

zephyr/scripts/twister -p mimxrt1060_evkb -T modules/lib/golioth --device-testing --hardware-map hardware-map.yaml -j 1

for HW platforms or:

zephyr/scripts/twister -p qemu_x86 -T modules/lib/golioth/samples -j 1

but I agree, that putting this somewhere in documentation in the repo is the way to go.