Closed MrKevinWeiss closed 3 years ago
It seems like there is still a reset problem but maybe outside the scope of this PR.
this PR has grown. The build system has been broken for a while it seems. Hopefully this will use FEATURES_REQUIRED
so one can make clean
without errors...
Though having build failed or skipped xunit generation would probably be better to include in the make system it is not trivial for me to figure out as we are using the FEATURES_*
resolution in RIOT and I don't want to change riot. For the moment it can simply exist in the Jenkins job, this means that it will only get generated from the CI and reproducing locally is not automatic.
Since this is a fairly simple take to do manually maybe it is fine.
I will clean up this PR and maybe we can get it in since it contains fixes and cleanups that are needed for other ongoing work.
Results are looking as expected.
@leandrolanzieri can we merge?
Hmmm something is wrong with the logs...
Hmmm I don't think the issue of the report is with PR but we should wait until it is figured out. ping @ozfox
Figured it out, it was just not rebasing the frontend. Here are the results.
We have PHiLIP.Write Reg expected Success but received Error
and spi (3.2) could not toggle to low, Check wiring
This overrides the good old classics such as
Resolving variable '${RESULT['result']}' failed
or'SUCCESS' does not contain 'FAIL'
or who could forget the great0 != 1
with a bit better messages that evaluate variables, print proper sentences, and explain what was expected s received.A test to show some of these messages is also added to
dist/tools/output_to_xunit
and can be checked withmake robot-clean robot-test
.EDIT: It turns out many other issues were discovered when working on this PR, so here it is.
Add a test to show failure messages without requiring hardware
Adding the
03__error_msg.robot
test. To check one can simply usemake -C dist/tools/output_to_xunit/ robot-clean robot-test
.Here we can see nice failure messages:
Better error messages/cleanup/fixes GPIO tests
Some non-common error messages are improved. Utilizing the
Skip If
keyword in Robot Framework V4, this also also refactoring some redundant functions.Build system fixes
It turns out there are some issues with using error. The build system now uses
FEATURE_REQUIRED
and FEATURES_PROVIDED` to enforce proper wiring.This is also useful for the CI to determine what builds actually fail and what should be failing/skipped
Scripting fixes
A small fix in the
output_to_xunit/output_to_xunit.py
script that prevents exceptions from being thrown if a benchmark test fails and doesn't contain an expected key.