search

prplfoundation / prplMesh

This repository moved to https://gitlab.com/prpl-foundation/prplmesh/prplMesh
Other
65 stars 32 forks source link

PPM-251 tests/run_bf.sh must return non-zero when tests fail #1556

Closed aciliketcap closed 4 years ago

aciliketcap commented 4 years ago

We tell bft to output test results in a JSON file and then we read that file to determine if any tests failed.

Previously bft was run with "exec", probably to have the script exit with whatever exit code bft returns. I had to remove it of course but that behavior is preserved and we return non-zero exit code when bft itself fails.

I think it is a nice addition to have boardfarm's test results in a file, may be we can use that information in other places in CI.

I used jq to parse the JSON file. I could have grepped and parsed with regular shell utils but I think jq is great to deal with JSON data and it is a must have for any sys-admin in these times.

aciliketcap commented 4 years ago

This PR depends on https://github.com/prplfoundation/prplMesh/pull/1557 since CI tests will continue failing until that PR is merged.

ghost commented 4 years ago

@MergifyIO rebase

mergify[bot] commented 4 years ago

Command rebase: success

aciliketcap commented 4 years ago

I wanted the CI job to return the failed test count but as far as I investigated GitLab CI job exits with 1 when the script in the job exits with non-zero value. So we exit with 1 when there are failed tests and print the number of failed tests at the bottom of the log. Another issue was "[Errno 2] No such file or directory: './results/test_results.json'" message in logs but it is printed by boardfarm and is not really related to our scripts or tests so I am not doing anything about it.