nk3 test: Add --deny-skipped option

robin-nitrokey commented 1 year ago

This patch adds an option to the nk3 test subcommand that causes one or more skipped tests to fail the test run. This can be used in production to make sure that all selected tests are actually executed.

Open questions:

Should --all automatically activate this flag?

Changes

Add --deny-skipped flag to nk3 test.

Checklist

[x] tested with Python3.9
[x] signed commits
[ ] updated documentation (e.g. parameter description, inline doc, docs.nitrokey)
[x] added labels

Test Environment and Execution

OS: Debian 11
device's model: NK3AM
device's firmware version: v1.3.1

jans23 commented 1 year ago

Should --all automatically activate this flag?

IMO yes, this is what I expect from it.

Why to introduce --deny-skipped additionally?

robin-nitrokey commented 1 year ago

--all decides which tests are executed, and --deny-skipped how the results are interpreted. The reasoning behind the skipped state is that we don’t want to indicate failures to end users that are not really a failure.

Nitrokey / pynitrokey