canonical / checkbox

Checkbox is a testing framework used to validate device compatibility with Ubuntu Linux. It’s the testing tool developed for the purposes of the Ubuntu Certification program.
https://checkbox.readthedocs.io
GNU General Public License v3.0
33 stars 50 forks source link

New match feature (new) #1422

Closed Hook25 closed 3 months ago

Hook25 commented 3 months ago

Description

Re-running an entire test plan is very time consuming, sometimes we have to do it to only fix a handful of jobs and this stresses our lab and makes us waste time. One solution on the Checkbox side is to have a new configuration that allows us to run a few specific tests.

Similarly to how exclude= works, we should add a new section to the [test selection] configuration that does the opposite of exclude. Namely: Given a bootstrapped testplan run all tests that match the match or that the matching tests depend on

If this is the dependency graph, where the arrow means: depends on (namely H depends on G)

image

When the following match constraint is applied:

[test selection]
match=
  B
  E

The following jobs will be run:

The following wont be run:

Note: This feature must support creating a submission after running

Resolved issues

Fixes: CHECKBOX-1525

Documentation

Documentation added in the reference section along with other sections. No section was added to the tutorial as I feel this is a more "advanced" topic

Tests

New metabox tests and unit tests added

codecov[bot] commented 3 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 45.24%. Comparing base (4123afe) to head (cb882e3). Report is 122 commits behind head on main.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1422 +/- ## ========================================== + Coverage 45.19% 45.24% +0.04% ========================================== Files 367 367 Lines 39119 39125 +6 Branches 6613 6616 +3 ========================================== + Hits 17681 17701 +20 + Misses 20764 20751 -13 + Partials 674 673 -1 ``` | [Flag](https://app.codecov.io/gh/canonical/checkbox/pull/1422/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=canonical) | Coverage Δ | | |---|---|---| | [checkbox-ng](https://app.codecov.io/gh/canonical/checkbox/pull/1422/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=canonical) | `67.69% <100.00%> (+0.10%)` | :arrow_up: | | [checkbox-support](https://app.codecov.io/gh/canonical/checkbox/pull/1422/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=canonical) | `59.71% <ø> (+0.01%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=canonical#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.