canonical / checkbox

Checkbox is a testing framework used to validate device compatibility with Ubuntu Linux. It’s the testing tool developed for the purposes of the Ubuntu Certification program.
https://checkbox.readthedocs.io
GNU General Public License v3.0
33 stars 49 forks source link

Update the list of rejected jobs if match function is used (bugfix) #1575

Closed pieqq closed 1 week ago

pieqq commented 2 weeks ago

Description

The jobs that are being pruned by the match function should be added to the list of rejected jobs, otherwise the metadata generated at the end of a test run is incorrect.

Because the list of rejected jobs is initially empty but can be altered in several places (run_alternate_selection() and finish_bootstrap()), it is updated in these functions instead of being replaced by another list.

The metadata that states if a test plan has been modified (custom_joblist) should also be set to True in these cases.

Resolved issues

Fix #1568

Documentation

Tests

Unit tests have been updated.

The following end to end tests have been run:

Normal call without use of match

  1. Run checkbox-cli
  2. Select camera-cert-automated test plan
  3. In the Camera category, de-select everything except camera/detect and run the tests

The generated submission.json shows:

{
    "title": "session title",
    "testplan_id": "com.canonical.certification::camera-cert-automated",
    "custom_joblist": true,
    (...)
    "rejected-jobs": [
        {
            <the de-selected tests are here>
        }

Normal call with use of match

The following launcher is used:

[test plan]
unit = com.canonical.certification::camera-cert-automated
forced = yes

[test selection]
match = com.canonical.certification::camera/detect
  1. Run checkbox-cli
  2. Notice that in the test selection screen, in the Camera cateogory, only camera/detect is presented
  3. De-select it, and run the tests

The submission.json generated looks like this:

{
    "title": "session title",
    "testplan_id": "com.canonical.certification::camera-cert-automated",
    "custom_joblist": true,
    "results": [
    ],
    (...)
    "rejected-jobs": [
        {
        <all the camera tests are listed here, including camera/detect>
        }

Automated run with the use of match

The following launcher is used:

[test plan]
unit = com.canonical.certification::camera-cert-automated
forced = yes

[test selection]
forced = yes
match = com.canonical.certification::camera/detect

The generated submission.json looks like this:

{
    "title": "session title",
    "testplan_id": "com.canonical.certification::camera-cert-automated",
    "custom_joblist": true,
    "results": [
        {
            "id": "camera/detect",
            "full_id": "com.canonical.certification::camera/detect",
            "name": "This Automated test attempts to detect a camera.",
            "certification_status": "blocker",
            "category": "Camera tests",
            "category_id": "com.canonical.plainbox::camera",
            "status": "pass",
            "outcome": "pass",
            "comments": null,
            "io_log": "/dev/video0: OK\n    name   : Integrated_Webcam_FHD: Integrat\n    driver : uvcvideo\n    version: 6.8.12\n    flags  : 0x84a00001 [ CAPTURE STREAMING ]\n    Format: MJPG (Motion-JPEG)\n    Resolutions: 1920x1080,1280x960,1280x720,640x480,640x360\n    Format: YUYV (YUYV 4:2:2)\n    Resolutions: 1920x1080,1280x720,640x480,640x360,320x240,320x180,160x120\n\n/dev/video1: OK\n    name   : Integrated_Webcam_FHD: Integrat\n    driver : uvcvideo\n    version: 6.8.12\n    flags  : 0x84a00001 [ CAPTURE STREAMING ]\n    Format: YUYV (YUYV)\n    Resolutions: 640x480\n\n/dev/video2: OK\n    name   : Integrated_Webcam_FHD: Integrat\n    driver : uvcvideo\n    version: 6.8.12\n    flags  : 0x84a00001 [ CAPTURE STREAMING ]\n    Format: GREY (8-bit Greyscale)\n    Resolutions: 640x360\n\n/dev/video3: OK\n    name   : Integrated_Webcam_FHD: Integrat\n    driver : uvcvideo\n    version: 6.8.12\n    flags  : 0x84a00001 [ CAPTURE STREAMING ]\n    Format: YUYV (YUYV)\n    Resolutions: 640x480\n\n",
            "type": "test",
            "project": "certification",
            "duration": 2.1785783767700195,
            "plugin": "shell",
            "template_id": null
        }
    ],
    (...)
    "rejected-jobs": [
        <every camera-related test is listed here, EXCEPT camera/detect>
        ]
codecov[bot] commented 2 weeks ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 48.00%. Comparing base (9c994f1) to head (c087555). Report is 2 commits behind head on main.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1575 +/- ## ======================================= Coverage 48.00% 48.00% ======================================= Files 371 371 Lines 39833 39837 +4 Branches 6730 6733 +3 ======================================= + Hits 19121 19125 +4 Misses 19994 19994 Partials 718 718 ``` | [Flag](https://app.codecov.io/gh/canonical/checkbox/pull/1575/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=canonical) | Coverage Δ | | |---|---|---| | [checkbox-ng](https://app.codecov.io/gh/canonical/checkbox/pull/1575/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=canonical) | `68.60% <100.00%> (+<0.01%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=canonical#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.