Add info to rejected jobs list in `submission.json` (new)

[pieqq]

Description

Developments in Checkbox and C3 require more information being shared between the two.

C3 uses the submission.json file generated by Checkbox when creating the submission archive. This JSON file contains a rejected-jobs section that, until now, only contained the full_id of the jobs being de-selected.

With this PR, the exported submission.json includes more information in the rejected-jobs array, similarly to what results contain. This includes partial id and full id, plugin (which allows us to know what kind of job this is, including attachment, resource, etc.), and effective category and certification status (blocker or non-blocker).

Resolved issues

https://warthogs.atlassian.net/browse/CHECKBOX-1597

Documentation

Tests

Ran the following test with the version in main and the version in this branch:

1. Run checkbox-cli
2. Select the 24.04 desktop test plan
3. Deselect all the jobs (by pressing D)
4. Run the tests

Compare the number of rejected jobs in both cases to make sure we have the same outcome:

$ jq '.["rejected-jobs"] | length' submission_before_patch.json
1217
$ jq '.["rejected-jobs"] | length' submission_after_patch.json
1217

Here is a sample submission JSON file from a run on my laptop: submission_after_effective.json

Note that, exactly like jobs that have been run (present in results section), the jobs in rejected-jobs include the effective certification_status and category. For instance, the camera/detect job is marked as a cert-blocker:

        {
            "id": "camera/detect",
            "full_id": "com.canonical.certification::camera/detect",
            "name": "This Automated test attempts to detect a camera.",
            "plugin": "shell",
            "certification_status": "blocker",
            "category": "",
            "category_id": "com.canonical.plainbox::camera",
            "template_id": null
        },

The rejected-jobs also include attachment and resource jobs:

        {
            "id": "camera/multiple-resolution-images-attachment_video0",
            "full_id": "com.canonical.certification::camera/multiple-resolution-images-attachment_video0",
            "name": "Attach an image from the multiple resolution images test for Integrated_Webcam_FHD__Integrat",
            "plugin": "attachment",
            "certification_status": "non-blocker",
            "category": "",
            "category_id": "com.canonical.plainbox::camera",
            "template_id": "com.canonical.certification::camera/multiple-resolution-images-attachment_name"
        },
...
        {
            "id": "cdimage",
            "full_id": "com.canonical.certification::cdimage",
            "name": "Collect information about installation media (casper)",
            "plugin": "resource",
            "certification_status": "non-blocker",
            "category": "Gathers information about the DUT",
            "category_id": "com.canonical.certification::information_gathering",
            "template_id": null
        },

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 47.59%. Comparing base (6a3e3c1) to head (99ee45a). Report is 13 commits behind head on main.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #1520 +/- ## ========================================== + Coverage 47.56% 47.59% +0.02% ========================================== Files 369 369 Lines 39596 39596 Branches 6691 6691 ========================================== + Hits 18833 18844 +11 + Misses 20051 20040 -11 Partials 712 712 ``` | [Flag](https://app.codecov.io/gh/canonical/checkbox/pull/1520/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=canonical) | Coverage Δ | | |---|---|---| | [checkbox-ng](https://app.codecov.io/gh/canonical/checkbox/pull/1520/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=canonical) | `68.21% <ø> (+0.07%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=canonical#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[andrejvelichkovski]

Just verified that C3 doesn't parse this list to any more details. I think this will be a useful addition to help us validate the testing coverage that wont break C3 compatibility.

[pieqq]

Turning this into draft again as I think there is a way to export effective category and effective certification status, which could be helpful.

[pieqq]

This PR is ready for review.

[andrejvelichkovski]

The changes look good to me! Are we now also adding the plugin field to each test result, perhaps we can parse this into C3 as well?

[pieqq]

The changes look good to me! Are we now also adding the plugin field to each test result, perhaps we can parse this into C3 as well?

The problem is we have a weird mapping between Checkbox and C3. In Checkbox,, a job can have the following plugin options:

• manual
• shell
• user-interact
• user-interact-verify
• attachment
• resource

I think in C3, this is emulated by the type field in Test model, which concatenate this (to be confirmed).

Similarly, in C3, the status of a TestResult is narrowed down to skip/fail/pass, whereas in Checkbox there are other possibilities (crashed, not-supported, etc.). I don't know why the raw information is not sent to C3, and C3 then has the logic to do any aggregation it would see fit.

But yes, we could start saving this info in C3 as well.