canonical / checkbox

Checkbox is a testing framework used to validate device compatibility with Ubuntu Linux. It’s the testing tool developed for the purposes of the Ubuntu Certification program.
https://checkbox.readthedocs.io
GNU General Public License v3.0
33 stars 49 forks source link

Add `plugin` and `template_id` to resource and attachment results in submission.json (bugfix) #1596

Closed pieqq closed 1 week ago

pieqq commented 1 week ago

When using check-jsonschema using a recent version of a submission.json file against the JSON schema in the repo, the following errors show up:

check-jsonschema --schemafile submission-schema/schema.json /home/pieq/.local/share/checkbox-ng/submission_2024-11-12T13.48.37.650193/submission.json 
Schema validation errors were encountered.
(...)
  /home/pieq/.local/share/checkbox-ng/submission_2024-11-12T13.48.37.650193/submission.json::$.resource-results[0]: 'plugin' is a required property
  /home/pieq/.local/share/checkbox-ng/submission_2024-11-12T13.48.37.650193/submission.json::$.resource-results[0]: 'template_id' is a required property

This is because resource jobs stored in the resource-results do not contain these fields, as the plugin is assumed to be resource. Same for attachment-results.

However, these are all jobs and therefore the different results arrays should match in keys and values returned (it's a bit of a mystery why they were separated to begin with...).

Adding these properties to resource-results and attachment-results.

Description

Resolved issues

Documentation

JSON schema updated.

Tests

New generated submission with resource results:

    "resource-results": [
        {
            "id": "bluez-internal-bnep-tests",
            "full_id": "com.canonical.certification::bluez-internal-bnep-tests",
            "name": "Gather list of tests provided by bluez bnep test binary",
            "certification_status": "non-blocker",
            "category": "Uncategorised",
            "category_id": "com.canonical.plainbox::uncategorised",
            "status": "pass",
            "outcome": "pass",
            "comments": null,
            "io_log": "bash: line 1: bnep-tester: command not found\n",
            "type": "test",
            "project": "certification",
            "duration": 0.1284027099609375,
            "plugin": "resource",
            "template_id": null
        },
...

and attachment results:

    "attachment-results": [
        {
...
        },
        {
            "id": "lspci_standard_config_json",
            "full_id": "com.canonical.certification::lspci_standard_config_json",
            "name": "Attach PCI configuration space hex dump",
            "certification_status": "non-blocker",
            "category": "Informational tests",
            "category_id": "com.canonical.plainbox::info",
            "status": "pass",
            "outcome": "pass",
            "comments": null,
            "io_log": "\"0b08\"\n",
            "duration": 1.9314079284667969,
            "plugin": "attachment",
            "template_id": null
        },
codecov[bot] commented 1 week ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 48.03%. Comparing base (542c748) to head (4ca16eb). Report is 2 commits behind head on main.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1596 +/- ## ======================================= Coverage 48.03% 48.03% ======================================= Files 371 371 Lines 39852 39852 Branches 6734 6734 ======================================= Hits 19142 19142 Misses 19993 19993 Partials 717 717 ``` | [Flag](https://app.codecov.io/gh/canonical/checkbox/pull/1596/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=canonical) | Coverage Δ | | |---|---|---| | [checkbox-ng](https://app.codecov.io/gh/canonical/checkbox/pull/1596/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=canonical) | `68.65% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=canonical#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

pieqq commented 1 week ago

@Hook25 thanks for your feedback. I've modified the PR to do as suggested. Maybe one day we'll be able to merge those arrays back together! xD