Some status of cases are still be non-blocker even apply blocker to .* to test plan

baconYao commented 2 months ago

Bug Description

In PR#1206 Certification team made every test job in client-cert-iot-.* test plans as cert blocker according to a decision made a few years ago which we have been using to judge the bug importance since then (All checkbox jobs being executed for the IoT project is a cert blocker)

Therefore, QA team started implementing this rule in each project. However, we observed that the status is still be non-blocker of some cases who are from base provider.

sideload submission on genio-1200 desktop image: https://certification.canonical.com/hardware/202307-31864/submission/395052/test-results/
sideload submission on baytown: https://certification.canonical.com/hardware/202401-33430/submission/394986/test-results/

Here is the simple PR to implement this feature. What I did is like below (nest the cert test plan and have certification_status_overrides field be defined):

bootstrap_include:
    baoshan_device_soc
include:
nested_part:
    .
    .
    com.canonical.certification::client-cert-desktop-22-04-automated
    .
    .
certification_status_overrides:
    apply blocker to .*
exclude:
    .
    .
    .

To Reproduce

Have a test plan that nests the com.canonical.certification::client-cert-desktop-22-04-automated
Add certification_status_overrides field
Sideload this test plan
Checkbox remote runs this auto plan and choose the miscellaneous and info sets then executing it.

Environment

OS: Classic Desktop / Server / Ubuntu Core
Checkbox type: Snap

Relevant log output

No response

Additional context

No response

syncronize-issues-to-jira[bot] commented 2 months ago

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/CHECKBOX-1577.

This message was autogenerated

pieqq commented 1 month ago

If everything is in the same namespace, the issue does not happen.

However, if the override is applied to a test plan in one of the provider (with one namespace), and that test plan is then nested in another provider (with another namespace), the override is not applied.

→ Issue confirmed.

pieqq commented 1 month ago

A workaround for you:

use

certification_status_overrides:
    apply blocker to .*::.*

instead of

certification_status_overrides:
    apply blocker to .*

The issue lies in the testplan unit where Checkbox adds the namespace to everything that does not have a namespace.

baconYao commented 1 week ago

Hi @pieqq I've tried the method you provided but still has non-blocker cases.

certification_status_overrides:
    apply blocker to .*::.*

The Submission-A use apply blocker to .*::.*

The Submission-B use apply blocker to .*

Compare Submission-A and Submission-B, we can find the miscellanea/submission-resources case has been blocker but info/systemd-analyze and info/systemd-analyze-critical-chain cases are still non-blocker

canonical / checkbox