canonical / checkbox

Checkbox is a testing framework used to validate device compatibility with Ubuntu Linux. It’s the testing tool developed for the purposes of the Ubuntu Certification program.
https://checkbox.readthedocs.io
GNU General Public License v3.0
33 stars 50 forks source link

Extend the wwan connection tests to support multiple cycle connection tests (new) #1534

Open stanley31huang opened 1 month ago

stanley31huang commented 1 month ago

Description

Extend the wwan connection tests to support multiple cycle connection tests, we found an issue that the LTE connection could not be established at second round.

As discussed internally with the QA team, we have all agreed that the default number of test runs for WWAN connections tests is 2.

Resolved issues

Here is the support ticket: 00379830.

Our SWE has identified this issue is related to the modem firmware and modem-manager snap, and they are trying to add a patch to modem-manager snap as a workaround.

Documentation

N/A

Tests

list bootstrapped results (with hacking resource):

(venv) stanley@stanley-ThinkPad-T490:~/Desktop/Git_Repos/checkbox-github/providers/base$ checkbox.checkbox-cli list-bootstrapped com.canonical.certification::wwan-automated
$PROVIDERPATH is defined, so following provider sources are ignored ['/home/stanley/.local/share/plainbox-providers-1', '/var/tmp/checkbox-providers-develop'] 
Using sideloaded provider: checkbox-provider-base, version 4.2.0.dev99 from /var/tmp/checkbox-providers/base
Skipped file: /var/tmp/checkbox-providers/base/bin/__pycache__/wwan_tests.cpython-38.pyc
Skipped file: /var/tmp/checkbox-providers/base/units/stress/suspend_cycles_reboot.md
com.canonical.certification::package
com.canonical.plainbox::manifest
com.canonical.certification::snap
com.canonical.certification::wwan/detect
com.canonical.certification::wwan/check-sim-present-honghai-modelc-testid-auto
com.canonical.certification::wwan/3gpp-scan-honghai-modelc-testid-auto
com.canonical.certification::wwan/gsm-connection-honghai-modelc-testid-auto
com.canonical.certification::wwan/gsm-reconnection-honghai-modelc-testid-auto
com.canonical.certification::sleep
com.canonical.certification::rtc
com.canonical.certification::suspend/suspend_advanced_auto
com.canonical.certification::after-suspend-wwan/3gpp-scan-honghai-modelc-testid-auto
codecov[bot] commented 1 month ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 47.76%. Comparing base (cd772da) to head (ab12ee7). Report is 3 commits behind head on main.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1534 +/- ## ========================================== + Coverage 47.72% 47.76% +0.04% ========================================== Files 370 370 Lines 39739 39738 -1 Branches 6719 6720 +1 ========================================== + Hits 18965 18982 +17 + Misses 20059 20041 -18 Partials 715 715 ``` | [Flag](https://app.codecov.io/gh/canonical/checkbox/pull/1534/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=canonical) | Coverage Δ | | |---|---|---| | [provider-base](https://app.codecov.io/gh/canonical/checkbox/pull/1534/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=canonical) | `24.77% <100.00%> (+0.10%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=canonical#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

stanley31huang commented 1 month ago

The PR is ready, but I did not test it on real machine.