mozilla / webcompat-team-okrs

These are quarterly team level OKR projects for the WebCompat Tools team
Mozilla Public License 2.0
11 stars 7 forks source link

2023H2 - Site Intervension Release #274

Closed denschub closed 10 months ago

denschub commented 1 year ago
softvision-raul-bucata commented 1 year ago

[Done] 1stManual run (H2 -2023) - QA Verification progress available here: https://docs.google.com/spreadsheets/d/1b1K-Zk35uLCk_6n2I-aON-4yc0bkZWQQPme1Em59QOQ/edit#gid=683163181

Items to be removed:

1840593 Remove UA Overrides/Interventions for cnki.net 1840595 Remove UA Overrides/Interventions for clalit.co.il

softvision-raul-bucata commented 1 year ago

Next QA run: 13.07.2023

wisniewskit commented 1 year ago

In the interests of speeding up our testing, I will run the automated tests myself for the v117 cycle, and update this comment as I find/file issues (so that Raul and Calin may focus on testing the interventions which cannot be automatically tested):

In addition, the following automated tests need to be revised (their related interventions are still needed and working, but the tests themselves have issues):

The following interventions were removed, but the test is still present:

And we should be able to add tests for the following interventions this cycle:

softvision-raul-bucata commented 1 year ago

[Done] 2nd Manual run (H2 -2023) - QA Verification progress available here: https://docs.google.com/spreadsheets/d/1b1K-Zk35uLCk_6n2I-aON-4yc0bkZWQQPme1Em59QOQ/edit#gid=235857233

Items to be removed:

softvision-raul-bucata commented 1 year ago

Next Run: 10.08.2023

softvision-raul-bucata commented 1 year ago

[Done] 3rd Manual run (H2 -2023) - QA Verification progress available here: https://docs.google.com/spreadsheets/d/1b1K-Zk35uLCk_6n2I-aON-4yc0bkZWQQPme1Em59QOQ/edit#gid=262782010

Items to be removed:

softvision-raul-bucata commented 1 year ago

[Done] 3rd Automation run (H2- 2023) I've also executed the Automation Test Suite and below are the results (VPN active pointing to the USA):

First run: Ran 225 checks (225 tests) Expected results: 92 Skipped: 123 tests (Android tests) Unexpected results: 10 test: 10 (10 fail)

Unexpected results: 10( 10 fails) FAILED tests/test_1631811_datastudio.py::test_enabled[with_interventions] - assert not True FAILED tests/test_1650317_livescience_com.py::test_enabled[with_interventions] - asyncio.exceptions.CancelledError FAILED tests/test_1799994_vivobarefoot_com.py::test_enabled[with_interventions] - webdriver.error.ScriptTimeoutException: script timeout (500): Timed out after 30000 ms FAILED tests/test_1800936_cov19ent_kdca_go_kr.py::test_enabled[with_interventions] - webdriver.bidi.error.UnknownErrorException: unknown error (Error: NS_ERROR_UNKNOWN_HOST) FAILED tests/test_1836140_indices_iriworldwide_com.py::test_enabled[with_interventions] - webdriver.error.NoSuchElementException: no such element (404) FAILED tests/test_1719859_saxoinvestor.py::test_disabled[without_interventions] - assert False FAILEDtests/test_1799994_vivobarefoot_com.py::test_disabled[without_interventions] - webdriver.error.ScriptTimeoutException: script timeout (500): Timed out after 30000 ms FAILED tests/test_1800936_cov19ent_kdca_go_kr.py::test_disabled[without_interventions] - webdriver.bidi.error.UnknownErrorException: unknown error (Error: NS_ERROR_UNKNOWN_HOST) FAILED tests/test_1836140_indices_iriworldwide_com.py::test_disabled[without_interventions] - asyncio.exceptions.CancelledError FAILED tests/test_1650317_livescience_com.py::test_disabled[without_interventions] - asyncio.exceptions.CancelledError

Log file:

desktop 10thAugust2023.zip

Android Intervention can not temporarily run in our test machines, due to various errors when attempting the start the emulator, or when running the test in a device, due to the device not being able to connect to pages.

softvision-raul-bucata commented 1 year ago

Next Run: 07.09.2023

softvision-raul-bucata commented 1 year ago

[Done] 4th Manual run (H2 -2023) - QA Verification progress available here: https://docs.google.com/spreadsheets/d/1b1K-Zk35uLCk_6n2I-aON-4yc0bkZWQQPme1Em59QOQ/edit#gid=582536337

softvision-raul-bucata commented 1 year ago

[Done] 4th Automation run (H2- 2023) I've also executed the Automation Test Suite and below are the results (VPN active pointing to the USA):

Desktop

First run:

Ran 233 checks (233 tests) Expected results: 94 Skipped: 126 tests (Android) Unexpected results: 13 test: 13 (13 fail)

Unexpected results: 13( 13 fails)

FAIL tests/test_1575000_lloydsbank.py::test_enabled[with_interventions] - asyncio.exceptions.CancelledError FAIL tests/test_1650317_livescience_com.py::test_enabled[with_interventions] - asyncio.exceptions.CancelledError FAIL tests/test_1799994_vivobarefoot_com.py::test_enabled[with_interventions] - webdriver.error.ScriptTimeoutException: script timeout (500): Timed out after 30000 msscript timeout (500): Timed out after 30000 ms FAIL tests/test_1800936_cov19ent_kdca_go_kr.py::test_enabled[with_interventions] - webdriver.error.NoSuchElementException: no such element (404) FAIL tests/test_1830813_onstove_com.py::test_enabled[with_interventions] - assert 0 == 120 FAIL tests/test_1836140_indices_iriworldwide_com.py::test_enabled[with_interventions] - webdriver.error.NoSuchElementException: no such element (404) FAIL tests/test_1849058_nicochannel_jp.py::test_enabled[with_interventions] - webdriver.error.NoSuchElementException: no such element (404) FAIL tests/test_1631811_datastudio.py::test_disabled[without_interventions] - assert False FAIL tests/test_1650317_livescience_com.py::test_disabled[without_interventions] - asyncio.exceptions.CancelledError FAIL tests/test_1719859_saxoinvestor.py::test_disabled[without_interventions] - assert False FAIL tests/test_1799994_vivobarefoot_com.py::test_disabled[without_interventions] - webdriver.error.ScriptTimeoutException: script timeout (500): Timed out after 30000 ms FAIL tests/test_1800936_cov19ent_kdca_go_kr.py::test_disabled[without_interventions] - webdriver.error.NoSuchElementException: no such element (404) FAIL tests/test_1836140_indices_iriworldwide_com.py::test_disabled[without_interventions] - asyncio.exceptions.CancelledError

Log file: September2023Desktop.txt

Android

First run:

Ran 117 checks (117 tests) Expected results: 89 Skipped: 19 tests Unexpected results: 9 test: 9 (9 fail)

Unexpected results: 9 ( (FAIL)

FAILED testing\webcompat\interventions\tests\test_1605611_maps_google_com.py::test_enabled[with_interventions] FAILED testing\webcompat\interventions\tests\test_1722955_frontgate_com.py::test_enabled[with_interventions] FAILED testing\webcompat\interventions\tests\test_1800936_cov19ent_kdca_go_kr.py::test_enabled[with_interventions] FAILED testing\webcompat\interventions\tests\test_1830747_babbel_com.py::test_enabled[with_interventions] FAILED testing\webcompat\interventions\tests\test_1836140_indices_iriworldwide_com.py::test_enabled[with_interventions] FAILED testing\webcompat\interventions\tests\test_1836178_atracker_pro.py::test_enabled[with_interventions] FAILED testing\webcompat\interventions\tests\test_1848711_vio_com.py::test_enabled[with_interventions] FAILED testing\webcompat\interventions\tests\test_1849018_carefirst_com.py::test_enabled[with_interventions] FAILED testing\webcompat\interventions\tests\test_1849058_nicochannel_jp.py::test_enabled[with_interventions]

Skipped

SKIPPED [1] testing\webcompat\interventions\tests\test_1457335_histography.py:7: Bug #1457335 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1472075_bankofamerica.py:6: Bug #1472075 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1570108_steamcommunity.py:111: Bug #1570108 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1577519_stream_directv_com.py:22: Region-locked, cannot test. Try using a VPN set to USA. SKIPPED [1] testing\webcompat\interventions\tests\test_1582582_sling_com.py:9: Bug #1582582 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1610344_directv_com_co.py:44: Bug #1610344 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1631811_datastudio.py:29: Bug #1631811 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1644830_missingmail_usps.py:52: Bug #1644830 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1650317_livescience_com.py:29: Bug #1650317 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1753461_naver.py:9: Bug #1753461 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1778168_watch_antennaplus_gr.py:9: Bug #1778168 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1786404_business_help_royalmail_com.py:59: Bug #1786404 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1799968_samsung_com.py:8: Bug #1799968 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1799994_vivobarefoot_com.py:52: Bug #1799994 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1800000_www_honda_co_uk.py:42: Bug #1800000 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1830813_onstove_com.py:18: Bug #1830813 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1836109_watch_tonton_com_my.py:13: Bug #1836109 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1836116_www_slushy_com.py:7: Bug #1836116 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1848716_elal_com.py:7: Bug #1848716 skipped on platform (android)

Log file: September2023Android.txt

softvision-raul-bucata commented 1 year ago

Next Run: 5.10.2023

softvision-raul-bucata commented 1 year ago

[Done] 5th Manual run (H2 -2023) - QA Verification progress available here: https://docs.google.com/spreadsheets/d/1b1K-Zk35uLCk_6n2I-aON-4yc0bkZWQQPme1Em59QOQ/edit#gid=945381760

Items to be removed:

softvision-raul-bucata commented 1 year ago

[Done] 5th Automation run (H2- 2023) I've also executed the Automation Test Suite and below are the results (VPN active pointing to the USA):

Desktop

First run:

Ran 245 checks (245 tests) Expected results: 101 Skipped: 137 tests (Android) Unexpected results: 7 test: 7 (7 fail)

Unexpected results: 7( 7 fails)

FAIL tests/test_1650317_livescience_com.py::test_enabled[with_interventions] - asyncio.exceptions.CancelledError FAIL tests/test_1799994_vivobarefoot_com.py::test_enabled[with_interventions] - webdriver.error.ScriptTimeoutException: script timeout (500): Timed out after 30000 ms FAIL tests/test_1830813_onstove_com.py::test_enabled[with_interventions] - assert 0 == 120 FAIL tests/test_1836181_conference_amwell_com.py::test_enabled[with_interventions] - webdriver.error.NoSuchElementException: no such element (404) FAIL tests/test_1855088_125039_hrmis2_eghrmis_gov_my.py::test_enabled[with_interventions] - assert not True FAIL tests/test_1650317_livescience_com.py::test_disabled[without_interventions] - asyncio.exceptions.CancelledError FAIL tests/test_1799994_vivobarefoot_com.py::test_disabled[without_interventions] - webdriver.error.ScriptTimeoutException: script timeout (500): Timed out after 30000 ms

Log file: October2023Desktop.txt

Android

First run: Ran 245 checks (245 tests) Expected results: 199 Skipped: 37 tests Unexpected results: 9 test: 9 (9 fail)

Unexpected results: 9 ( (FAIL)

FAILED testing\webcompat\interventions\tests\test_1654877_preev_com.py::test_disabled[without_interventions] FAILED testing\webcompat\interventions\tests\test_1836140_indices_iriworldwide_com.py::test_disabled[without_interventions] FAIL tests/test_1836140_indices_iriworldwide_com.py::test_enabled[with_interventions] - webdriver.bidi.error.UnknownErrorException: unknown error (Error: NS_ERROR_NET_TIMEOUT) FAILED testing\webcompat\interventions\tests\test_1605611_maps_google_com.py::test_enabled[with_interventions] FAILED testing\webcompat\interventions\tests\test_1654877_preev_com.py::test_enabled[with_interventions] FAILED testing\webcompat\interventions\tests\test_1829126_otsuka_co_jp.py::test_enabled[with_interventions] FAILED testing\webcompat\interventions\tests\test_1836140_indices_iriworldwide_com.py::test_enabled[with_interventions] FAILED testing\webcompat\interventions\tests\test_1836181_conference_amwell_com.py::test_enabled[with_interventions] FAILED testing\webcompat\interventions\tests\test_1848711_vio_com.py::test_enabled[with_interventions] FAILED testing\webcompat\interventions\tests\test_1849018_carefirst_com.py::test_enabled[with_interventions]

Skipped: 19

SKIPPED [1] testing\webcompat\interventions\tests\test_1457335_histography.py:15: Bug #1457335 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1472075_bankofamerica.py:14: Bug #1472075 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1577519_stream_directv_com.py:22: Region-locked, cannot test. Try using a VPN set to USA. SKIPPED [1] testing\webcompat\interventions\tests\test_1582582_sling_com.py:21: Bug #1582582 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1610344_directv_com_co.py:51: Bug #1610344 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1631811_datastudio.py:36: Bug #1631811 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1644830_missingmail_usps.py:59: Bug #1644830 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1650317_livescience_com.py:36: Bug #1650317 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1753461_naver.py:17: Bug #1753461 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1778168_watch_antennaplus_gr.py:17: Bug #1778168 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1786404_business_help_royalmail_com.py:66: Bug #1786404 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1799968_samsung_com.py:16: Bug #1799968 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1799994_vivobarefoot_com.py:59: Bug #1799994 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1800000_www_honda_co_uk.py:49: Bug #1800000 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1830813_onstove_com.py:25: Bug #1830813 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1836109_watch_tonton_com_my.py:21: Bug #1836109 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1836116_www_slushy_com.py:15: Bug #1836116 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1848716_elal_com.py:16: Bug #1848716 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1457335_histography.py:7: Bug #1457335 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1472075_bankofamerica.py:6: Bug #1472075 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1570108_steamcommunity.py:111: Bug #1570108 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1577519_stream_directv_com.py:22: Region-locked, cannot test. Try using a VPN set to USA. SKIPPED [1] testing\webcompat\interventions\tests\test_1582582_sling_com.py:9: Bug #1582582 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1610344_directv_com_co.py:44: Bug #1610344 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1631811_datastudio.py:29: Bug #1631811 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1644830_missingmail_usps.py:52: Bug #1644830 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1650317_livescience_com.py:29: Bug #1650317 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1753461_naver.py:9: Bug #1753461 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1778168_watch_antennaplus_gr.py:9: Bug #1778168 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1786404_business_help_royalmail_com.py:59: Bug #1786404 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1799968_samsung_com.py:8: Bug #1799968 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1799994_vivobarefoot_com.py:52: Bug #1799994 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1800000_www_honda_co_uk.py:42: Bug #1800000 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1830813_onstove_com.py:18: Bug #1830813 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1836109_watch_tonton_com_my.py:13: Bug #1836109 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1836116_www_slushy_com.py:7: Bug #1836116 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1848716_elal_com.py:7: Bug #1848716 skipped on platform (android)

Log file:

October2023Android.txt

softvision-raul-bucata commented 1 year ago

Next QA Run: 02.11.2023

softvision-raul-bucata commented 1 year ago

[Done] 6th Manual run (H2 -2023) - QA Verification progress available here: https://docs.google.com/spreadsheets/d/1b1K-Zk35uLCk_6n2I-aON-4yc0bkZWQQPme1Em59QOQ/edit#gid=501710986

Items to be removed:

softvision-raul-bucata commented 1 year ago

[Done] 6th Automation run (H2- 2023) I've also executed the Automation Test Suite and below are the results (VPN active pointing to the USA):

Desktop

First run:

Ran 119 checks (119 tests) Expected results: 46 Skipped: 68 tests Unexpected results: 5 test: 5 (5 fail)

Unexpected results: 5( 5 fails) FAIL tests/test_1650317_livescience_com.py::test_enabled[with_interventions] - asyncio.exceptions.CancelledError FAIL tests/test_1786404_business_help_royalmail_com.py::test_enabled[with_interventions] - assert False FAIL tests/test_1800936_cov19ent_kdca_go_kr.py::test_enabled[with_interventions] - webdriver.bidi.error.UnknownErrorException: unknown error (Error: NS_ERROR_UNKNOWN_HOST) FAIL tests/test_1830813_onstove_com.py::test_enabled[with_interventions] - assert 0 == 120 FAIL tests/test_1855088_125039_hrmis2_eghrmis_gov_my.py::test_enabled[with_interventions] - assert not True

We could not perform the Android automated run due to various errors (SDK related)

Log file: November2023Desktop.txt

softvision-raul-bucata commented 1 year ago

Next Run: 30.11.2023, will be performed on the 29th due to National Holidays in Romania

softvision-raul-bucata commented 1 year ago

[Done] 7th Manual run (H2 -2023) - QA Verification progress available here: https://docs.google.com/spreadsheets/d/1b1K-Zk35uLCk_6n2I-aON-4yc0bkZWQQPme1Em59QOQ/edit#gid=1955027764

Items to be removed:

softvision-raul-bucata commented 1 year ago

[Done] 6th Automation run (H2- 2023) I've also executed the Automation Test Suite and below are the results (VPN active pointing to the USA):

Desktop

First run:

Ran 237 checks (237 tests) Expected results: 93 Skipped: 135 tests (Android) Unexpected results: 9 test: 9 (9 fail)

Unexpected results: 5( 5 fails) FAIL tests/test_1650317_livescience_com.py::test_enabled[with_interventions] - asyncio.exceptions.CancelledError FAIL tests/test_1784199_liveatlasathens_com.py::test_enabled[with_interventions] - webdriver.error.NoSuchElementException: no such element (404) FAIL tests/test_1800936_cov19ent_kdca_go_kr.py::test_enabled[with_interventions] - webdriver.bidi.error.UnknownErrorException: unknown error (Error: NS_ERROR_UNKNOWN_HOST) FAIL tests/test_1457335_histography.py::test_disabled[without_interventions] - AssertionError: assert 'http://histography.io/' == 'http://histography.io/browser_support.htm' FAIL tests/test_1650317_livescience_com.py::test_disabled[without_interventions] - asyncio.exceptions.CancelledError FAIL tests/test_1784199_liveatlasathens_com.py::test_disabled[without_interventions] - webdriver.error.NoSuchElementException: no such element (404) FAIL tests/test_1800000_www_honda_co_uk.py::test_disabled[without_interventions] - assert not True FAIL tests/test_1800936_cov19ent_kdca_go_kr.py::test_disabled[without_interventions] - webdriver.bidi.error.UnknownErrorException: unknown error (Error: NS_ERROR_UNKNOWN_HOST)

We could not perform the Android automated run due to a URL error

Log file:

DecemberRun2023Desktop.txt

softvision-raul-bucata commented 1 year ago

This was the RUN for 2023. We completed the last (7th run) earlier, due to Romanian National Holidays. (30th of November, 1st of December)

This OKR can be closed.