Closed denschub closed 10 months ago
[Done] 1stManual run (H2 -2023) - QA Verification progress available here: https://docs.google.com/spreadsheets/d/1b1K-Zk35uLCk_6n2I-aON-4yc0bkZWQQPme1Em59QOQ/edit#gid=683163181
Items to be removed:
1840593 Remove UA Overrides/Interventions for cnki.net 1840595 Remove UA Overrides/Interventions for clalit.co.il
Next QA run: 13.07.2023
In the interests of speeding up our testing, I will run the automated tests myself for the v117 cycle, and update this comment as I find/file issues (so that Raul and Calin may focus on testing the interventions which cannot be automatically tested):
In addition, the following automated tests need to be revised (their related interventions are still needed and working, but the tests themselves have issues):
The following interventions were removed, but the test is still present:
And we should be able to add tests for the following interventions this cycle:
[Done] 2nd Manual run (H2 -2023) - QA Verification progress available here: https://docs.google.com/spreadsheets/d/1b1K-Zk35uLCk_6n2I-aON-4yc0bkZWQQPme1Em59QOQ/edit#gid=235857233
Items to be removed:
Next Run: 10.08.2023
[Done] 3rd Manual run (H2 -2023) - QA Verification progress available here: https://docs.google.com/spreadsheets/d/1b1K-Zk35uLCk_6n2I-aON-4yc0bkZWQQPme1Em59QOQ/edit#gid=262782010
Items to be removed:
[Done] 3rd Automation run (H2- 2023) I've also executed the Automation Test Suite and below are the results (VPN active pointing to the USA):
First run: Ran 225 checks (225 tests) Expected results: 92 Skipped: 123 tests (Android tests) Unexpected results: 10 test: 10 (10 fail)
Unexpected results: 10( 10 fails) FAILED tests/test_1631811_datastudio.py::test_enabled[with_interventions] - assert not True FAILED tests/test_1650317_livescience_com.py::test_enabled[with_interventions] - asyncio.exceptions.CancelledError FAILED tests/test_1799994_vivobarefoot_com.py::test_enabled[with_interventions] - webdriver.error.ScriptTimeoutException: script timeout (500): Timed out after 30000 ms FAILED tests/test_1800936_cov19ent_kdca_go_kr.py::test_enabled[with_interventions] - webdriver.bidi.error.UnknownErrorException: unknown error (Error: NS_ERROR_UNKNOWN_HOST) FAILED tests/test_1836140_indices_iriworldwide_com.py::test_enabled[with_interventions] - webdriver.error.NoSuchElementException: no such element (404) FAILED tests/test_1719859_saxoinvestor.py::test_disabled[without_interventions] - assert False FAILEDtests/test_1799994_vivobarefoot_com.py::test_disabled[without_interventions] - webdriver.error.ScriptTimeoutException: script timeout (500): Timed out after 30000 ms FAILED tests/test_1800936_cov19ent_kdca_go_kr.py::test_disabled[without_interventions] - webdriver.bidi.error.UnknownErrorException: unknown error (Error: NS_ERROR_UNKNOWN_HOST) FAILED tests/test_1836140_indices_iriworldwide_com.py::test_disabled[without_interventions] - asyncio.exceptions.CancelledError FAILED tests/test_1650317_livescience_com.py::test_disabled[without_interventions] - asyncio.exceptions.CancelledError
Log file:
Android Intervention can not temporarily run in our test machines, due to various errors when attempting the start the emulator, or when running the test in a device, due to the device not being able to connect to pages.
Next Run: 07.09.2023
[Done] 4th Manual run (H2 -2023) - QA Verification progress available here: https://docs.google.com/spreadsheets/d/1b1K-Zk35uLCk_6n2I-aON-4yc0bkZWQQPme1Em59QOQ/edit#gid=582536337
[Done] 4th Automation run (H2- 2023) I've also executed the Automation Test Suite and below are the results (VPN active pointing to the USA):
Desktop
First run:
Ran 233 checks (233 tests) Expected results: 94 Skipped: 126 tests (Android) Unexpected results: 13 test: 13 (13 fail)
Unexpected results: 13( 13 fails)
FAIL tests/test_1575000_lloydsbank.py::test_enabled[with_interventions] - asyncio.exceptions.CancelledError FAIL tests/test_1650317_livescience_com.py::test_enabled[with_interventions] - asyncio.exceptions.CancelledError FAIL tests/test_1799994_vivobarefoot_com.py::test_enabled[with_interventions] - webdriver.error.ScriptTimeoutException: script timeout (500): Timed out after 30000 msscript timeout (500): Timed out after 30000 ms FAIL tests/test_1800936_cov19ent_kdca_go_kr.py::test_enabled[with_interventions] - webdriver.error.NoSuchElementException: no such element (404) FAIL tests/test_1830813_onstove_com.py::test_enabled[with_interventions] - assert 0 == 120 FAIL tests/test_1836140_indices_iriworldwide_com.py::test_enabled[with_interventions] - webdriver.error.NoSuchElementException: no such element (404) FAIL tests/test_1849058_nicochannel_jp.py::test_enabled[with_interventions] - webdriver.error.NoSuchElementException: no such element (404) FAIL tests/test_1631811_datastudio.py::test_disabled[without_interventions] - assert False FAIL tests/test_1650317_livescience_com.py::test_disabled[without_interventions] - asyncio.exceptions.CancelledError FAIL tests/test_1719859_saxoinvestor.py::test_disabled[without_interventions] - assert False FAIL tests/test_1799994_vivobarefoot_com.py::test_disabled[without_interventions] - webdriver.error.ScriptTimeoutException: script timeout (500): Timed out after 30000 ms FAIL tests/test_1800936_cov19ent_kdca_go_kr.py::test_disabled[without_interventions] - webdriver.error.NoSuchElementException: no such element (404) FAIL tests/test_1836140_indices_iriworldwide_com.py::test_disabled[without_interventions] - asyncio.exceptions.CancelledError
Log file: September2023Desktop.txt
Android
First run:
Ran 117 checks (117 tests) Expected results: 89 Skipped: 19 tests Unexpected results: 9 test: 9 (9 fail)
Unexpected results: 9 ( (FAIL)
FAILED testing\webcompat\interventions\tests\test_1605611_maps_google_com.py::test_enabled[with_interventions] FAILED testing\webcompat\interventions\tests\test_1722955_frontgate_com.py::test_enabled[with_interventions] FAILED testing\webcompat\interventions\tests\test_1800936_cov19ent_kdca_go_kr.py::test_enabled[with_interventions] FAILED testing\webcompat\interventions\tests\test_1830747_babbel_com.py::test_enabled[with_interventions] FAILED testing\webcompat\interventions\tests\test_1836140_indices_iriworldwide_com.py::test_enabled[with_interventions] FAILED testing\webcompat\interventions\tests\test_1836178_atracker_pro.py::test_enabled[with_interventions] FAILED testing\webcompat\interventions\tests\test_1848711_vio_com.py::test_enabled[with_interventions] FAILED testing\webcompat\interventions\tests\test_1849018_carefirst_com.py::test_enabled[with_interventions] FAILED testing\webcompat\interventions\tests\test_1849058_nicochannel_jp.py::test_enabled[with_interventions]
Skipped
SKIPPED [1] testing\webcompat\interventions\tests\test_1457335_histography.py:7: Bug #1457335 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1472075_bankofamerica.py:6: Bug #1472075 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1570108_steamcommunity.py:111: Bug #1570108 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1577519_stream_directv_com.py:22: Region-locked, cannot test. Try using a VPN set to USA. SKIPPED [1] testing\webcompat\interventions\tests\test_1582582_sling_com.py:9: Bug #1582582 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1610344_directv_com_co.py:44: Bug #1610344 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1631811_datastudio.py:29: Bug #1631811 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1644830_missingmail_usps.py:52: Bug #1644830 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1650317_livescience_com.py:29: Bug #1650317 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1753461_naver.py:9: Bug #1753461 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1778168_watch_antennaplus_gr.py:9: Bug #1778168 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1786404_business_help_royalmail_com.py:59: Bug #1786404 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1799968_samsung_com.py:8: Bug #1799968 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1799994_vivobarefoot_com.py:52: Bug #1799994 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1800000_www_honda_co_uk.py:42: Bug #1800000 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1830813_onstove_com.py:18: Bug #1830813 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1836109_watch_tonton_com_my.py:13: Bug #1836109 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1836116_www_slushy_com.py:7: Bug #1836116 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1848716_elal_com.py:7: Bug #1848716 skipped on platform (android)
Log file: September2023Android.txt
Next Run: 5.10.2023
[Done] 5th Manual run (H2 -2023) - QA Verification progress available here: https://docs.google.com/spreadsheets/d/1b1K-Zk35uLCk_6n2I-aON-4yc0bkZWQQPme1Em59QOQ/edit#gid=945381760
Items to be removed:
[Done] 5th Automation run (H2- 2023) I've also executed the Automation Test Suite and below are the results (VPN active pointing to the USA):
Desktop
First run:
Ran 245 checks (245 tests) Expected results: 101 Skipped: 137 tests (Android) Unexpected results: 7 test: 7 (7 fail)
Unexpected results: 7( 7 fails)
FAIL tests/test_1650317_livescience_com.py::test_enabled[with_interventions] - asyncio.exceptions.CancelledError FAIL tests/test_1799994_vivobarefoot_com.py::test_enabled[with_interventions] - webdriver.error.ScriptTimeoutException: script timeout (500): Timed out after 30000 ms FAIL tests/test_1830813_onstove_com.py::test_enabled[with_interventions] - assert 0 == 120 FAIL tests/test_1836181_conference_amwell_com.py::test_enabled[with_interventions] - webdriver.error.NoSuchElementException: no such element (404) FAIL tests/test_1855088_125039_hrmis2_eghrmis_gov_my.py::test_enabled[with_interventions] - assert not True FAIL tests/test_1650317_livescience_com.py::test_disabled[without_interventions] - asyncio.exceptions.CancelledError FAIL tests/test_1799994_vivobarefoot_com.py::test_disabled[without_interventions] - webdriver.error.ScriptTimeoutException: script timeout (500): Timed out after 30000 ms
Log file: October2023Desktop.txt
Android
First run: Ran 245 checks (245 tests) Expected results: 199 Skipped: 37 tests Unexpected results: 9 test: 9 (9 fail)
Unexpected results: 9 ( (FAIL)
FAILED testing\webcompat\interventions\tests\test_1654877_preev_com.py::test_disabled[without_interventions] FAILED testing\webcompat\interventions\tests\test_1836140_indices_iriworldwide_com.py::test_disabled[without_interventions] FAIL tests/test_1836140_indices_iriworldwide_com.py::test_enabled[with_interventions] - webdriver.bidi.error.UnknownErrorException: unknown error (Error: NS_ERROR_NET_TIMEOUT) FAILED testing\webcompat\interventions\tests\test_1605611_maps_google_com.py::test_enabled[with_interventions] FAILED testing\webcompat\interventions\tests\test_1654877_preev_com.py::test_enabled[with_interventions] FAILED testing\webcompat\interventions\tests\test_1829126_otsuka_co_jp.py::test_enabled[with_interventions] FAILED testing\webcompat\interventions\tests\test_1836140_indices_iriworldwide_com.py::test_enabled[with_interventions] FAILED testing\webcompat\interventions\tests\test_1836181_conference_amwell_com.py::test_enabled[with_interventions] FAILED testing\webcompat\interventions\tests\test_1848711_vio_com.py::test_enabled[with_interventions] FAILED testing\webcompat\interventions\tests\test_1849018_carefirst_com.py::test_enabled[with_interventions]
Skipped: 19
SKIPPED [1] testing\webcompat\interventions\tests\test_1457335_histography.py:15: Bug #1457335 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1472075_bankofamerica.py:14: Bug #1472075 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1577519_stream_directv_com.py:22: Region-locked, cannot test. Try using a VPN set to USA. SKIPPED [1] testing\webcompat\interventions\tests\test_1582582_sling_com.py:21: Bug #1582582 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1610344_directv_com_co.py:51: Bug #1610344 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1631811_datastudio.py:36: Bug #1631811 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1644830_missingmail_usps.py:59: Bug #1644830 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1650317_livescience_com.py:36: Bug #1650317 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1753461_naver.py:17: Bug #1753461 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1778168_watch_antennaplus_gr.py:17: Bug #1778168 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1786404_business_help_royalmail_com.py:66: Bug #1786404 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1799968_samsung_com.py:16: Bug #1799968 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1799994_vivobarefoot_com.py:59: Bug #1799994 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1800000_www_honda_co_uk.py:49: Bug #1800000 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1830813_onstove_com.py:25: Bug #1830813 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1836109_watch_tonton_com_my.py:21: Bug #1836109 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1836116_www_slushy_com.py:15: Bug #1836116 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1848716_elal_com.py:16: Bug #1848716 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1457335_histography.py:7: Bug #1457335 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1472075_bankofamerica.py:6: Bug #1472075 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1570108_steamcommunity.py:111: Bug #1570108 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1577519_stream_directv_com.py:22: Region-locked, cannot test. Try using a VPN set to USA. SKIPPED [1] testing\webcompat\interventions\tests\test_1582582_sling_com.py:9: Bug #1582582 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1610344_directv_com_co.py:44: Bug #1610344 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1631811_datastudio.py:29: Bug #1631811 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1644830_missingmail_usps.py:52: Bug #1644830 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1650317_livescience_com.py:29: Bug #1650317 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1753461_naver.py:9: Bug #1753461 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1778168_watch_antennaplus_gr.py:9: Bug #1778168 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1786404_business_help_royalmail_com.py:59: Bug #1786404 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1799968_samsung_com.py:8: Bug #1799968 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1799994_vivobarefoot_com.py:52: Bug #1799994 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1800000_www_honda_co_uk.py:42: Bug #1800000 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1830813_onstove_com.py:18: Bug #1830813 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1836109_watch_tonton_com_my.py:13: Bug #1836109 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1836116_www_slushy_com.py:7: Bug #1836116 skipped on platform (android) SKIPPED [1] testing\webcompat\interventions\tests\test_1848716_elal_com.py:7: Bug #1848716 skipped on platform (android)
Log file:
Next QA Run: 02.11.2023
[Done] 6th Manual run (H2 -2023) - QA Verification progress available here: https://docs.google.com/spreadsheets/d/1b1K-Zk35uLCk_6n2I-aON-4yc0bkZWQQPme1Em59QOQ/edit#gid=501710986
Items to be removed:
[Done] 6th Automation run (H2- 2023) I've also executed the Automation Test Suite and below are the results (VPN active pointing to the USA):
Desktop
First run:
Ran 119 checks (119 tests) Expected results: 46 Skipped: 68 tests Unexpected results: 5 test: 5 (5 fail)
Unexpected results: 5( 5 fails) FAIL tests/test_1650317_livescience_com.py::test_enabled[with_interventions] - asyncio.exceptions.CancelledError FAIL tests/test_1786404_business_help_royalmail_com.py::test_enabled[with_interventions] - assert False FAIL tests/test_1800936_cov19ent_kdca_go_kr.py::test_enabled[with_interventions] - webdriver.bidi.error.UnknownErrorException: unknown error (Error: NS_ERROR_UNKNOWN_HOST) FAIL tests/test_1830813_onstove_com.py::test_enabled[with_interventions] - assert 0 == 120 FAIL tests/test_1855088_125039_hrmis2_eghrmis_gov_my.py::test_enabled[with_interventions] - assert not True
We could not perform the Android
automated run due to various errors (SDK related)
Log file: November2023Desktop.txt
Next Run: 30.11.2023, will be performed on the 29th due to National Holidays in Romania
[Done] 7th Manual run (H2 -2023) - QA Verification progress available here: https://docs.google.com/spreadsheets/d/1b1K-Zk35uLCk_6n2I-aON-4yc0bkZWQQPme1Em59QOQ/edit#gid=1955027764
Items to be removed:
[Done] 6th Automation run (H2- 2023) I've also executed the Automation Test Suite and below are the results (VPN active pointing to the USA):
Desktop
First run:
Ran 237 checks (237 tests) Expected results: 93 Skipped: 135 tests (Android) Unexpected results: 9 test: 9 (9 fail)
Unexpected results: 5( 5 fails) FAIL tests/test_1650317_livescience_com.py::test_enabled[with_interventions] - asyncio.exceptions.CancelledError FAIL tests/test_1784199_liveatlasathens_com.py::test_enabled[with_interventions] - webdriver.error.NoSuchElementException: no such element (404) FAIL tests/test_1800936_cov19ent_kdca_go_kr.py::test_enabled[with_interventions] - webdriver.bidi.error.UnknownErrorException: unknown error (Error: NS_ERROR_UNKNOWN_HOST) FAIL tests/test_1457335_histography.py::test_disabled[without_interventions] - AssertionError: assert 'http://histography.io/' == 'http://histography.io/browser_support.htm' FAIL tests/test_1650317_livescience_com.py::test_disabled[without_interventions] - asyncio.exceptions.CancelledError FAIL tests/test_1784199_liveatlasathens_com.py::test_disabled[without_interventions] - webdriver.error.NoSuchElementException: no such element (404) FAIL tests/test_1800000_www_honda_co_uk.py::test_disabled[without_interventions] - assert not True FAIL tests/test_1800936_cov19ent_kdca_go_kr.py::test_disabled[without_interventions] - webdriver.bidi.error.UnknownErrorException: unknown error (Error: NS_ERROR_UNKNOWN_HOST)
We could not perform the Android
automated run due to a URL error
Log file:
This was the RUN for 2023. We completed the last (7th run) earlier, due to Romanian National Holidays. (30th of November, 1st of December)
This OKR can be closed.