search

medic / cht-core

The CHT Core Framework makes it faster to build responsive, offline-first digital health apps that equip health workers to provide better care in their communities. It is a central resource of the Community Health Toolkit.
https://communityhealthtoolkit.org
GNU Affero General Public License v3.0
467 stars 217 forks source link

Protractor test failures are not retried and failed retries clear results from previous failure #8103

Closed dianabarsan closed 1 year ago

dianabarsan commented 1 year ago

Describe the issue We added retries to protractor e2e test failures. However, they do not seem to work: the retry will fail immediately with an error:

 Re-running tests , attempt :  1
 Re-running the following test files : /home/runner/work/cht-core/cht-core/tests/e2e/protractor/client-side-transitions/forms/index.js,/home/runner/work/cht-core/cht-core/tests/e2e/protractor/client-side-transitions/muting.js,/home/runner/work/cht-core/cht-core/tests/e2e/protractor/contacts/contact-summary.js,/home/runner/work/cht-core/cht-core/tests/e2e/protractor/contacts/reports-subject.js,/home/runner/work/cht-core/cht-core/tests/e2e/protractor/sms/africas-talking.js,/home/runner/work/cht-core/cht-core/tests/e2e/protractor/sms/rapidpro.spec.js,/home/runner/work/cht-core/cht-core/tests/e2e/protractor/sms/registration-by-sms.js,/home/runner/work/cht-core/cht-core/tests/e2e/protractor/sms/send-message.js,/home/runner/work/cht-core/cht-core/tests/e2e/protractor/targets/target-aggregates.spec.js,/home/runner/work/cht-core/cht-core/tests/e2e/protractor/translations/incorrect-locale.spec.js,/home/runner/work/cht-core/cht-core/tests/e2e/protractor/client-side-transitions/forms/index.js,/home/runner/work/cht-core/cht-core/tests/e2e/protractor/client-side-transitions/muting.js,/home/runner/work/cht-core/cht-core/tests/e2e/protractor/contacts/contact-summary.js,/home/runner/work/cht-core/cht-core/tests/e2e/protractor/contacts/reports-subject.js,/home/runner/work/cht-core/cht-core/tests/e2e/protractor/sms/africas-talking.js,/home/runner/work/cht-core/cht-core/tests/e2e/protractor/sms/rapidpro.spec.js,/home/runner/work/cht-core/cht-core/tests/e2e/protractor/sms/registration-by-sms.js,/home/runner/work/cht-core/cht-core/tests/e2e/protractor/sms/send-message.js,/home/runner/work/cht-core/cht-core/tests/e2e/protractor/targets/target-aggregates.spec.js,/home/runner/work/cht-core/cht-core/tests/e2e/protractor/translations/incorrect-locale.spec.js

Closing report
[12:10:57] E/launcher - callback is not a function
[12:10:57] E/launcher - TypeError: callback is not a function
    at /home/runner/work/cht-core/cht-core/node_modules/protractor-jasmine2-screenshot-reporter/index.js:525:11
    at FSReqCallback.oncomplete (node:fs:192:23)

Example run: https://github.com/medic/cht-core/actions/runs/4269504105/jobs/7433297666#step:16:634 The report folder is also overwritten, and the other failure screenshots are deleted, which makes debugging harder.

Describe the improvement you'd like Fix it so retries will happen.

Describe alternatives you've considered Have it so the retrying will not clear previous results folder. even if retries don't work.

garethbowen commented 1 year ago

Example of successful capture of run failure: https://github.com/medic/cht-core/actions/runs/4277828531/jobs/7447091600#step:16:2097

I'm now getting the screenshot and logs say:

 Executed 1 of 2 specs (1 FAILED) (1 PENDING) in 1 sec.
>> 01:28:07.816 INFO [ActiveSessions$1.onStop] - Removing session 52785e2b69b9f77df80e35649e74f1b8 (org.openqa.selenium.chrome.ChromeDriverService)
[01:28:07] I/launcher - 0 instance(s) of WebDriver still running
[01:28:07] I/launcher - chrome #01 failed 1 test(s)
[01:28:07] I/launcher - overall: 1 failed spec(s)
Closing report
[ProtractorJasmineRetry] Test failed after 5 attempts, exiting with code 1
garethbowen commented 1 year ago

Merged. No AT required.