Flaky e2e test: `incorrect-locale` has been failing constantly in "after each" hook

[tatilepizs]

Describe the issue E2e test /tests/e2e/default/translations/incorrect-locale.wdio-spec.js has been failing constantly in the "after each" hook due to timing

------------------------------------------------------------------
[chrome 90.0.4430.72 linux #0-5] Running: chrome (v90.0.4430.72) on linux
[chrome 90.0.4430.72 linux #0-5] Session ID: 62d09db66cd013a39458898546df0548
[chrome 90.0.4430.72 linux #0-5]
[chrome 90.0.4430.72 linux #0-5] » /tests/e2e/default/translations/incorrect-locale.wdio-spec.js
[chrome 90.0.4430.72 linux #0-5] Testing Incorrect locale
[chrome 90.0.4430.72 linux #0-5]    ✖ should work with incorrect locale
[chrome 90.0.4430.72 linux #0-5]    ✖ "after each" hook for Testing Incorrect locale
[chrome 90.0.4430.72 linux #0-5]
[chrome 90.0.4430.72 linux #0-5] 1 failing (5m 5.1s)
[chrome 90.0.4430.72 linux #0-5]
[chrome 90.0.4430.72 linux #0-5] 1) Testing Incorrect locale should work with incorrect locale
[chrome 90.0.4430.72 linux #0-5] Timeout of 120000ms exceeded. The execution in the test "Testing Incorrect locale should work with incorrect locale" took too long. Try to reduce the run time or increase your timeout for test specs (https://webdriver.io/docs/timeouts). (/home/runner/work/cht-core/cht-core/tests/e2e/default/translations/incorrect-locale.wdio-spec.js)
[chrome 90.0.4430.72 linux #0-5] Error: Timeout of 120000ms exceeded. The execution in the test "Testing Incorrect locale should work with incorrect locale" took too long. Try to reduce the run time or increase your timeout for test specs (https://webdriver.io/docs/timeouts). (/home/runner/work/cht-core/cht-core/tests/e2e/default/translations/incorrect-locale.wdio-spec.js)
[chrome 90.0.4430.72 linux #0-5]     at createTimeoutError (/home/runner/work/cht-core/cht-core/node_modules/mocha/lib/errors.js:498:15)
[chrome 90.0.4430.72 linux #0-5]     at Test.Runnable._timeoutError (/home/runner/work/cht-core/cht-core/node_modules/mocha/lib/runnable.js:429:10)
[chrome 90.0.4430.72 linux #0-5]     at Timeout.<anonymous> (/home/runner/work/cht-core/cht-core/node_modules/mocha/lib/runnable.js:244:24)
[chrome 90.0.4430.72 linux #0-5]     at listOnTimeout (node:internal/timers:573:17)
[chrome 90.0.4430.72 linux #0-5]     at processTimers (node:internal/timers:514:7)
[chrome 90.0.4430.72 linux #0-5]
[chrome 90.0.4430.72 linux #0-5] 2) Testing Incorrect locale "after each" hook for Testing Incorrect locale
[chrome 90.0.4430.72 linux #0-5] Timeout
[chrome 90.0.4430.72 linux #0-5] Error: Timeout
[chrome 90.0.4430.72 linux #0-5]     at listOnTimeout (node:internal/timers:573:17)
[chrome 90.0.4430.72 linux #0-5]     at processTimers (node:internal/timers:514:7)
------------------------------------------------------------------

This is an example run

Here is the artifact in case it is helpful -> incorrect-locale-ci-webdriver-default-core-minimum-browser.zip

[garethbowen]

@tatilepizs Could you please include the log as a text snippet rather than a screenshot so we can find this issue when doing a text search?

[tatilepizs]

Sure, I updated the description.

[dianabarsan]

I think the failure is both in the test and in the hook. Looking at the reported screenshots, it looks like the language actually never changes:

[ralfudx]

The way the test was written, only a few components on the page will have translations as seen in the createLanguage function at the top of the spec file:

'n.month': '{MONTHS, plural, =1{1 luna} other{# luni}}',
'n.week': '{WEEKS, plural, =1{1 saptamana} other{# saptamani}}',
'reports.none.n.months':
  '{MONTHS, plural, =1{No reports in the last month.} other{No reports in the last # months.}}',
'task.days.left': '{DAYS, plural, =1{1 day left} other{# days left}',
'tasks.none.n.weeks': '{WEEKS, plural, =1{No tasks in the next week.} other{No tasks in the next # weeks.}}',
'Reports': 'HilReports',
'view.all': 'View all'

So the expected screenshot when the test runs successfully will look like this:

Also, the validations for the language translations are only for the Tasks filter and the HilReports filter.

From my investigations I noticed intermittent failures happening in the afterEach hook majorly because of a timeout which seems to happen in the revertSettings.

Timeout of 120000ms exceeded. The execution in the test "Testing Incorrect locale should work with incorrect locale" took too long. Try to reduce the run time or increase your timeout for test specs (https://webdriver.io/docs/timeouts). (/home/runner/work/cht-core/cht-core/tests/e2e/default/translations/incorrect-locale.wdio-spec.js)

Moving this function from the afterEach block to the main test block seemed to make the test more stable and we can accommodate this fix for now because we have only one test in the spec file - /tests/e2e/default/translations/incorrect-locale.wdio-spec.js

After executing the test about 20 times here, there was no related failure - the only failure on Attempt#5 & Attempt#20 happened because of this error - A previous test did not call revertSettings in the createLanguage function and isn't one of the previous functions in the afterEach block.