as you noted, unit tests have recently started failing. There were no code nor tests changes since the last time CI was green, so it's really disturbing that this has happened. The only differences I noticed between these CI runs are:
Chrome version (used to be 103, now it's 105)
CKEditor version (bumped from 4.19.1 to 4.20.0)
Possibly different versions of some dev dependencies (hard to track since there is no lock file)
I can reproduce failing tests locally as well.
I don't think it's related to Chrome versions (but I haven't tested though) and it's certainly not related to CKEditor versions (I've tested multiple older versions with the same result). My bet is that a newer version of some dependency must have been installed which introduced some subtle bugs to our setup. This is possible since the setup we use is quite exotic (Karma + Chrome + testing-library + async runtime loading of CKEditor dependencies) and proved to be "flaky" in the past.
While debugging I noticed that our testing setup now started to have troubles with proper cleanup of CKEditor instances between runs. This led to some inconsistencies in the assertions. I couldn't pinpoint the exact cause of that though.
Following mitigations have been applied:
Removed beforeEach check that asserted cleanup of editor instances. Instances get cleaned up eventually but they might not get cleaned up before next test starts.
Adjusted a few expectations in useCKEditor tests.
Refactored tests in registerEditorEventHandler to not rely on useCKEditor. That was another source of "flakiness". Now a mock is passed instead of a real editor instance.
The CI is green again ✅. E2E tests pass correctly and I've additionally tested samples manually. I haven't noticed any issues.
As a long term solution we might discuss a different approach to testing.
Hi @jacekbogdanski,
as you noted, unit tests have recently started failing. There were no code nor tests changes since the last time CI was green, so it's really disturbing that this has happened. The only differences I noticed between these CI runs are:
I can reproduce failing tests locally as well.
I don't think it's related to Chrome versions (but I haven't tested though) and it's certainly not related to CKEditor versions (I've tested multiple older versions with the same result). My bet is that a newer version of some dependency must have been installed which introduced some subtle bugs to our setup. This is possible since the setup we use is quite exotic (Karma + Chrome + testing-library + async runtime loading of CKEditor dependencies) and proved to be "flaky" in the past.
While debugging I noticed that our testing setup now started to have troubles with proper cleanup of CKEditor instances between runs. This led to some inconsistencies in the assertions. I couldn't pinpoint the exact cause of that though.
Following mitigations have been applied:
beforeEachcheck that asserted cleanup of editor instances. Instances get cleaned up eventually but they might not get cleaned up before next test starts.useCKEditortests.registerEditorEventHandlerto not rely onuseCKEditor. That was another source of "flakiness". Now a mock is passed instead of a real editor instance.The CI is green again ✅. E2E tests pass correctly and I've additionally tested samples manually. I haven't noticed any issues.
As a long term solution we might discuss a different approach to testing.