keymanapp / keyman

Keyman cross platform input methods system running on Android, iOS, Linux, macOS, Windows and mobile and desktop web
https://keyman.com/
Other
382 stars 108 forks source link

bug(web): automated test instability for prediction test #11844

Closed jahorton closed 15 hours ago

jahorton commented 1 month ago

Build failure seen for "Test: Language Modeling Layer (Common)":


15:56:57 
  LMLayer using the trie model

15:56:57 
    Prediction

15:56:57 
    Regression tests

15:56:57 
      should use the default searchTermToKey()

15:56:58 
        expected +0 to be at least 1

15:56:58 
        AssertionError: expected +0 to be at least 1
            at file:///C:/BuildAgent/work/99b311828f4ee7c/keyman/common/predictive-text/unit_tests/headless/worker-trie-integration.js:85:16

The worker's "test mode" is on, so we shouldn't be exiting early... but perhaps recent changes from #11784 and its predecessors have had an unintended consequence here.

Originally posted by @jahorton in https://github.com/keymanapp/keyman/issues/11264#issuecomment-2181818200

jahorton commented 2 weeks ago

Just saw another one:

04:03:41           expected undefined to be truthy
04:03:41           AssertionError: expected undefined to be truthy
              at Context.<anonymous> (file:///C:/BuildAgent/work/99b311828f4ee7c/keyman/common/web/lm-worker/src/test/mocha/cases/worker-model-compositor.js:194:16

From https://build.palaso.org/buildConfiguration/Keyman_Common_LMLayer_TestPullRequests/476399?buildTab=log&linesState=400.982.983.984.995&logView=flowAware&focusLine=983

This is for a relatively newer unit test, so the instability cause might be a little different, but it still falls under the same overall category.

jahorton commented 2 weeks ago

Ah, so for the original issue, the worker's "test mode" is not on - the affected test is an integration test, ensuring that code outside the worker interfaces properly with code inside the worker. That codepath actually does have a way to set the "test mode" flag, but it never got enabled for... whatever reason.

The error seen just above this comment does have test-mode setting access, though, so its failure is for a different reason.