keymanapp / keyman

Keyman cross platform input methods system running on Android, iOS, Linux, macOS, Windows and mobile and desktop web
https://keyman.com/
Other
378 stars 106 forks source link

bug(ios): potential context desync after multitaps + paragraph breaks #10696

Closed jahorton closed 4 months ago

jahorton commented 4 months ago

Possibly related to #10204 / #10633 and the related underlying behaviors.

Being able to elaborate further may be tricky without further investigation, but I'm pretty sure we can all agree that the following behavior, after the newline, is wrong.

This is an issue against master that is merely perpetuated and/or changed, but not resolved, by #10589 at present.

repro

Motivated by https://github.com/keymanapp/keyman/pull/10589#issuecomment-1940600108:

Finally, I have identified what caused the issue in the PR. Here is what I did while following the steps: [...]

  1. Next, I pressed the key to move to the next line.

Debugger-based inspection of the keyboard WebView indicates that, at the following frame:

image

... the KMW-internal context is:

Screen Shot 2024-02-13 at 3 21 19 PM

... or similar. It appears that the attempt to resynchronize context for backspaces (see #10633) may be having adverse consequences as complicated new text is typed. The textDocumentProxy context window also seems to be acting pretty oddly here, but that's just an inference at this time.

Other patterns sometimes results in all preceding context (having been maintained within the Web engine) suddenly being duplicated - something that is definitely concerning and erroneous behavior.


Repro:

  1. Download this KMP: https://jahorton.github.io/diacritic_rota.kmp
  2. Open / install it for use within the Keyman app.
  3. Double-tap the 8 key.
    • Expected total result: a
  4. Triple-tap the '◌̀' key - a circumflex diacritic should appear over the a.
  5. Triple-tap the 9 key.
    • Expected total result:
  6. Tap the '◌̀' key. A diacritic should appear over the ê.
  7. Open the in-app Settings menu.
    • Note, for test maintenance: the point of this step and the next one is to trigger a context-reset.
  8. Close the in-app Settings menu.
  9. Double-tap the 8 key.
    • Expected final result: aềa
    • For clarity:
      • With a copy of the à's accent-mark on top the original ê.
        • It looks different in raw-text form here due to GitHub's font / rendering.
  10. Double-tap the ENTER key.
  11. Repeat steps 3-6, then step 9.

Done using Simulator targeting iPhone 13 Pro, iOS 15.4. Interestingly, it doesn't appear to repro in Simulator targeting iPhone SE (3rd gen), iOS 16.2? At least... not the same sequence. That said, it was with an iPhone 13 Pro (non-simulated) where we saw weirdness in the tests for #10589, which led to discovery of this issue.

jahorton commented 4 months ago

I wonder if recent work from #10662 might be useful here; when the iOS context-window is jumped to the right, but the caret is at the same relative location, that "common substring" method could be useful for detecting when a "context reset" is actually a "context window jump".

jahorton commented 4 months ago

Making good use of time-honored, reliable debugging methods - aka the humble console.warn statement:

Screen Shot 2024-02-14 at 1 09 09 PM

Screen Shot 2024-02-14 at 1 11 25 PM

The messages are generated from ios-host.js, setKeymanVal().

Once the context window shifts locations (upon reaching the new paragraph)... it appears that something is retriggering a context change.

So, what might that be?

First context change of pair Screen Shot 2024-02-14 at 1 17 07 PM

Second context change of pair Screen Shot 2024-02-14 at 1 17 15 PM

After further investigation, it appears that two Swift-side context updates are triggered:

  1. When KMW reports context transformation data to the app, the final Swift-side context is passed through to KMW for synchronization.
  2. In-app only: the changes passed through via the textDocumentProxy get propagated to the in-app TextView, which itself triggers a context update. As the context window between textDocumentProxy is now, uh, "less complete" than that of the TextView, they appear "different" and trigger a full context reset within KMW.
    • I'd been noticing that deadkeys weren't working after the paragraph break, which is explained well by this observation.