bug(ios): iPhone lock turns keyboard into blank gray box

trosel commented 2 years ago

Describe the bug keyboard does not restart when it gets stuck. you can’t leave the keyboard. you can’t force a refresh on the keyboard. only option is to system restart the ios device

To Reproduce

Start to type a message in the app
leave the app with the keyboard open
come back to the app
the keyboard is open with no buttons on it. you cant close the keyboard or go back to the system keyboard because it’s just a grey box, no buttons.

Expected behavior if the keyboard errors out, it should refresh or reboot so buttons are usable again.

Screenshots F4082CDE-F848-484B-8D39-256A39CFBDC0

Keyman for iPhone/iPad/Android:

Device: iphone 8+
OS: iOS 15.1
Keyman version 14.0.286
Target application: Element (matrix chat)

Keyboard

Keyboard name: shaw imperial
Keyboard version: 1.1
Language name: English (shavian)

trosel commented 2 years ago

i think i found another way to replicate this.

let the phone screen “fall asleep” with the keyboard open
wake the phone and unlock it
when you unlock, the keyboard should be blanked out

the only way to fix it is to turn the phone off and on again. because you can’t return to your main keyboard unless you can exit this keyman keyboard.

a simpler solution would be adding a way to turn the keyboard off manually. or restart it manually. maybe from the keyman app?

sgschantz commented 2 years ago

Was not able to reproduce on simulator, but was able to reproduce on an iPhone 11 Pro with iOS 15.3.1.

Open Messages Open Keyman Keyboard (Shaw Imperial) Lock iPhone Unlock iPhone Messages app is shown, but keyboard is a blank gray box that will not refresh or respond to taps. I can switch to other keyboards via the globe key, but switching back to Keyman will bring me back to the gray box again.

reproduced with Messages and Notes with Hebrew and Shavian keyboards must restart phone to restore use of Keyman

trosel commented 2 years ago

@sgschantz given that this is not the only issue i've experienced with keyman locking up my whole system keyboard, can you add a way to simply force restart the keyman app? Maybe inside the keyman app you can add a button that lets users force quit and restart the keyman keyboard?

sgschantz commented 2 years ago

can no longer reproduce with beta, 15.0.225

@trosel Keyman is a keyboard extension that is invoked by iOS and attached to other apps. There are strict limitations in what these extensions can do and no support for any way to restart the extension and reattach it to the hosting app. For any issues that you encounter, please report, and, if possible, include details steps on how to reproduce. Thanks!

mcdurdin commented 2 years ago

There are two possible reasons that the keyboard doesn't render:

There is a script error within the keyboard.html page that blocks the initial render of the keyboard. One potential workaround in this case is to trigger a page-reload (plus trigger an error report, hopefully) if you get a touch on the document body itself within the keyboard.html web host? Because that should only be possible if the osk has failed to render.
The web control itself fails to initialize and/or size correctly to its bounds, and we see the backing element for it instead of the web control. Detecting a touch on the backing element may be possible here? This particular issue seems rare these days.

(We could also differentiate between the two error modes with color or similar?)

sgschantz commented 2 years ago

@mcdurdin It looks like this is dependent upon 'Allow Full Access'. I re-tested with the version first reported, 14.0.286 and could not reproduce even there. But when I went into Settings -> General -> Keyboard -> Keyboards -> Keyman and disabled Allow Full Access, I was able to get the blank gray box on lock/unlock every time on both 14 and the latest beta of 15.

I cannot reproduce with any version or setting on the simulator.

mcdurdin commented 2 years ago

Good investigation! I am very sorry to hear that it only occurs with Allow Full Access off -- because that is incredibly painful to debug because you can't get access to the webview and you cannot persist data and make it accesible later AFAIK (I may be wrong on this -- I hope I am -- and if so, then we should look at caching web error reports and then switching Allow Full Access back on and sucking them out).

In the past, trying to diagnose issues here, I've resorted to logging directly to the html web document for keyboard.html and trying to log sentry errors directly to that so that at least they are visible ...

trosel commented 2 years ago

I indeed have “allow full access” off. It just seems invasive from a privacy standpoint.

But if it needs to be on to work the best, maybe there can be an added note somewhere that addresses the privacy concern?

mcdurdin commented 2 years ago

The option shouldn't need to be on. It's tricky, because we cannot debug the keyboard with Allow Full Access off, and in this case the issue only arises in that situation. So it is complicated to trace the cause, but we'll get there.

Here's some background on the option: https://blog.keyman.com/2014/12/keyman-for-iphone-and-ipad-no-longer-requires-allow-full-access/

From my perspective, it's a bit frustrating. The Apple system keyboard has 'Allow Full Access' on but you are never warned that your data will be sent to Apple... so they generate a worry in users' minds which is frankly a little bit over the top: every other app on your device can and probably does 'phone home' all sorts of data, and while a keyboard app does obviously have greater access across all apps, all this question does is raise alarm bells for which the user has no real pathway to resolve.

The exact same set of privacy concerns exist for Keyman on every other platform, but we get probably 10x the concern from iOS users as compared to all other platforms, combined -- despite it being the smallest user base. This is entirely due to the presence of this toggle.

At the end of the day, the Keyman app source code is all open, which means that (in theory) anyone who is concerned can audit the code for themselves. We don't collect user data -- only technical diagnostic data when there is a crash (which is not possible with Allow Full Access off), and that is only kept for a short time -- the logs expire I think after 90 days.

jahorton commented 2 years ago

Something that may prove useful: https://developer.apple.com/documentation/uikit/uiinputviewcontroller/2875763-hasfullaccess?changes=_2

It's possible to write a condition for the keyboard based on whether or not it even has full access. So, we could add a condition somewhere to auto-reset the page whenever the system keyboard doesn't have the permission. Of course... the question is currently "where?".

jahorton commented 2 years ago

One other note: there is a second workaround, at least based on testing with my personal device:

If you add "Allow Full Access" after the fact, the keyboard will display again when next loaded. I didn't go so far as to test whether or not it'd then, uh, "self-wrong" if the permission were subsequently removed, but this will at least allow a quick way in to restore keyboard-swapping behavior.

jahorton commented 2 years ago

While by no means strongly connected, I at least found report of a similar effect with PDFs in the Apple forums:

https://developer.apple.com/forums/thread/114740

No workaround was mentioned outside of refreshing the page entirely, though there was also no mention of anything "full access" related.

jahorton commented 2 years ago

Though I know it doesn't directly reproduce the bug, I thought I'd experiment and see what effects are noticeable in Simulator for this, to see if anything related might be detectable.

The "closest" repro I could initially find was to use Safari's address bar as the active field when locking and unlocking the simulated device. Granted, this may have cross-effects with the underlying bug of #2300, but my other attempts clearly had cleared the keyboard ("first responder" state) and then restored it when the app was restored, which didn't keep the keyboard active across the unlock->lock process.

In this scenario - using Safari's address bar - it's worth noting that the keyboard's WebView crashes upon device unlock. In Safari's "Develop" menu, there will be a new instance of the keyboard's WebView listed.

Searching online for related things led me to this: https://developer.apple.com/documentation/webkit/wknavigationdelegate/1455639-webviewwebcontentprocessdidtermi

We don't yet implement it, but it's a callback that's triggered whenever the WebView crashes. Sounds like something worth implementing in some fashion. It may also be worth considering in relation to #6552... and it may also be worth checking to see if this bug persists on that branch, just to be sure.

Fortunately, I didn't stop there and found another scenario that preserves "first responder" state: the Messages app.

Select one of the default contacts
Select the message-creation text area

Before locking:

After unlocking:

However, two additional things of note with this experiment:

Rotating the keyboard while in the Messages text area does not crash it, unlike with #2300 . Thus, this crash / bug is distinct from that one!
During one of my repro attempts, I saw a flash of the old keyboard instance right before it crashed during the unlock process. I'm fairly certain the crash happens on unlock.

Assuming this pattern holds on real devices... the answer may be to implement the WebView crash handler above and either force-reload the page or rebuild the WebView when it's triggered.

For one last experiment, I repeated the last scenario, but enabled Full Access for the Keyman keyboard. (Again, still on Simulator). There was no discernable difference in behavior - that is, the keyboard still crashed and was reloaded.

In the past, trying to diagnose issues here, I've resorted to logging directly to the html web document for keyboard.html and trying to log sentry errors directly to that so that at least they are visible ...

"Fun" detail here - when I used Safari to inspect the "old" instance of the WebView still in its debug listing... there were no (distinct) errors in its log! Only the usual suspects that are always there (related to inspecting in Safari):

An extra copy of these errors is produced if the inspector is closed, then reopened... so I'm confident in saying that these are unrelated. (Also, note that they're near-all visibly related to source code mapping.)

jahorton commented 2 years ago

So, the article that led me to that crash-notification method I referenced in the above comment:

https://segmentfault.com/a/1190000040652799/en

Something worth noting:

WebContent process crashes

[...] At the same time, if the user's device memory is tight, the system may take the initiative to KILL WebContent process. That is, it may happen that the App process (foreground) is normal, but the WebContent crashes and the page reloads.

Come to think of it, locking the device sounds like a very natural use-case for the OS to request memory cleanup - especially for third-party app extensions. They're pretty strict on how much memory keyboard app-extensions are supposed to use, after all.

Also, a neat takeaway:

... WebContent process ...

The web page within a WKWebView is managed by its own process, which can crash independently of our app's main process.

sgschantz commented 1 year ago

I used the Xcode Instruments app to view the active processes on the iPhone while the app was running and can see a process named 'com.apple.WebKit.WebContent' disappear but not until unlock. A new process with the same name appears immediately after. And this happens with Allow Full Access enabled, as Josh observed.

When rotating the device with the system keyboard active in Safari (to reproduce #6552), I see a process of the same name disappear and a new WebContent process created. The difference for the rotate issue, though, is that the old process does not disappear until after the new WebContent process is created. With the unlock bug there is a gap from when the old process is gone until the new process appears.

So it looks like 1) the crash happens at unlock 2) the crash happens even when Allow Full Access is enabled and causes a new WebContent process to be created 3) if Allow Full Access is disabled, SWKeyboard cannot recover and use the new WebContent process, resulting in the blank screen

sgschantz commented 1 year ago

During the lock-unlock sequence that causes the WebContent process to die, the statement is written to the logs:

2022-05-23 13:45:03.207847+0700 SWKeyboard[16133:1080989] Could not signal service com.apple.WebKit.WebContent: 113: Could not find specified service