zulip / zulip-mobile

Zulip mobile apps for Android and iOS.
https://zulip.com/apps/
Apache License 2.0
1.28k stars 643 forks source link

"RuntimeException: Probable deadlock detected due to WebView API being called on incorrect thread while the UI thread is blocked." #4051

Open chrisbobbe opened 4 years ago

chrisbobbe commented 4 years ago

Marking P1 because this is new (within the last two weeks) and may (unconfirmed) be a cause of symptoms like https://github.com/zulip/zulip-mobile/issues/4033 which we've had multiple reports of.

Here's the Sentry link (only accessible to core team members).

Looks like it was first seen on 26.25.148 (GitHub, CZO).

It may be too early to tell, but it looks like an upward trend is developing (screenshot from Sentry):

image (The pink dot on the time axis, at left, is first seen; green, on the right, is last seen)

Looks like it may be coming from https://github.com/chromium/chromium/blob/master/android_webview/java/src/org/chromium/android_webview/WebViewChromiumRunQueue.java#L75?

gnprice commented 4 years ago

Hmm, very interesting! I actually saw that error message, even, when looking at Sentry earlier today but didn't think of the plausible connection with that symptom.

The timing is puzzling because it doesn't really line up with the rollout of a new version -- the first occurrence was 2020-04-08, and it really takes off starting 2020-04-13, but the majority of our Android users were on 26.25.148 by 2020-04-05, and had moved on to 26.26.149 by 2020-04-12.

I wonder if it could be related to a Chrome update. :-/ From the trusty Wikipedia table at [[Google Chrome version history]] and its source here, the update to Chrome 81 for Android was announced 2020-04-07 -- and that graph is a pretty plausible shape for the rollout of a new Chrome version.

I seem to have Chrome 81 on my phone, and I haven't observed the #4033 symptom, but of course as reported it's only some of the time.

Googling finds this SO question and answer: https://stackoverflow.com/questions/58519749/android-10-probable-deadlock-detected-due-to-webview-api-being-called-on-incor and the answer looks like it explains the inmost few stack frames of our stack trace too.

This feels likely to be an issue in RN, or possibly react-native-webview -- it doesn't seem like it should be possible for us to trigger this from JS if those layers are implemented correctly. But I'm also not sure quite what the issue means. Maybe a bunch of what's happening in this stack trace should be happening on the UI thread, not a worker thread? But also, what even is happening -- why for example is something on a TextView interacting with something on the WebView?

gnprice commented 4 years ago

I spent some time this afternoon digging into what's happening here. I haven't fully unraveled it, but here's some notes.

BTW here's that stack trace, for reference here:

java.lang.RuntimeException: Probable deadlock detected due to WebView API being called on incorrect thread while the UI thread is blocked.
    at Kq.a(PG:13)
    at com.android.webview.chromium.WebViewChromium.onCheckIsTextEditor(PG:4)
    at android.webkit.WebView.onCheckIsTextEditor(WebView.java:3035)
    at android.view.inputmethod.InputMethodManager.checkFocusNoStartInput(InputMethodManager.java:1901)
    at android.view.inputmethod.InputMethodManager.checkFocus(InputMethodManager.java:1863)
    at android.view.inputmethod.InputMethodManager.restartInput(InputMethodManager.java:1584)
    at android.widget.TextView.setText(TextView.java:6191)
    at android.widget.TextView.setText(TextView.java:6124)
    at android.widget.EditText.setText(EditText.java:122)
    at android.widget.TextView.setText(TextView.java:6076)
    at com.facebook.react.views.textinput.ReactTextInputLocalData.apply(ReactTextInputLocalData.java:42)
    at com.facebook.react.views.textinput.ReactTextInputShadowNode.measure(ReactTextInputShadowNode.java:105)
    at com.facebook.yoga.YogaNode.measure(YogaNode.java:740)
    at com.facebook.yoga.YogaNode.jni_YGNodeCalculateLayout(YogaNode.java)
    at com.facebook.yoga.YogaNode.calculateLayout(YogaNode.java:280)
    at com.facebook.react.uimanager.ReactShadowNodeImpl.calculateLayout(ReactShadowNodeImpl.java:421)
    at com.facebook.react.uimanager.UIImplementation.calculateRootLayout(UIImplementation.java:968)
    at com.facebook.react.uimanager.UIImplementation.updateViewHierarchy(UIImplementation.java:710)
    at com.facebook.react.uimanager.UIImplementation.dispatchViewUpdates(UIImplementation.java:670)
    at com.facebook.react.uimanager.UIImplementation.dispatchViewUpdatesIfNeeded(UIImplementation.java:685)
    at com.facebook.react.uimanager.UIImplementation.setViewLocalData(UIImplementation.java:272)
    at com.facebook.react.uimanager.UIManagerModule$3.runGuarded(UIManagerModule.java:449)
    at com.facebook.react.bridge.GuardedRunnable.run(GuardedRunnable.java:24)
    at android.os.Handler.handleCallback(Handler.java:883)
    at android.os.Handler.dispatchMessage(Handler.java:100)
    at com.facebook.react.bridge.queue.MessageQueueThreadHandler.dispatchMessage(MessageQueueThreadHandler.java:29)
    at android.os.Looper.loop(Looper.java:241)
    at com.facebook.react.bridge.queue.MessageQueueThreadImpl$4.run(MessageQueueThreadImpl.java:232)
    at java.lang.Thread.run(Thread.java:919)

java.util.concurrent.TimeoutException: null
    at java.util.concurrent.FutureTask.get(FutureTask.java:206)
    at Kq.a(PG:11)
    at com.android.webview.chromium.WebViewChromium.onCheckIsTextEditor(PG:4)
    [... then the rest of the frames all repeat too ...]

(cont'd)

gnprice commented 4 years ago

Then,

chrisbobbe commented 4 years ago

Wow — thanks for this fantastically detailed investigation and writeup, @gnprice.

[Two changes in Chromium] look highly relevant [...] But the timing doesn't really work for either of them to be the actual trigger of us starting to see this bug. It's possible that the bits of code they change are involved, but some other change would have to be the trigger. It's also possible that those two changes are just a red herring.

This sounds like the conclusion; a Chrome update may be combined with another, unknown, factor, to cause #4033, or it might be irrelevant to #4033.

gnprice commented 4 years ago
  • Looking at relevant bits of the Chromium tree (with git log -L in particular), here's a change that looks potentially relevant. It causes the callback of an evaluateJavaScript call to be invoked on the current thread, where previously it was invoked on the UI thread. That's new in 84.0.4118.0.

    • Which is still a canary version -- not something anyone's likely to be running unless they're working on Chromium. And the commit is dated 2020-04-16, just the day before this issue was filed and a week after those first errors in Sentry.

I had another thought about this in a conversation today with @chrisbobbe: one plausible story which would make this so-relevant-looking commit and its timing not a coincidence is:

If that is what happened, then that would make it pretty likely that Chrome 84 fixes the problem! So, if we ever acquire a way to reproduce the issue -- or if we can get a user who's regularly seeing the issue to try this -- a useful line of investigation would be to upgrade to Chrome 84, which is now in the "dev" channel. See the end of our platform-versions.md for details.

AFAICT we don't have any direct evidence that the errors we're seeing are related to evaluateJavaScript. But there are these suggestive pointers in that direction:

gnprice commented 4 years ago

An update here: Chrome 83 for Android has been rolling out since 2020-06-08 ("it'll become available on Google Play over the next few weeks"), and Chrome 84 is now in beta.

EDIT 2020-06-30: Actually it looks like each of those happened a bit sooner than that: Chrome 83 for Android reached stable on 2020-05-19, and Chrome 84 reached beta on 2020-05-28. The later announcements were for patch releases. (The text of the announcements is pretty misleading about this; now I know, I guess. For future reference here's the feeds for Chrome Beta for Android releases and Chrome [stable] for Android releases.)

zhuangzhitu commented 4 years ago

An update here: Chrome 83 for Android has been rolling out since 2020-06-08 ("it'll become available on Google Play over the next few weeks"), and Chrome 84 is now in beta.

Hi, how do you know this method is related to evaluateJavaScript? In which code does Chrome 84 fix the problem, thank you

gnprice commented 4 years ago

Hi, how do you know this method is related to evaluateJavaScript?

I don't know that this error is related to evaluateJavaScript. I do think it's likely that it's related, for the reasons explained in https://github.com/zulip/zulip-mobile/issues/4051#issuecomment-622178004 .

In which code does Chrome 84 fix the problem, thank you

The fix in Chrome 84 that I think potentially fixes this issue is the one linked at the top of that comment https://github.com/zulip/zulip-mobile/issues/4051#issuecomment-622178004. Here's a direct link: https://github.com/chromium/chromium/commit/ebaae77df0baa249fae314decaaf9382d207712d#diff-fc6b16862b884d41c2a923e5e945a674

@zhuangzhitu Are you using Zulip, and seeing symptoms that you think may be connected to this issue? If so I'd be very interested in hearing more, and in your help trying some things to help debug. Are the symptoms you're seeing similar to #4033, or something else?

zhuangzhitu commented 4 years ago

Hi, how do you know this method is related to evaluateJavaScript?

I don't know that this error is related to evaluateJavaScript. I do think it's likely that it's related, for the reasons explained in #4051 (comment) .

In which code does Chrome 84 fix the problem, thank you

The fix in Chrome 84 that I think potentially fixes this issue is the one linked at the top of that comment #4051 (comment). Here's a direct link: chromium/chromium@ebaae77#diff-fc6b16862b884d41c2a923e5e945a674

@zhuangzhitu Are you using Zulip, and seeing symptoms that you think may be connected to this issue? If so I'd be very interested in hearing more, and in your help trying some things to help debug. Are the symptoms you're seeing similar to #4033, or something else?

I encountered a similar problem with webView, but this repository is irrelevant, and I wanted to see how I could solve this problem. It all happened with Android 10 system and a lot of crashes occurred, but the local recurrence was not found. Collapse of threads are InputConnectionHandlerThread, I doubt that I don't know where got stuck, causing the message hasn't been scheduling, basic it is stuck a little more than 4 s, 4.09 s, for example, I ask you have no corresponding train of thought? The error stack is:

0 Java. Lang. RuntimeException: Probable deadlock detected due to the WebView API being called on incorrect thread while the UI thread is blocked. 1 The at com.org.chromium.android_webview.WebViewChromiumRunQueue.runBlockingFuture (Zc. Java: 13) 2 At com.org. Webview. Chromium. WebViewChromiumFactoryProvider. RunOnUiThreadBlocking (WebViewChromiumFactoryProvider. Java: 30) 3 At com. org.Webview. Chromium. WebViewChromium. OnCheckIsTextEditor (WebViewChromium. Java: 3) 4 An android. Its. WebView. OnCheckIsTextEditor (WebView. Java: 3058) 5 An android. View. Inputmethod. InputMethodManager. CheckFocusNoStartInput (InputMethodManager. Java: 1909) 6 An android. View. Inputmethod. InputMethodManager. CheckFocus (InputMethodManager. Java: 1871) 7 An android. View. Inputmethod. InputMethodManager. UpdateSelection (InputMethodManager. Java: 2091) 8 The at com.org.chromium.content.browser.input.InputMethodManagerWrapperImpl.updateSelection (ak. Java: 13) 9 The at com.org.chromium.content.browser.input.ImeAdapterImpl.onCurrentModeChanged (ImeAdapterImpl. Java: 29) onDIPScaleChanged onDisplayModesChanged onRotationChanged onWindowAndroidChanged updateSelection 10 The at com.org.chromium.content.browser.input.ThreadedInputConnection.updateSelection (Rk Java: 13) 11 The at com.org.chromium.content.browser.input.ThreadedInputConnection.processPendingInputStates (Rk Java: 9) The access of $000 12 The at com.org.chromium.content.browser.input.ThreadedInputConnection$2.run (Jk. Java: 1) 13 An android. OS. Handler. HandleCallback (Handler. Java: 888) 14 An android. OS. Handler. DispatchMessage (Handler. Java: 100) 15 . An android OS. Stars. Loop (213). Which Java: 16 An android. OS. HandlerThread. Run (HandlerThread. Java: 67) 17 Under Caused by: Java. Util. Concurrent. TimeoutException 18 The at Java. Util. Concurrent. FutureTask. Get (FutureTask. Java: 206) 19

(edited to add <details> wrapper --gnprice)

zhuangzhitu commented 4 years ago

Are you using Zulip, and seeing symptoms that you think may be connected to this issue? If so I'd be very interested in hearing more, and in your help trying some things to help debug. Are the symptoms you're seeing similar to #4033, or something else?

Can you provide more analysis? thank you

gnprice commented 4 years ago

@zhuangzhitu It looks like the top few frames of the stack trace you're seeing are the same as ours, from checkFocus inward.

If I understand you right, you've said you can't reproduce the issue you're seeing. We've been unable to reproduce the issue we're seeing too. That does make it harder to debug.

If you do manage to reproduce it, or if you're in contact with someone using your app who is regularly seeing your issue, I'd suggest installing (or getting them to install) Chrome 84 from beta, as described at https://github.com/zulip/zulip-mobile/issues/4033#issuecomment-622173988 . (That comment says the dev channel, but it's now reached the beta channel.) As discussed above, I have a guess that Chrome 84 may fix our issue, and quite possibly yours as well. But it's only a guess -- so if you end up managing to confirm whether it fixes your issue or doesn't fix it, I'll be interested in hearing the result.

If that guess is right, then the good news is that within a few weeks Chrome 84 for Android should be released to stable and (almost) all our users will get it. So if we don't manage to confirm before then whether it fixes the issue... then we'll find out then for sure.

Can you provide more analysis? thank you

I don't think I have any analysis to add that isn't above. I wish you luck in reproducing your issue, or in learning more about it. And whatever we find out about our issue in the future, it's going to wind up on this issue thread, so I think you'll learn about it as a result.

3c133ps3d commented 4 years ago

I spent a lot time trying to fix this problem. Solution is simple, just call: (context.getSystemService(Context.INPUT_METHOD_SERVICE) as InputMethodManager).restartInput(this) at your react-native's onDropViewInstance method.

gnprice commented 4 years ago

@3c133ps3d Very interesting, thanks for the tip! Can you say more (or is there a writeup elsewhere you can point to) about how you discovered that solution, or how you determined that it was a solution for this issue?


Meanwhile, Sentry tells us that the rate at which our users were seeing this dropped sharply around 2020-06-27: image That coincides with when we released our v26.29.152 (the rollout to most users is generally a few days after the headline date of the release). Which included an upgrade to RN v0.60, so that's certainly a change that would plausibly affect this issue. Sadly it still happens, though, just at a much lower rate, and it's still happening to users on v27.154, our latest.

Meanwhile Chrome 84 for Android rolled out as stable starting 2020-07-14, so almost all our Android users should have it by now. It doesn't appear to have affected the rate of this error. That probably means that the Chrome bugfix I mentioned above at https://github.com/zulip/zulip-mobile/issues/4051#issuecomment-616855833 is not in fact related.

gnprice commented 3 years ago

We're continuing to see this in Sentry, but now at a low rate where it's affecting many fewer users than it originally did.

mym0404 commented 1 year ago

Thank you for detailed tracking.

I suffered this issue recently because of conditional(if-else) rendering of WebView in my RN project. RN 0.70.1 RNWebView 11.18.1

Yup, it was an Android Q(10) issue.

My WebView use injectJavaScript imperative method of RN webview that eventually call this.

Issue was fixed by removing conditional rendering of webview. Instead I mount a webview and change content of WebView with injectJavaScript. If need hide content, I just manipulate opacity and pointerEvent instead of completely unmount.

Screen lifecycle (Issued)

Mount -> Unmount -> Mount -> Unmount -> .... -> Unmount

Screen lifecycle (issue gone)

Mount -> Hide(`opacity`, `pointerEvent`) -> Show -> Hide -> .... -> Unmount
Vatousiadis commented 1 year ago

(Irrelevant to this repo) For the past week we started seeing this error too in Android 10 devices consistently. What is weird is that in 2 years of working with webview it is the 1st time we see this issue and it happenned started happening out of the blue, since we have not changed our webview's implementation in months. I am battleing this crash for a week now, so any kind of help woudl be greatly appreciated, below is our stacktrace

GC0 in c at line 47 com.android.webview.chromium.WebViewChromiumFactoryProvider in h at line 8 com.android.webview.chromium.h in a at line 6 com.android.webview.chromium.WebViewChromium in onCheckIsTextEditor at line 17 android.webkit.WebView in onCheckIsTextEditor at line 3041 android.view.inputmethod.InputMethodManager in checkFocusNoStartInput at line 1906 android.view.inputmethod.InputMethodManager in checkFocus at line 1868 android.view.inputmethod.InputMethodManager in restartInput at line 1584 android.widget.TextView in setText at line 6191 android.widget.TextView in setText at line 6124 android.widget.EditText in setText at line 122 android.widget.TextView in <init> at line 1616 android.widget.EditText in <init> at line 88 android.widget.EditText in <init> at line 84 android.widget.EditText in <init> at line 80 android.widget.EditText in <init> at line 76 com.facebook.react.views.textinput.ReactTextInputShadowNode in createInternalEditText at line 265 com.facebook.react.views.textinput.ReactTextInputShadowNode in setThemedContext at line 85 com.facebook.react.uimanager.UIImplementation in createView at line 255 com.facebook.react.uimanager.UIManagerModule in createView at line 496 com.facebook.react.uimanager.ReanimatedUIManager in createView at line 72 java.lang.reflect.Method in invoke com.facebook.react.bridge.JavaMethodWrapper in invoke at line 372 com.facebook.react.bridge.JavaMethodWrapper in invoke at line 372 com.facebook.react.bridge.JavaModuleWrapper in invoke at line 188 com.facebook.jni.NativeRunnable in run android.os.Handler in handleCallback at line 883 android.os.Handler in dispatchMessage at line 100 com.facebook.react.bridge.queue.MessageQueueThreadHandler in dispatchMessage at line 27 android.os.Looper in loop at line 224 com.facebook.react.bridge.queue.MessageQueueThreadImpl$4 in run at line 228 java.lang.Thread in run at line 919

jja08111 commented 7 months ago

Same issue here.

Fatal Exception: java.lang.RuntimeException: Probable deadlock detected due to WebView API being called on incorrect thread while the UI thread is blocked.
       at WV.XX.c(chromium-TrichromeWebViewGoogle6432.aab-stable-609923033:44)
       at com.android.webview.chromium.WebViewChromiumFactoryProvider.h(chromium-TrichromeWebViewGoogle6432.aab-stable-609923033:8)
       at com.android.webview.chromium.h.a(chromium-TrichromeWebViewGoogle6432.aab-stable-609923033:6)
       at com.android.webview.chromium.WebViewChromium.onCheckIsTextEditor(chromium-TrichromeWebViewGoogle6432.aab-stable-609923033:19)
       at android.webkit.WebView.onCheckIsTextEditor(WebView.java:3054)
       at android.view.inputmethod.InputMethodManager.checkFocusNoStartInput(InputMethodManager.java:2494)
       at android.view.inputmethod.InputMethodManager.checkFocus(InputMethodManager.java:2451)
       at android.view.inputmethod.InputMethodManager.restartInput(InputMethodManager.java:2037)
       at android.widget.TextView.setText(TextView.java:6697)
       at android.widget.TextView.setText(TextView.java:6630)
       at android.widget.EditText.setText(EditText.java:140)
       at android.widget.TextView.<init>(TextView.java:1879)
       at android.widget.EditText.<init>(EditText.java:106)
       at android.widget.EditText.<init>(EditText.java:102)
       at android.widget.EditText.<init>(EditText.java:98)
       at android.widget.EditText.<init>(EditText.java:94)
       at com.facebook.react.views.textinput.ReactTextInputShadowNode.createInternalEditText(ReactTextInputShadowNode.java:236)
       at com.facebook.react.views.textinput.ReactTextInputShadowNode.setThemedContext(ReactTextInputShadowNode.java:82)
       at com.facebook.react.uimanager.UIImplementation.createView(UIImplementation.java:239)
       at com.facebook.react.uimanager.UIManagerModule.createView(UIManagerModule.java:419)
       at com.facebook.react.uimanager.ReanimatedUIManager.createView(ReanimatedUIManager.java:51)
       at java.lang.reflect.Method.invoke(Method.java)
       at com.facebook.react.bridge.JavaMethodWrapper.invoke(JavaMethodWrapper.java:372)
       at com.facebook.react.bridge.JavaModuleWrapper.invoke(JavaModuleWrapper.java:188)
       at com.facebook.jni.NativeRunnable.run(NativeRunnable.java)
       at android.os.Handler.handleCallback(Handler.java:883)
       at android.os.Handler.dispatchMessage(Handler.java:100)
       at com.facebook.react.bridge.queue.MessageQueueThreadHandler.dispatchMessage(MessageQueueThreadHandler.java:27)
       at android.os.Looper.loop(Looper.java:237)
       at com.facebook.react.bridge.queue.MessageQueueThreadImpl$4.run(MessageQueueThreadImpl.java:228)
       at java.lang.Thread.run(Thread.java:919)
Caused by java.util.concurrent.TimeoutException:
       at java.util.concurrent.FutureTask.get(FutureTask.java:206)
       at WV.XX.c(chromium-TrichromeWebViewGoogle6432.aab-stable-609923033:25)
       at com.android.webview.chromium.WebViewChromiumFactoryProvider.h(chromium-TrichromeWebViewGoogle6432.aab-stable-609923033:8)
       at com.android.webview.chromium.h.a(chromium-TrichromeWebViewGoogle6432.aab-stable-609923033:6)
       at com.android.webview.chromium.WebViewChromium.onCheckIsTextEditor(chromium-TrichromeWebViewGoogle6432.aab-stable-609923033:19)
       at android.webkit.WebView.onCheckIsTextEditor(WebView.java:3054)
       at android.view.inputmethod.InputMethodManager.checkFocusNoStartInput(InputMethodManager.java:2494)
       at android.view.inputmethod.InputMethodManager.checkFocus(InputMethodManager.java:2451)
       at android.view.inputmethod.InputMethodManager.restartInput(InputMethodManager.java:2037)
       at android.widget.TextView.setText(TextView.java:6697)
       at android.widget.TextView.setText(TextView.java:6630)
       at android.widget.EditText.setText(EditText.java:140)
       at android.widget.TextView.<init>(TextView.java:1879)
       at android.widget.EditText.<init>(EditText.java:106)
       at android.widget.EditText.<init>(EditText.java:102)
       at android.widget.EditText.<init>(EditText.java:98)
       at android.widget.EditText.<init>(EditText.java:94)
       at com.facebook.react.views.textinput.ReactTextInputShadowNode.createInternalEditText(ReactTextInputShadowNode.java:236)
       at com.facebook.react.views.textinput.ReactTextInputShadowNode.setThemedContext(ReactTextInputShadowNode.java:82)
       at com.facebook.react.uimanager.UIImplementation.createView(UIImplementation.java:239)
       at com.facebook.react.uimanager.UIManagerModule.createView(UIManagerModule.java:419)
       at com.facebook.react.uimanager.ReanimatedUIManager.createView(ReanimatedUIManager.java:51)
       at java.lang.reflect.Method.invoke(Method.java)
       at com.facebook.react.bridge.JavaMethodWrapper.invoke(JavaMethodWrapper.java:372)
       at com.facebook.react.bridge.JavaModuleWrapper.invoke(JavaModuleWrapper.java:188)
       at com.facebook.jni.NativeRunnable.run(NativeRunnable.java)
       at android.os.Handler.handleCallback(Handler.java:883)
       at android.os.Handler.dispatchMessage(Handler.java:100)
       at com.facebook.react.bridge.queue.MessageQueueThreadHandler.dispatchMessage(MessageQueueThreadHandler.java:27)
       at android.os.Looper.loop(Looper.java:237)
       at com.facebook.react.bridge.queue.MessageQueueThreadImpl$4.run(MessageQueueThreadImpl.java:228)
       at java.lang.Thread.run(Thread.java:919)