firebase / firebase-android-sdk

Firebase Android SDK
https://firebase.google.com
Apache License 2.0
2.28k stars 580 forks source link

App crashing due to IllegalStateException #1339

Closed sudsingh438 closed 4 years ago

sudsingh438 commented 4 years ago

[REQUIRED] Step 2: Describe your environment

[REQUIRED] Step 3: Describe the problem

Crashes are being reported on play console with below stack trace:

java.lang.IllegalStateException: at com.google.firebase.iid.FirebaseInstanceId.zzl (FirebaseInstanceId.java:75) at com.google.firebase.iid.FirebaseInstanceId.getId (FirebaseInstanceId.java:49) at com.google.firebase.perf.internal.zzf.zzbu (zzf.java:181) at com.google.firebase.perf.internal.zzf.zzbt (zzf.java:44) at com.google.firebase.perf.internal.zzf.zzc (zzf.java:195) at com.google.firebase.perf.internal.zze.run (zze.java:2) at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1167) at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:641) at java.lang.Thread.run (Thread.java:764)

Steps to reproduce:

We are not able to reproduce it locally.

google-oss-bot commented 4 years ago

I found a few problems with this issue:

andirayo commented 4 years ago

@sudsingh438 :

Questions

  1. Which version of the Firebase Instance ID are you using?
  2. Are you able to share the complete IllegalStateException stack trace that might show the cause?
  3. When initializing Firebase (i.e. class FirebaseApp) are you initializing with a full set of Firebase options (e.g. using a google-services.json from the Firebase Console)?

Background

If the initialization of Firebase (i.e. instance of FirebaseApp) is missing one of the following required parameters, this might explain the issue: Application ID, Project ID, API key.

The latest version of the Instance ID SDK depends on the new Firebase Installations SDK which is a change that might cause issues like this.

(@ankitaj224 : for visibility)

sudsingh438 commented 4 years ago

@andirayo

1) Firebase instance id version : 20.1.1 2) The one that I shared is the description is the complete stack trace from play console.

Screenshot 2020-03-12 at 1 41 45 AM

This stack trace is the one we got from Sentry

java.lang.IllegalStateException: null
    at com.google.firebase.iid.FirebaseInstanceId.o(com.google.firebase:firebase-iid@@20.1.1:12)
    at com.google.firebase.iid.FirebaseInstanceId.a(com.google.firebase:firebase-iid@@20.1.1:15)
    at com.google.firebase.perf.internal.f.c(com.google.firebase:firebase-perf@@19.0.5:5)
    at com.google.firebase.perf.internal.f.b(com.google.firebase:firebase-perf@@19.0.5:12)
    at com.google.firebase.perf.internal.f.a(com.google.firebase:firebase-perf@@19.0.5:52)
    at com.google.firebase.perf.internal.e.run(com.google.firebase:firebase-perf@@19.0.5:1)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
    at java.lang.Thread.run(Thread.java:764)

3) Yes we are using google-services.json from play console. We started seeing this issue in our app from last 1 week, before that we didnt encounter this issue before that.

Also, if it helps we are using below mentioned firebase component with version: firebase-config:19.1.2(This we recently upgraded from 19.1.0) firebase-dynamic-links:19.1.0 firebase-messaging:20.1.2(This we recently upgraded from 19.0.3) firebase-perf:19.0.5

yoavgray commented 4 years ago

We have the exact same issue and it all started after we upgraded to:

com.google.firebase:firebase-bom:25.0.0

from 24.7.0

asfdfdfd commented 4 years ago

I have the same crash outburst. It started this week. I've updated:

com.google.firebase:firebase-messaging:20.1.0 -> 20.1.2
com.google.firebase:firebase-core:17.2.2 -> 17.2.3
com.google.firebase:firebase-config:19.1.1 -> 19.1.2

Downgraded today, will see if this going to fix situation.

I create custom Firebase instance with code that looks something like this:

val firebaseOptions = FirebaseOptions.fromResource(context)
FirebaseApp.initializeApp(context, firebaseOptions, "MyCustomInstance")

But Firebase Performance uses default Firebase instance.

FirebasePerformance.getInstance()
roiamiel1 commented 4 years ago

@asfdfdfd did downgrading solved this bug for you?

MattSkala commented 4 years ago

We started to experience the same crash after updating the following dependencies:

com.google.firebase:firebase-analytics 17.2.2 -> 17.2.3
com.google.firebase:firebase-messaging 20.1.0 -> 20.1.2
com.google.firebase:firebase-config 19.1.1 -> 19.1.2
com.google.firebase:firebase-firestore 21.4.0 -> 21.4.1
andirayo commented 4 years ago

First of all: We are very sorry for the inconvenience!

A downgrade to the old version of the SDKs, particularly a version of FirebaseInstanceId (a.k.a. firebase-iid) before 20.1.0 should solve the issue.

We are trying to identify the problem. @asfdfdfd : We expect this or similar problems to occur if the FirebaseOptions are initialized without mandatory parameters Project ID, Application ID, or API key. However judging from @sudsingh438 's response before, I don't believe this is the issue here.

We believe that the underlying FirebaseInstallations SDK which is a new dependency of FirebaseInstanceId is causing this problem.

Is anybody able to reproduce this issue locally and can share with us steps to reproduce it?

kozaxinan commented 4 years ago

We are also not able to reproduce this locally but we have some extra information, 97% of crashes happen in background. So I am not even sure that users get any crash dialog at all.

sudsingh438 commented 4 years ago

For us, also around 85%+ crashes are reported in background

andirayo commented 4 years ago

Thanks for the information, @kozaxinan , @sudsingh438 ! We would appreciate any information that could help us pinpoint this problem:

Q1) Are your applications actually crashing (or is there a background thread crashing, but the application continues to run)?

Q2) How many crashes per app usage are there? If you write "97% of crashes" and "85%+ crashes" are you referring to 97%/85% of these IllegalStateException crashes or of all your app crashes? And (repeating Q1) by "crash" do you mean that the application is actually stopping?

Q3) Are the crashes happening randomly for end-users or repeatedly happening for the same end-user (e.g. crash-looping)?

Q4) If you are willing to share your Application ID or Project ID (which is non-secret information about your application), we would be able to look into our logs to see if the request patterns from your application to Firebase look healthy.

MattSkala commented 4 years ago

@andirayo Q1) From the stack trace it looks like a background thread, but I can't tell if the application continues to run after the crash. In Crashlytics, I can see that 80% of those crashes occur when the application is in background. After examining the logs, it looks like some of those crashes occurred around the time when those users received push notifications, if it helps. Q2) The crash is relatively rare. It is our most frequently occurring crash this week, but we still have 99.9% crash-free users. Q3) There are 51 reported crashes reported by 48 users for the last 7 days, so it usually happens only once to each user. Q4) Feel free to check our App ID: 1:499050530164:android:1508e54a25da7ef9

ygnessin commented 4 years ago

@andirayo thank you for looking into this. Hopefully this information is helpful:

Q1) I can't tell for sure but just like @MattSkala said the thread that is crashing does seem like a background thread. The trace is identical to the one in the issue description. Based on our logs it appears that some of the crashes might be occurring right after a push notification is received, but not all of them display that. Furthermore, we see the number of crashes increase significantly around the time that our daily push notifications are sent. However, this usually correlates with a rise in traffic anyway, so it's hard to say for sure that pushes are the cause.

Q2) For us the crash is affecting close to 1% of our total users. So while not super common, still high enough to cause serious concern (we'll be rolling back until there is a fix).

Q3) So far we have seen 128 crashes across 108 users. So they seem to be mostly unique, with a little bit of overlap.

Q4) I will follow up with my team and get back to you.

andirayo commented 4 years ago

@MattSkala, @ygnessin : Thank you so much for sharing this information. That really helps to better understand the situation.

@MattSkala : I looked into our server logs for the requests regarding Firebase identifiers from your application and can see exclusively successful requests. I will involved the FCM/IID team into this, given that this might be related to push notifications arriving (while the application is in the background).

maxkohne commented 4 years ago

we had just released a new version of the app with these new versions when we started seeing this crash: com.google.firebase:firebase-messaging from 20.1.0 -> 20.1.2 com.google.firebase:firebase-analytics from 17.2.2 -> 17.2.3

we were already using com.google.firebase:firebase-perf 19.0.5 before the crash happened so we have ruled out that it isn't that library.

we have decided to downgrade messaging and analytics for the time being while this is being investigated

andirayo commented 4 years ago

@maxkohne , @ygnessin , and others: By any chance, are you able to also share a StackTrace with us?

maxkohne commented 4 years ago

same exact stacktrace as what the author posted

Fatal Exception: java.lang.IllegalStateException at com.google.firebase.iid.FirebaseInstanceId.zzl(FirebaseInstanceId.java:75) at com.google.firebase.iid.FirebaseInstanceId.getId(FirebaseInstanceId.java:49) at com.google.firebase.perf.internal.zzf.zzbu(zzf.java:181) at com.google.firebase.perf.internal.zzf.zzbt(zzf.java:44) at com.google.firebase.perf.internal.zzf.zzc(zzf.java:195) at com.google.firebase.perf.internal.zze.run(zze.java:2) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641) at java.lang.Thread.run(Thread.java:764)

andirayo commented 4 years ago

@maxkohne : Thank you so much!

andirayo commented 4 years ago

internal: b/151480886

andirayo commented 4 years ago

Update

We identified that the IllegalStateException is thrown in FirebaseInstanceId.getResultOrThrowException (FirebaseInstanceId.java:486) where task.getException() returns NULL.

We currently believe that some form of race condition and/or deadlock causes this issue.

We assume that the race condition occurs when multiple Firebase services require a client identifier (a.k.a. FID / Instance ID) at the same time, possibly in different threads or even different processes. In the situation that multiple Firebase SDKs call FirebaseInstanceId.getId in parallel, a race condition might occur that leads to a deadlock. Once the deadline for waiting for the identifier times out, an IllegalStateException is thrown.

Background

FirebaseInstanceId is an infrastructure service that is used by various Firebase services like Messaging (FCM), Google Analytics, and Performance Monitoring (FirePerf). With the latest version (v20.1.0), FirebaseInstanceId introduced a new dependency: The Firebase Installations SDK (FIS). We assume that the deadlock occurs in FIS and it is FirebaseInstanceId that times out waiting for the identifier, the response of FIS.

Next steps

To fix this issue, we need to be able to reproduce the issue locally.

At this point, we believe that an incoming push notification wakes up, i.e. triggers processes in an application that is running in background. These processes could include Google Analytics recording the receival of the push notification and Firebase Performance Monitoring (FirePerf) recording some event or latency. Thus, such an application would have to use at least Messaging (FCM) to receive push notifications and two Firebase services like Google Analytics and FirePerf that record events and thus need access to the client identifier.

Questions

Generally: Any information to achieve reproducing this crash would be highly appreciated!

Q5) Could you please share your Gradle Android dependency tree with us: We would like to understand which Firebase SDKs (and which versions) you are using in your application.

Q6) Do you happen to have access to a full Android bugreport (including errors and logs from all threads)?

MattSkala commented 4 years ago

@andirayo Thanks for the update

Q5) Our dependency tree obtained from ./gradlew dependencies: https://pastebin.com/A8uGPzHh Q6) I only have access to what I see in Crashlytics, these are stack traces for all threads: https://pastebin.com/XTve0qt3

sudsingh438 commented 4 years ago

@andirayo Please find the dependency list and stacktrace of one of the issue Dependency list : https://pastebin.com/gcRZyxNQ Stacktrace from crashlytics for all thread : https://pastebin.com/uJw7LgJm

kozaxinan commented 4 years ago

@andirayo We have more than 380 crashes. 98% of them are in background. According to threads, it is happening when we have notification. App uses network before notification displayed to fetch some information.

Q1) Not sure, we have crash report in firebase but not it looks like crash happens when notification received. FirebaseMessagingService is running at the moment of report.

Q2) 387 crashes for 370 reports. This 50 times more than 2nd most occurring crash. 98% percent of 387 reports happened in background. I am not sure but probably firebase messaging service fails to continue after this report.

Q3) Random users. 387 crashes for 370 reports

Q4) AppId : 1:208472424340:android:a243f98a00873753

Q5) Firebase dependencies :

object Firebase {

      const val analytics = "com.google.firebase:firebase-analytics:17.2.3"
      const val config = "com.google.firebase:firebase-config:19.1.2"
      const val core = "com.google.firebase:firebase-core:17.2.3"
      const val messaging = "com.google.firebase:firebase-messaging:20.1.2"
      const val perf = "com.google.firebase:firebase-perf:19.0.5"
    }

Q6) Stacktrace from firebase for all thread : https://0bin.net/paste/XYQs-BXx0WRDEe+g#F662Og7LOng22wN2W+Ob7UrBPg+5vOamqked4eRCex0

andirayo commented 4 years ago

@MattSkala , @sudsingh438 : Thanks so much for the stack traces! These are really valuable to us as they show what is happening in FirebaseInstallations during the crash.

@kozaxinan : Thanks a lot for the information. This is very helpful as it confirms our current assumptions. I also checked all client-server requests for the given AppId. Except for 1 request all ended successfully and thus I don't think server communication is the cause for this.

If possible, could you please resent the link to your stack trace as the link above does not seem to work for me.

kozaxinan commented 4 years ago

@andirayo I updated the link.

andirayo commented 4 years ago

@kozaxinan : Thank you for the updated link!

@MattSkala , @sudsingh438, @kozaxinan, and @all : The stack traces containing all threads are the most helpful for us right now.

We would appreciate if you find the time to share more stack traces, given that the actual call stacks of the main thread seem to differ between cases, and also the call stacks of FirebaseInstallations differ.

asfdfdfd commented 4 years ago

@roiamiel1 yes, downgrade solved the problem.

@andirayo

Q1) I don't know, because i was not able to reproduce it on my devices. 73% of affected devices developed by HUAWEI, 64% is Android 9, 78% background.

Q2) Crashlytics told me that "This issue has 1569 crashes affecting 1348 users".

Q3) Some users has repeated crashes. For example one user has 4 crashes one after another in one minute.

Q4) Project ID "mirror-4543d", Application ID "com.mirrorai.app"

Q5) dependencies.txt

Q6) com.mirrorai.app_1769554677256386823(2).txt

com.mirrorai.app_1769554677256386823(1).txt

com.mirrorai.app_1769554677256386823.txt

andirayo commented 4 years ago

@asfdfdfd : Thank you so much, especially for sending 3 stack traces! That is very helpful. Also thanks for the crash details! I looked into our server logs regarding your Project ID. Almost all requests are successful. There are a few auth errors that I will look into, but they are unrelated to this issue.

andirayo commented 4 years ago

Bandaid Fix

The best solution for this problem currently still is to downgrade to the old version of the SDKs, particularly a version of FirebaseInstanceId (a.k.a. firebase-iid) before v20.1.0.

Update

We were not able to make much progress on the weekend, but I am positive that we will make progress tomorrow when we can coordinate with the teams of Firebase Messaging (FCM) and Firebase Performance Monitoring (FirePerf).

Assumptions

Ideas

Next steps

sudsingh438 commented 4 years ago

@andirayo We have observed few IllegalStateException crashes are happening after null pointer exception is being thrown by the perf. This is from one of the logs

2020-03-15T21:12:42:096 GMT+05:30 - E - java.lang.NullPointerException: Attempt to invoke virtual method 'void com.google.firebase.perf.internal.v.a(boolean)' on a null object reference
    at com.google.firebase.perf.internal.f.b(com.google.firebase:firebase-perf@@19.0.5:74)
    at com.google.firebase.perf.internal.i.run(com.google.firebase:firebase-perf@@19.0.5:1)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1162)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:636)
    at java.lang.Thread.run(Thread.java:764)

2020-03-15T21:12:43:450 GMT+05:30 - E - java.lang.IllegalStateException
    at com.google.firebase.iid.FirebaseInstanceId.o(com.google.firebase:firebase-iid@@20.1.1:12)
    at com.google.firebase.iid.FirebaseInstanceId.a(com.google.firebase:firebase-iid@@20.1.1:15)
    at com.google.firebase.perf.internal.f.c(com.google.firebase:firebase-perf@@19.0.5:5)
    at com.google.firebase.perf.internal.f.b(com.google.firebase:firebase-perf@@19.0.5:12)
    at com.google.firebase.perf.internal.f.a(com.google.firebase:firebase-perf@@19.0.5:52)
    at com.google.firebase.perf.internal.e.run(com.google.firebase:firebase-perf@@19.0.5:1)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1162)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:636)
    at java.lang.Thread.run(Thread.java:764)
Shudy commented 4 years ago

Good morning, We are also having the same/similar problems with our app.

We got 23 crash error from 23 different users in 7 days . and also 96% are in background (https://i.imgur.com/5naCAwz.png)

Pasted error trace: https://pastebin.com/x7zyapyt

kenyee commented 4 years ago

Looks like best bet for now is to lock the firebase-iid library to 20.0.2?

andirayo commented 4 years ago

@sudsingh438 : Thank you for this information. I will try to look into this! @Shudy : Thanks so much for sharing this! That is very helpful! @kenyee : I think there was a version v20.0.3 on Github, but yes v20.0.2 solves this issue.

Update

We identified a potential issue when a network connection times out or fails and multiple threads are accessing / waiting for the Firebase client identifier. We are trying to reproduce this issue in order to work out a fix for this.

andirayo commented 4 years ago

Update

guness commented 4 years ago

on our side crash was happening while trying to access the token, now seems fixed on com.google.firebase:firebase-messaging:20.1.3.


ok, update: it is still crashing on Android 10 emulator with version 20.1.3, problem seems to be still there.

andirayo commented 4 years ago

@guness :
Thank you so much for the update. Firebase Messaging v20.1.3 does not contain the fix, though.

The fix is included in Firebase Installations v16.2.0 which is already available on Github, but will only be released to Maven on Thursday, March 26.

cosgunhalil commented 4 years ago

Hi everyone, We are having problems that looks similar this problem with our app. I am not sure about the similarity, but I want to share the problem with you. You can see the details of our problem in here.

Do you have any suggestion to solve the issue that caused by java.io.IOException?

adiamartya commented 4 years ago

@andirayo We pushed an update to our app yesterday only. In this update we have used firebase-messaging:20.1.3. I am assuming that we don't have to push another update. It will be fixed automatically?

andirayo commented 4 years ago

@adiamartya : Unfortunately, Firebase Messaging v20.1.3 does not contain the fix, though.

The fix is included in Firebase Installations v16.2.0 which is already available on Github, but will only be released to Maven on Thursday, March 26. fyi: Firebase Messaging v20.1.4 will depend on Firebase Installations v16.2.0.

To fix this issue you will have to push an update that contains Firebase Installations v16.2.0 or higher.

adiamartya commented 4 years ago

@andirayo Thanks a lot for the information. So, currently we are using firebase-core v17.2.3 and we aren't using firebase installations. Shall we remove firebase-core and directly use analytics sdk and firebase installations? Also, I am assuming that we have to include firebase installations v16.2.0 anyhow.

andirayo commented 4 years ago

@adiamartya : Firebase Installations is an infrastructure service (identity provider and authentication scheme) that is used under the hood by many Firebase services (like Analytics). Usually, you do not need to explicitly include it, but you will get it as transitive dependency (through Firebase InstanceId).

To answer your question: Yes, we recommend to directly depend on the Firebase services that you are using and thus we recommend to directly depend on Firebase Analytics, rather than Core.

If you are affected by this issue, unfortunately you only have 2 options: A) Downgrade to SDKs that use a version of Firebase InstanceId before v20.1.0. The easiest way to do this is to depend on Firebase Messaging v20.1.0 (or before) and lock the version of Firebase InstanceId to v20.0.2 in your Gradle file. B) Or wait until March 26 and upgrade your SDKs to depend on Firebase InstanceId v20.1.3 of higher (depending on Firebase Installations v16.2.0 or higher).

I know this all sounds very complicated, so let me try to explain the dependency tree a bit:

A similar situation is true for Google Analytics:

But if you use Firebase Messaging (FCM) in the same application the Firebase InstanceId dependency will be resolved to v20.1.2, which is causing the issue.

adiamartya commented 4 years ago

Thanks a lot. Will wait for 26th and will push update with Firebase Messaging v20.1.4 :)

adiamartya commented 4 years ago

@andirayo Hi. Update is still not available. I am just confirming World is facing hard times. So, delay is understandable. Just confirming :)

schof commented 4 years ago

Can we can get an update on v20.1.4?

andirayo commented 4 years ago

@schof, @adiamartya: Unfortunately our M67 release was delayed due to some GMSCore issues. The release will unfortunately go out after the weekend, on March 30, 2020. Sorry!

Finalized new version numbers:

andirayo commented 4 years ago

Good news, the M67 release is completed and available on Maven. https://maven.google.com/web/index.html (fyi: Javadocs and release notes will follow soon.)

Looking forward to hear if the new release fixes the issue for you!

adiamartya commented 4 years ago

@andirayo Thanks man. Meeting deadlines during pandemic is really awesome.

schof commented 4 years ago

@andirayo quick correction to your above post. Firebase Messaging is v20.1.4 and not v20.2.2.

andirayo commented 4 years ago

@schof : You are right. I checked with the Messaging team. Some things must have moved around in the past weeks. Our release doc wasn't updated. Thanks for pointing this out! I corrected my post above.

dcampogiani commented 4 years ago

@andirayo thanks for the update.

I'm upgrading firebase-messaging from 20.1.3 to 20.1.4 but now I get this error:

Unexpected failure during lint analysis (this is a bug in lint or one of the libraries it depends on) Message: com.android.tools.lint.detector.api.Project.getGradleProjectModel()Lcom/android/builder/model/AndroidProject; The crash seems to involve the detector com.google.firebase.installations.lint.IncompatibleIidVersionDetector. You can try disabling it with something like this: android { lintOptions { disable "IncompatibleIidVersion" } }

Stack: NoSuchMethodError:IncompatibleIidVersionDetector.beforeCheckEachProject(IncompatibleIidVersionDetector.kt:56)←LintDriver.checkProject(LintDriver.kt:894)←LintDriver.analyze(LintDriver.kt:422)←LintCliClient.run(LintCliClient.java:238)←LintGradleClient.run(LintGradleClient.java:261)←LintGradleExecution.runLint(LintGradleExecution.java:305)←LintGradleExecution.lintSingleVariant(LintGradleExecution.java:392)←LintGradleExecution.analyze(LintGradleExecution.java:94)

Any idea how can I fix it?