Closed sudsingh438 closed 4 years ago
I found a few problems with this issue:
@sudsingh438 :
IllegalStateException
stack trace that might show the cause?FirebaseApp
) are you initializing with a full set of Firebase options (e.g. using a google-services.json
from the Firebase Console)?If the initialization of Firebase (i.e. instance of FirebaseApp
) is missing one of the following required parameters, this might explain the issue: Application ID, Project ID, API key.
The latest version of the Instance ID SDK depends on the new Firebase Installations SDK which is a change that might cause issues like this.
(@ankitaj224 : for visibility)
@andirayo
1) Firebase instance id version : 20.1.1 2) The one that I shared is the description is the complete stack trace from play console.
This stack trace is the one we got from Sentry
java.lang.IllegalStateException: null
at com.google.firebase.iid.FirebaseInstanceId.o(com.google.firebase:firebase-iid@@20.1.1:12)
at com.google.firebase.iid.FirebaseInstanceId.a(com.google.firebase:firebase-iid@@20.1.1:15)
at com.google.firebase.perf.internal.f.c(com.google.firebase:firebase-perf@@19.0.5:5)
at com.google.firebase.perf.internal.f.b(com.google.firebase:firebase-perf@@19.0.5:12)
at com.google.firebase.perf.internal.f.a(com.google.firebase:firebase-perf@@19.0.5:52)
at com.google.firebase.perf.internal.e.run(com.google.firebase:firebase-perf@@19.0.5:1)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
at java.lang.Thread.run(Thread.java:764)
3) Yes we are using google-services.json from play console. We started seeing this issue in our app from last 1 week, before that we didnt encounter this issue before that.
Also, if it helps we are using below mentioned firebase component with version: firebase-config:19.1.2(This we recently upgraded from 19.1.0) firebase-dynamic-links:19.1.0 firebase-messaging:20.1.2(This we recently upgraded from 19.0.3) firebase-perf:19.0.5
We have the exact same issue and it all started after we upgraded to:
com.google.firebase:firebase-bom:25.0.0
from 24.7.0
I have the same crash outburst. It started this week. I've updated:
com.google.firebase:firebase-messaging:20.1.0 -> 20.1.2
com.google.firebase:firebase-core:17.2.2 -> 17.2.3
com.google.firebase:firebase-config:19.1.1 -> 19.1.2
Downgraded today, will see if this going to fix situation.
I create custom Firebase instance with code that looks something like this:
val firebaseOptions = FirebaseOptions.fromResource(context)
FirebaseApp.initializeApp(context, firebaseOptions, "MyCustomInstance")
But Firebase Performance uses default Firebase instance.
FirebasePerformance.getInstance()
@asfdfdfd did downgrading solved this bug for you?
We started to experience the same crash after updating the following dependencies:
com.google.firebase:firebase-analytics 17.2.2 -> 17.2.3
com.google.firebase:firebase-messaging 20.1.0 -> 20.1.2
com.google.firebase:firebase-config 19.1.1 -> 19.1.2
com.google.firebase:firebase-firestore 21.4.0 -> 21.4.1
First of all: We are very sorry for the inconvenience!
A downgrade to the old version of the SDKs, particularly a version of FirebaseInstanceId
(a.k.a. firebase-iid
) before 20.1.0
should solve the issue.
We are trying to identify the problem.
@asfdfdfd : We expect this or similar problems to occur if the FirebaseOptions
are initialized without mandatory parameters Project ID, Application ID, or API key. However judging from @sudsingh438 's response before, I don't believe this is the issue here.
We believe that the underlying FirebaseInstallations
SDK which is a new dependency of FirebaseInstanceId
is causing this problem.
Is anybody able to reproduce this issue locally and can share with us steps to reproduce it?
We are also not able to reproduce this locally but we have some extra information, 97% of crashes happen in background. So I am not even sure that users get any crash dialog at all.
For us, also around 85%+ crashes are reported in background
Thanks for the information, @kozaxinan , @sudsingh438 ! We would appreciate any information that could help us pinpoint this problem:
Q1) Are your applications actually crashing (or is there a background thread crashing, but the application continues to run)?
Q2) How many crashes per app usage are there? If you write "97% of crashes" and "85%+ crashes" are you referring to 97%/85% of these IllegalStateException crashes or of all your app crashes? And (repeating Q1) by "crash" do you mean that the application is actually stopping?
Q3) Are the crashes happening randomly for end-users or repeatedly happening for the same end-user (e.g. crash-looping)?
Q4) If you are willing to share your Application ID or Project ID (which is non-secret information about your application), we would be able to look into our logs to see if the request patterns from your application to Firebase look healthy.
@andirayo Q1) From the stack trace it looks like a background thread, but I can't tell if the application continues to run after the crash. In Crashlytics, I can see that 80% of those crashes occur when the application is in background. After examining the logs, it looks like some of those crashes occurred around the time when those users received push notifications, if it helps. Q2) The crash is relatively rare. It is our most frequently occurring crash this week, but we still have 99.9% crash-free users. Q3) There are 51 reported crashes reported by 48 users for the last 7 days, so it usually happens only once to each user. Q4) Feel free to check our App ID: 1:499050530164:android:1508e54a25da7ef9
@andirayo thank you for looking into this. Hopefully this information is helpful:
Q1) I can't tell for sure but just like @MattSkala said the thread that is crashing does seem like a background thread. The trace is identical to the one in the issue description. Based on our logs it appears that some of the crashes might be occurring right after a push notification is received, but not all of them display that. Furthermore, we see the number of crashes increase significantly around the time that our daily push notifications are sent. However, this usually correlates with a rise in traffic anyway, so it's hard to say for sure that pushes are the cause.
Q2) For us the crash is affecting close to 1% of our total users. So while not super common, still high enough to cause serious concern (we'll be rolling back until there is a fix).
Q3) So far we have seen 128 crashes across 108 users. So they seem to be mostly unique, with a little bit of overlap.
Q4) I will follow up with my team and get back to you.
@MattSkala, @ygnessin : Thank you so much for sharing this information. That really helps to better understand the situation.
@MattSkala : I looked into our server logs for the requests regarding Firebase identifiers from your application and can see exclusively successful requests. I will involved the FCM/IID team into this, given that this might be related to push notifications arriving (while the application is in the background).
we had just released a new version of the app with these new versions when we started seeing this crash:
com.google.firebase:firebase-messaging
from 20.1.0
-> 20.1.2
com.google.firebase:firebase-analytics
from 17.2.2
-> 17.2.3
we were already using com.google.firebase:firebase-perf
19.0.5
before the crash happened so we have ruled out that it isn't that library.
we have decided to downgrade messaging
and analytics
for the time being while this is being investigated
@maxkohne , @ygnessin , and others: By any chance, are you able to also share a StackTrace with us?
same exact stacktrace as what the author posted
Fatal Exception: java.lang.IllegalStateException at com.google.firebase.iid.FirebaseInstanceId.zzl(FirebaseInstanceId.java:75) at com.google.firebase.iid.FirebaseInstanceId.getId(FirebaseInstanceId.java:49) at com.google.firebase.perf.internal.zzf.zzbu(zzf.java:181) at com.google.firebase.perf.internal.zzf.zzbt(zzf.java:44) at com.google.firebase.perf.internal.zzf.zzc(zzf.java:195) at com.google.firebase.perf.internal.zze.run(zze.java:2) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641) at java.lang.Thread.run(Thread.java:764)
@maxkohne : Thank you so much!
internal: b/151480886
We identified that the IllegalStateException is thrown in FirebaseInstanceId.getResultOrThrowException
(FirebaseInstanceId.java:486) where task.getException()
returns NULL
.
We currently believe that some form of race condition and/or deadlock causes this issue.
We assume that the race condition occurs when multiple Firebase services require a client identifier (a.k.a. FID / Instance ID) at the same time, possibly in different threads or even different processes.
In the situation that multiple Firebase SDKs call FirebaseInstanceId.getId
in parallel, a race condition might occur that leads to a deadlock.
Once the deadline for waiting for the identifier times out, an IllegalStateException
is thrown.
FirebaseInstanceId is an infrastructure service that is used by various Firebase services like Messaging (FCM), Google Analytics, and Performance Monitoring (FirePerf).
With the latest version (v20.1.0
), FirebaseInstanceId introduced a new dependency: The Firebase Installations SDK (FIS).
We assume that the deadlock occurs in FIS and it is FirebaseInstanceId that times out waiting for the identifier, the response of FIS.
To fix this issue, we need to be able to reproduce the issue locally.
At this point, we believe that an incoming push notification wakes up, i.e. triggers processes in an application that is running in background. These processes could include Google Analytics recording the receival of the push notification and Firebase Performance Monitoring (FirePerf) recording some event or latency. Thus, such an application would have to use at least Messaging (FCM) to receive push notifications and two Firebase services like Google Analytics and FirePerf that record events and thus need access to the client identifier.
Generally: Any information to achieve reproducing this crash would be highly appreciated!
Q5) Could you please share your Gradle Android dependency tree with us: We would like to understand which Firebase SDKs (and which versions) you are using in your application.
Q6) Do you happen to have access to a full Android bugreport (including errors and logs from all threads)?
@andirayo Thanks for the update
Q5) Our dependency tree obtained from ./gradlew dependencies
: https://pastebin.com/A8uGPzHh
Q6) I only have access to what I see in Crashlytics, these are stack traces for all threads: https://pastebin.com/XTve0qt3
@andirayo Please find the dependency list and stacktrace of one of the issue Dependency list : https://pastebin.com/gcRZyxNQ Stacktrace from crashlytics for all thread : https://pastebin.com/uJw7LgJm
@andirayo We have more than 380 crashes. 98% of them are in background. According to threads, it is happening when we have notification. App uses network before notification displayed to fetch some information.
Q1) Not sure, we have crash report in firebase but not it looks like crash happens when notification received. FirebaseMessagingService is running at the moment of report.
Q2) 387 crashes for 370 reports. This 50 times more than 2nd most occurring crash. 98% percent of 387 reports happened in background. I am not sure but probably firebase messaging service fails to continue after this report.
Q3) Random users. 387 crashes for 370 reports
Q4) AppId : 1:208472424340:android:a243f98a00873753
Q5) Firebase dependencies :
object Firebase {
const val analytics = "com.google.firebase:firebase-analytics:17.2.3"
const val config = "com.google.firebase:firebase-config:19.1.2"
const val core = "com.google.firebase:firebase-core:17.2.3"
const val messaging = "com.google.firebase:firebase-messaging:20.1.2"
const val perf = "com.google.firebase:firebase-perf:19.0.5"
}
Q6) Stacktrace from firebase for all thread : https://0bin.net/paste/XYQs-BXx0WRDEe+g#F662Og7LOng22wN2W+Ob7UrBPg+5vOamqked4eRCex0
@MattSkala , @sudsingh438 :
Thanks so much for the stack traces! These are really valuable to us as they show what is happening in FirebaseInstallations
during the crash.
@kozaxinan : Thanks a lot for the information. This is very helpful as it confirms our current assumptions. I also checked all client-server requests for the given AppId. Except for 1 request all ended successfully and thus I don't think server communication is the cause for this.
If possible, could you please resent the link to your stack trace as the link above does not seem to work for me.
@andirayo I updated the link.
@kozaxinan : Thank you for the updated link!
@MattSkala , @sudsingh438, @kozaxinan, and @all : The stack traces containing all threads are the most helpful for us right now.
We would appreciate if you find the time to share more stack traces, given that the actual call stacks of the main thread seem to differ between cases, and also the call stacks of FirebaseInstallations
differ.
@roiamiel1 yes, downgrade solved the problem.
@andirayo
Q1) I don't know, because i was not able to reproduce it on my devices. 73% of affected devices developed by HUAWEI, 64% is Android 9, 78% background.
Q2) Crashlytics told me that "This issue has 1569 crashes affecting 1348 users".
Q3) Some users has repeated crashes. For example one user has 4 crashes one after another in one minute.
Q4) Project ID "mirror-4543d", Application ID "com.mirrorai.app"
Q5) dependencies.txt
Q6) com.mirrorai.app_1769554677256386823(2).txt
@asfdfdfd : Thank you so much, especially for sending 3 stack traces! That is very helpful. Also thanks for the crash details! I looked into our server logs regarding your Project ID. Almost all requests are successful. There are a few auth errors that I will look into, but they are unrelated to this issue.
The best solution for this problem currently still is to downgrade to the old version of the SDKs, particularly a version of FirebaseInstanceId
(a.k.a. firebase-iid
) before v20.1.0.
We were not able to make much progress on the weekend, but I am positive that we will make progress tomorrow when we can coordinate with the teams of Firebase Messaging (FCM) and Firebase Performance Monitoring (FirePerf).
FirebaseInstanceId
times out after waiting for an FID from FirebaseInstallations for over 30 seconds.#getId
is called while another thread or process is still communicating with the FirebaseInstallations server?#onMessageReceive
.#getId
?FirebaseInstanceId
?@andirayo We have observed few IllegalStateException crashes are happening after null pointer exception is being thrown by the perf. This is from one of the logs
2020-03-15T21:12:42:096 GMT+05:30 - E - java.lang.NullPointerException: Attempt to invoke virtual method 'void com.google.firebase.perf.internal.v.a(boolean)' on a null object reference
at com.google.firebase.perf.internal.f.b(com.google.firebase:firebase-perf@@19.0.5:74)
at com.google.firebase.perf.internal.i.run(com.google.firebase:firebase-perf@@19.0.5:1)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1162)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:636)
at java.lang.Thread.run(Thread.java:764)
2020-03-15T21:12:43:450 GMT+05:30 - E - java.lang.IllegalStateException
at com.google.firebase.iid.FirebaseInstanceId.o(com.google.firebase:firebase-iid@@20.1.1:12)
at com.google.firebase.iid.FirebaseInstanceId.a(com.google.firebase:firebase-iid@@20.1.1:15)
at com.google.firebase.perf.internal.f.c(com.google.firebase:firebase-perf@@19.0.5:5)
at com.google.firebase.perf.internal.f.b(com.google.firebase:firebase-perf@@19.0.5:12)
at com.google.firebase.perf.internal.f.a(com.google.firebase:firebase-perf@@19.0.5:52)
at com.google.firebase.perf.internal.e.run(com.google.firebase:firebase-perf@@19.0.5:1)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1162)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:636)
at java.lang.Thread.run(Thread.java:764)
Good morning, We are also having the same/similar problems with our app.
We got 23 crash error from 23 different users in 7 days . and also 96% are in background (https://i.imgur.com/5naCAwz.png)
Pasted error trace: https://pastebin.com/x7zyapyt
Looks like best bet for now is to lock the firebase-iid library to 20.0.2?
@sudsingh438 : Thank you for this information. I will try to look into this!
@Shudy : Thanks so much for sharing this! That is very helpful!
@kenyee : I think there was a version v20.0.3
on Github, but yes v20.0.2
solves this issue.
We identified a potential issue when a network connection times out or fails and multiple threads are accessing / waiting for the Firebase client identifier. We are trying to reproduce this issue in order to work out a fix for this.
v20.1.3
, on Thursday, March 26, 2020.on our side crash was happening while trying to access the token, now seems fixed on
com.google.firebase:firebase-messaging:20.1.3
.
ok, update: it is still crashing on Android 10 emulator with version 20.1.3
, problem seems to be still there.
@guness :
Thank you so much for the update.
Firebase Messaging v20.1.3
does not contain the fix, though.
The fix is included in Firebase Installations v16.2.0
which is already available on Github, but will only be released to Maven on Thursday, March 26.
Hi everyone, We are having problems that looks similar this problem with our app. I am not sure about the similarity, but I want to share the problem with you. You can see the details of our problem in here.
Do you have any suggestion to solve the issue that caused by java.io.IOException?
@andirayo We pushed an update to our app yesterday only. In this update we have used firebase-messaging:20.1.3. I am assuming that we don't have to push another update. It will be fixed automatically?
@adiamartya :
Unfortunately, Firebase Messaging v20.1.3
does not contain the fix, though.
The fix is included in Firebase Installations v16.2.0
which is already available on Github, but will only be released to Maven on Thursday, March 26.
fyi: Firebase Messaging v20.1.4
will depend on Firebase Installations v16.2.0
.
To fix this issue you will have to push an update that contains Firebase Installations v16.2.0
or higher.
@andirayo Thanks a lot for the information. So, currently we are using firebase-core v17.2.3 and we aren't using firebase installations. Shall we remove firebase-core and directly use analytics sdk and firebase installations? Also, I am assuming that we have to include firebase installations v16.2.0 anyhow.
@adiamartya : Firebase Installations is an infrastructure service (identity provider and authentication scheme) that is used under the hood by many Firebase services (like Analytics). Usually, you do not need to explicitly include it, but you will get it as transitive dependency (through Firebase InstanceId).
To answer your question: Yes, we recommend to directly depend on the Firebase services that you are using and thus we recommend to directly depend on Firebase Analytics, rather than Core.
If you are affected by this issue, unfortunately you only have 2 options:
A) Downgrade to SDKs that use a version of Firebase InstanceId before v20.1.0
. The easiest way to do this is to depend on Firebase Messaging v20.1.0
(or before) and lock the version of Firebase InstanceId to v20.0.2
in your Gradle file.
B) Or wait until March 26 and upgrade your SDKs to depend on Firebase InstanceId v20.1.3
of higher (depending on Firebase Installations v16.2.0
or higher).
I know this all sounds very complicated, so let me try to explain the dependency tree a bit:
v20.1.3
depends on Firebase Instance ID v20.1.2
v20.1.2
depends on Firebase Installations v16.1.0
A similar situation is true for Google Analytics:
v17.2.3
depends on Firebase Analytics v17.2.3
v17.2.3
depends on Play Services Measurement API v17.2.3
v17.2.3
depends on Firebase Instance ID v20.0.2
(or higher)But if you use Firebase Messaging (FCM) in the same application the Firebase InstanceId dependency will be resolved to v20.1.2
, which is causing the issue.
Thanks a lot. Will wait for 26th and will push update with Firebase Messaging v20.1.4 :)
@andirayo Hi. Update is still not available. I am just confirming World is facing hard times. So, delay is understandable. Just confirming :)
Can we can get an update on v20.1.4?
@schof, @adiamartya: Unfortunately our M67 release was delayed due to some GMSCore issues. The release will unfortunately go out after the weekend, on March 30, 2020. Sorry!
Finalized new version numbers:
v20.1.4
(version changed last minute)v20.1.4
v16.2.0
Good news, the M67 release is completed and available on Maven. https://maven.google.com/web/index.html (fyi: Javadocs and release notes will follow soon.)
Looking forward to hear if the new release fixes the issue for you!
@andirayo Thanks man. Meeting deadlines during pandemic is really awesome.
@andirayo quick correction to your above post. Firebase Messaging is v20.1.4
and not v20.2.2
.
@schof : You are right. I checked with the Messaging team. Some things must have moved around in the past weeks. Our release doc wasn't updated. Thanks for pointing this out! I corrected my post above.
@andirayo thanks for the update.
I'm upgrading firebase-messaging from 20.1.3 to 20.1.4 but now I get this error:
Unexpected failure during lint analysis (this is a bug in lint or one of the libraries it depends on)
Message: com.android.tools.lint.detector.api.Project.getGradleProjectModel()Lcom/android/builder/model/AndroidProject;
The crash seems to involve the detector com.google.firebase.installations.lint.IncompatibleIidVersionDetector
.
You can try disabling it with something like this:
android {
lintOptions {
disable "IncompatibleIidVersion"
}
}
Stack: NoSuchMethodError:IncompatibleIidVersionDetector.beforeCheckEachProject(IncompatibleIidVersionDetector.kt:56)←LintDriver.checkProject(LintDriver.kt:894)←LintDriver.analyze(LintDriver.kt:422)←LintCliClient.run(LintCliClient.java:238)←LintGradleClient.run(LintGradleClient.java:261)←LintGradleExecution.runLint(LintGradleExecution.java:305)←LintGradleExecution.lintSingleVariant(LintGradleExecution.java:392)←LintGradleExecution.analyze(LintGradleExecution.java:94)
Any idea how can I fix it?
[REQUIRED] Step 2: Describe your environment
[REQUIRED] Step 3: Describe the problem
Crashes are being reported on play console with below stack trace:
java.lang.IllegalStateException: at com.google.firebase.iid.FirebaseInstanceId.zzl (FirebaseInstanceId.java:75) at com.google.firebase.iid.FirebaseInstanceId.getId (FirebaseInstanceId.java:49) at com.google.firebase.perf.internal.zzf.zzbu (zzf.java:181) at com.google.firebase.perf.internal.zzf.zzbt (zzf.java:44) at com.google.firebase.perf.internal.zzf.zzc (zzf.java:195) at com.google.firebase.perf.internal.zze.run (zze.java:2) at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1167) at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:641) at java.lang.Thread.run (Thread.java:764)
Steps to reproduce:
We are not able to reproduce it locally.