FIRESTORE (8.2.1) INTERNAL ASSERTION FAILED: Unexpected state

[Benny739]

We're seeing this error a lot in sentry atm. I can not reproduce this or say where it comes from or what it consequences are. I guess it is related to offline support.

[google-oss-bot]

I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.

[wu-hui]

Hi there,

Our SDK strips error messages (for sizes), and the error message you provide is the stripped one. This is not very helpful for debugging unfortunately.

Given that you cannot reproduce this yourself, the only thing we can try to to roll some of your users to a SDK without the stripping, and see if we can get some better error messages.

If I provide you instructions to get a SDK without stripping, are you willing to roll this out to a portion of your users?

Thanks

[Benny739]

Hi,

thanks for your help. Yes we're willing to try it out. What are the risks when using unstripped error messages?

[bigjimhere]

This error started appearing in our Sentry logs some time around version 7.20.0, with offline persistence disabled. The error occurred sporadically when firebase.firestore().disableNetwork() was called seconds after app startup. Removing firebase.firestore().disableNetwork() from our code resolved the issue.

My guess is that our was app was occasionally calling disableNetwork() while Firestore was still initialising. This may have been interrupting Firebase/Firestore authentication in some way, perhaps by cutting off the connection before authentication could complete. Read and write operations consequently seemed to fail, perhaps due to an "unexpected auth state" (see below).

If you are not using disableNetwork() in your app perhaps Firestore initialisation/authentication is being interrupted in some other way? https://stackoverflow.com/questions/63319638/firestore-internal-assertion-failed-unexpected-state-when-unit-testing-with-j

Our app initialises multiple Firebase instances on startup. Each instance immediately starts reading a fairly large amount of data using Firestore onSnapshot listeners. We were seeing the following errors in Sentry, all of which were resolved when we stopped using disableNetwork():

"FIRESTORE (8.2.6) INTERNAL ASSERTION FAILED: Unexpected state"

"FirebaseError: missing stream token" (less frequently)

"TypeError: undefined is not an object (evaluating 'e.batchId')" (very rarely)

"FirebaseError: the first request in a stream cannot contain writes" (very rarely)

The "Unexpected state" error seemed to cause both reads and writes to fail ("hang") without a promise rejection. We were able to reproduce this at the time, but I'd need to do more testing to confirm. Prior to 7.20.0 we had been able to reproduce hanging writes, but in earlier versions the writes failed silently without the "Unexpected state" error. Offline persistence was enabled for those earlier tests. If the "Unexpected state" error was the cause in those earlier cases it is possible that the writes failed silently because the error messages above weren't added to the SDK until some time around 7.20.0. For example I noticed the following in the release notes for 7.23.0- "The SDK now includes more information in the error message for failed IndexedDB transactions."

The "missing stream token" error caused reads to fail but I can't confirm whether or not writes were affected. As a side note we would see onSnapshot listeners failing with the following error message around the time that this error was logged in Sentry: "FirebaseError: [code=already-exists]: Target ID already exists: 6". This error also disappeared when we stopped using disableNetwork().

I'm not sure what effect the last two errors above had on the application.

I've attached a stack trace for "FIRESTORE (8.2.1) INTERNAL ASSERTION FAILED: Unexpected state"

Firestore Unexpected State Error.txt

UPDATE:

It looks like the "Unexpected state" error was thrown when our app called disableNetwork() on multiple Firestore instances, waited a few seconds and then called enableNetwork() on those same instances. This was likely happening while Firestore was initialising.

[wu-hui]

@bigjimhere Are you describing the same project as @Benny739 here, or seeing similar issues with your own project?

Either way, thanks for the information. I will try to call disableNetwork with multiple firebase instance myself.

@Benny739 I will provide a SDK with all error messages enabled, hopefully soon.

[bigjimhere]

@wu-hui I'm referring to issues with my own project.

[mlahp7]

We too got a ton of errors in our Sentry that begin with FIRESTORE (8.2.4) INTERNAL ASSERTION FAILED: Unexpected state. It seems to have started happening for us yesterday. @bigjimhere @Benny739 is the error coming from just one user or multiple?

[mcarriere]

We're experiencing this issue with FIRESTORE (8.2.4) INTERNAL ASSERTION FAILED but we're not calling disableNetwork anywhere in our code. It happened over 43k times from a single browser today. Luckily it was an internal user so we could tell them to refresh the page but it's not really something we want to tell our clients.

[bigjimhere]

@mlahp7 In our case multiple users were affected. The issue occurred on a wide range of devices and browsers (desktop and mobile). Looking at the Sentry reports I suspect that a weak internet connection may have been involved. The user that experienced this issue by far the most was reportedly accessing our app through a small business VPN. Others were using our hybrid mobile app in areas where connection is known to be unreliable. We tried experimentalForceLongPolling and autoDetectLongPolling but these settings did not solve the issue. For us only removing our implementation of disableNetwork and enableNetwork solved the issue.

[bigjimhere]

The issue could be caused by losing or disabling the connection while firebase.auth().currentUser.getIdTokenResult or firebase.auth().currentUser.getIdToken are in progress. I haven't done any testing so this is only speculative. I noticed that our code was calling both of these functions around the same time as disableNetwork.

[wu-hui]

To everyone who are reporting issues, please include full stacktrace, this will help me find exactly where the error happens.

@Benny739 in particular:

I realized you don't need to try a custom build of the SDK, if you can provide below information full stacktrace of the error report, which version you use and which build (CDN or NPM).

If you are using NPM build and also does obfuscation on top of Firebase, you probably need to first remove your obfuscation using something like https://github.com/mifi/stacktracify

The stacktrace we can parse should have traces like node_modules/@firebase/firestore/dist/index.esm.js:5782:31 for example.

To others:

Please note the error message here is very generic, and we cannot help if we cannot point them to exactly where the error happens, your issue may or may not be related to what the original issue is about. Please try to provide full stack trace as well.

[bigjimhere]

I've now seen this error in a different context to the one I mentioned above. The error occurred on 8.2.7 on iOS 14.4 with offline persistence enabled. The error was preceded by another error: "INTERNAL UNHANDLED ERROR: e is not a function. (In 'e()', 'e' is undefined)"

Stack trace for both errors:

Unexpected State Error Stack Trace.txt

[bigjimhere]

We just saw the Unexpected state error multiple times in a single session on OSX 10.15.4 running Safari 13.1.

We're monitoring online presence using '.info/connected' (Firebase RTDB). Both "Unexpected state" and "e is not a function. (In 'e()', 'e' is undefined)" occurred multiple times in this session when RTDB went offline:

---

In my first post here I mentioned that the Unexpected state error was occurring when we called disableNetwork and enableNetwork with offline persistence disabled- it's worth noting that we were calling those methods whenever RTDB went offline/online.

The "Unexpected state" error was also thrown multiple times in this session when trying to set up an onSnapshot listener with offline persistence enabled.

The error "Connection to Indexed Database server lost. Refresh the page to try again" also occurred early on in this session. Could be related to https://bugs.webkit.org/show_bug.cgi?id=197050

[wu-hui]

Thanks for the stack trace.

Your first error 'e' is undefined seems to be a IDB transaction failure error. The second one is a webchannel failure, which might be related to the network conditions.

The bug you posted could certainly be the cause of the first error though. Do you only observe it from safari?

Pasting the converted stack trace for further investigation:

[2021-02-18T07:08:38.970Z]  @firebase/firestore: Firestore (8.2.7): INTERNAL UNHANDLED ERROR:  e is not a function. (In 'e()', 'e' is undefined)
Xr@../firestore/dist/prebuilt.js:5313:56
Xr@../../node_modules/tslib/tslib.es6.js:100:27
Xr@../../node_modules/tslib/tslib.es6.js:72:65

Xr@https://www.gstatic.com/firebasejs/8.2.7/firebase.js in Qv@ionic: at line 1:473665
../firestore/dist/prebuilt.js:15793:19
../webchannel-wrapper/dist/temp/node_modules/google-closure-library/closure/goog/events/eventtarget.js:351:27
../webchannel-wrapper/dist/temp/node_modules/google-closure-library/closure/goog/events/eventtarget.js:481:11
../webchannel-wrapper/dist/temp/node_modules/google-closure-library/closure/goog/labs/net/webchannel/webchannelbasetransport.js:368:2
../webchannel-wrapper/dist/temp/node_modules/google-closure-library/closure/goog/labs/net/webchannel/webchannelbase.js:2219:49
../webchannel-wrapper/dist/temp/node_modules/google-closure-library/closure/goog/labs/net/webchannel/channelrequest.js:822:6
../webchannel-wrapper/dist/temp/node_modules/google-closure-library/closure/goog/labs/net/webchannel/channelrequest.js:580:6
../webchannel-wrapper/dist/temp/node_modules/google-closure-library/closure/goog/events/eventtarget.js:351:27
../webchannel-wrapper/dist/temp/node_modules/google-closure-library/closure/goog/events/eventtarget.js:481:11
../webchannel-wrapper/dist/temp/node_modules/google-closure-library/closure/goog/net/xhrio.js:884:4
../webchannel-wrapper/dist/temp/node_modules/google-closure-library/closure/goog/net/xhrio.js:841:2

[bigjimhere]

@wu-hui it's easiest to summarise:

Case 1: start app with offline persistence disabled > call disableNetwork across multiple instances > call enableNetwork across multiple instances > Unexpected state error. Seen on many devices and browsers (iOS, Android, Edge on Windows 10, Chrome on Mac OSX etc). We can reproduce the error in this case.

Case 2: offline persistence enabled. Unexpected state error seen in conjunction with 'e' is undefined and Connection to Indexed Database server lost. Only seen on iOS 14.4 and Safari 13.1 on Mac OSX. We can't reproduce the error in this case.

It seems like poor network conditions or simulated poor network conditions (disableNetwork) are common between the two cases. That said, in case 2 background/foreground activity may have also played a role. This is evidence for the WebKit bug being involved (https://bugs.webkit.org/show_bug.cgi?id=197050).

Both cases caused commit operations to fail (hang) without a promise rejection.

[schmidt-sebastian]

FWIW, the corresponding source code is:

 async runTransaction(t, e, n, s) {
        const i = "readonly" === e;
        let r = 0;
        for (;;) {
            ++r;
            try {
                this.db = await this.Hn(t);
                const e = Es.open(this.db, t, i ? "readonly" : "readwrite", n), r = s(e).catch((t => (
                e.abort(t), ws.reject(t)))).On();
                return r.catch((() => {})), 
                await e.qn, r;
            } catch (t) {
                const e = "FirebaseError" !== t.name && r < 3;
                if (R("SimpleDb", "Transaction failed with error:", t.message, "Retrying:", e), 
                this.close(), !e) return Promise.reject(t);  // This is Xr@../firestore/dist/prebuilt.js:5313:56
            }
        }
  }

This translates to https://github.com/firebase/firebase-js-sdk/blob/4090271bb71023c3e6587d8bd8315ebf99b3ccd7/packages/firestore/src/local/simple_db.ts#L433. e is the boolean retryable.

[bigjimhere]

More evidence for the Webkit bug being a cause of this error, this time on iOS 12:

UnknownError: Connection to Indexed Database server lost. occurred shortly after.

[bigjimhere]

We just saw 'e' is not a function immediately followed by Unexpected state on Chrome 89.0.4389 on Mac OSX 10.15.7, with offline persistence enabled. This is the first time we've seen an 'e'-related bug outside of iOS/Safari. The user was resuming a long-running session so it's probably safe to assume that the tab was in the background immediately before the error was thrown.

This error was also being thrown regularly around the same time as the above errors:

Firestore (8.2.7): Connection WebChannel transport errored: [object Object]

[marcoancona]

We also saw this error in our Sentry report. This so far happened for one user only, triggering ~1.8K of such errors in a few seconds.

[zeevl]

We're seeing this as well, version 8.4.1, electron app with persistence enabled:

[92m(errors) Error: FIRESTORE (8.4.1) INTERNAL ASSERTION FAILED: Unexpected state
    at Mr (file:///Applications/ScribeWare%205.app/Contents/Resources/app.asar/dist/vendor/firebase/firebase-firestore.js:1:46421)
    at sp.Tc (file:///Applications/ScribeWare%205.app/Contents/Resources/app.asar/dist/vendor/firebase/firebase-firestore.js:1:268833)
    at sp.enqueue (file:///Applications/ScribeWare%205.app/Contents/Resources/app.asar/dist/vendor/firebase/firebase-firestore.js:1:267636)
    at sp.enqueueAndForget (file:///Applications/ScribeWare%205.app/Contents/Resources/app.asar/dist/vendor/firebase/firebase-firestore.js:1:267321)
    at file:///Applications/ScribeWare%205.app/Contents/Resources/app.asar/dist/vendor/firebase/firebase-firestore.js:1:300518
    at ag (file:///Applications/ScribeWare%205.app/Contents/Resources/app.asar/dist/vendor/firebase/firebase-firestore.js:1:300724)
    at og (file:///Applications/ScribeWare%205.app/Contents/Resources/app.asar/dist/vendor/firebase/firebase-firestore.js:1:299086)
    at Rg.update (file:///Applications/ScribeWare%205.app/Contents/Resources/app.asar/dist/vendor/firebase/firebase-firestore.js:1:312801)
    at R (file:///Applications/ScribeWare%205.app/Contents/Resources/app.asar/dist/static/js/main.bf484a7e.chunk.js:2:858748)

[zeevl]

@wu-hui We are seeing this with an increasing number of users now, and it's causing data loss and requires an application restart.

As such, it's becoming a very severe and urgent bug for us. I'm happy to help diagnose however I can, please let me know?

[bigjimhere]

Same here. We activated offline persistence around a month ago on a cross-platform app with around 200 daily users. You can see the Sentry logs here, the issue is happening fairly frequently:

An application restart is not always a great solution. Sometimes the error occurs right as the user is saving data, and that data is lost when the app resets without the changes having been saved into offline persistence.

I repeat @zeevl 's sentiment- let us know if we can help to diagnose further.

[wu-hui]

Hi @zeevl

The stack trace you posted here is basically saying the SDK's internal work queue is failed already when trying to schedule work. Are there other errors in your report? The error message will always be "Unexpected state" but they will have different stack trace, so it is not easy to find all of them, but they can be helpful to identify what went wrong in the first place.

@bigjimhere

Thanks for your analysis! Yeah, there could be multiple issues going on in this thread, our stripped error message does not help here.

For your case 1, you said you could reproduce this. Can you provide a mini reproduction for me to try out?

You are right that the e is not a function error might be related to underlying IndexedDB implementation, and it is known to be problematic for some browsers.

---

Below is the actual stack trace from zeevl's report, pasting here for future reference:

[92m(errors) Error: FIRESTORE (8.4.1) INTERNAL ASSERTION FAILED: Unexpected state
    at  (../src/util/assert.ts:40:8)
    at fail (../src/util/async_queue_impl.ts:220:6)
    at __PRIVATE_verifyNotFailed (../src/util/async_queue_impl.ts:114:9)
    at enqueue (../src/util/async_queue_impl.ts:89:9)
    at enqueueAndForget (../src/core/firestore_client.ts:438:20)
    at ensureFirestoreConfigured (../src/exp/reference_impl.ts:771:17)
    at executeWrite (../src/exp/reference_impl.ts:366:9)
    at updateDoc (../src/api/database.ts:730:15)

[zeevl]

@wu-hui here's the error immediately preceding the Unexpected state:

TypeError: e is not a function
    at au.<anonymous> (file:///C:/resources/app.asar/dist/vendor/firebase/firebase-firestore.js:1:107604)
    at file:///C:/resources/app.asar/dist/vendor/firebase/firebase-firestore.js:1:2037
    at Object.throw (file:///C:/resources/app.asar/dist/vendor/firebase/firebase-firestore.js:1:2142)
    at i (file:///C:/resources/app.asar/dist/vendor/firebase/firebase-firestore.js:1:940)

might be the same as @bigjimhere?

FWIW, our application currently runs on electron 10.1.6, which uses chrome 85.0.4183.121

We also tried upgrading to firebase 8.6.5 and are still seeing the same errors.

[wu-hui]

Yeah, it is the same error as @bigjimhere sees.

It is from error handling of a failed IDB transaction, I am not sure why it is a TypeError yet. Likely IDB is not stable, and this error actually stopped the SDK from retrying that transaction.

[bigjimhere]

@wu-hui I've put together a minimal example of my larger reproduction but unfortunately it doesn't reproduce the error described in my case 1 above. I'll keep experimenting to see if I can get the mini repro to reproduce the error. I can't be certain but I suspect that my case 1 is a separate issue to the e is not a function error.

I'm not sure if this is related but for what it's worth:

Resurfacing commit errors

"In the future there may be additional support for resurfacing commit errors that may occur when a page has been shutdown in the middle of a transaction being committed and is thus not around to receive the errors from the database task queue. Such an implementation would involve saving any errors that occur during committing so that if the page is loaded again at another time it can be delivered the errors that occurred previously and handle them."

https://andreas-butler.github.io/idb-transaction-commit/EXPLAINER.html

[wu-hui]

Thank you! It is a pretty useful link.

Yeah, it does look like the errors here are all around IDB transactions (and its error handling from our SDK).

[bigjimhere]

The error message now has a bit more information- "AsyncQueue Failed to persist write"

@firebase/firestore: Firestore (8.6.8): AsyncQueue Failed to persist write: TypeError: e is not a function

[wu-hui]

Thanks for more information. Unfortunately these all seem to confirm that it is the underlying indexeddb transaction failed, I am not sure what the path forward would be at this piont.

@myatmin8 Yeah, the problem goes away because you removed enablePersistence, which will use memory instead of indexeddb for storage.

[bigjimhere]

The e is not a function error shows this information (v8.6.8). It may explain the TypeError:

../firestore/dist/prebuilt.js in call at line 5476:57

5472. // Note: We cannot use an instanceof check for FirestoreException, since the
5473. // exception is wrapped in a generic error by our async/await handling.
5474. const e = "FirebaseError" !== t.name && r < 3;
5475. if (b("SimpleDb", "Transaction failed with error:", t.message, "Retrying:", e),
5476. this.close(), !e) return Promise.reject(t);

[rrousselGit]

I faced a similar error when trying to use jest+ firestore emulator.

Sounds like my case is a different than the one described here, but in any case:

The problem was solved by doing jest --env=node as per https://stackoverflow.com/a/63465382/8394265

[AshleyMedway]

Hi there,

Our SDK strips error messages (for sizes), and the error message you provide is the stripped one. This is not very helpful for debugging unfortunately.

Given that you cannot reproduce this yourself, the only thing we can try to to roll some of your users to a SDK without the stripping, and see if we can get some better error messages.

If I provide you instructions to get a SDK without stripping, are you willing to roll this out to a portion of your users?

Thanks

Hi, can you provide the instructions for getting the SDK with the messages in?

[schmidt-sebastian]

If you are building from Node, are you able to swap out node_modules/@firebase/firestore/dist/index.esm2017.js?

This is the file with all debug information: https://gist.github.com/schmidt-sebastian/434bb80040d079fd75da355a3430dead

[AshleyMedway]

@schmidt-sebastian is it possible to build the library myself with the messages in? we actually use a range of firestore versions across different applications?

[AshleyMedway]

Any update here? I've got the yarn build working but its not clear how to produce the build with the debugging messages included.

[schmidt-sebastian]

@AshleyMedway Sorry, we are just coming back from a long weekend.

The build I provided is from running yarn build:debug in packages/firestore. That, however, only creates a single build artifact.

If you want all build artifacts you need to replace rollup.config.js with https://gist.github.com/schmidt-sebastian/52004ec096d08a5270565af974312d41 Then just invoke yarn build.

[AshleyMedway]

@schmidt-sebastian perfect thank you, hope you had a great thanks giving

[augustmuir]

I experienced the same error message on 9.6.6 with NextJS. I believe this error message could be presented due to a range of errors - as I see 100+ Stackoverflow questions with this error message.

After lots of debugging I realized I accidently used SQL capitalization:

.orderBy('time', 'ASC') = "INTERNAL ASSERTION FAILED: Unexpected state" .orderBy('time', 'asc') = No Errors!

This was a pain to debug, and my mistake was so small. Maybe better error reporting is needed in cases like this? When you get then Google this error message it easily leads you down a path of debugging things completely irrelevant to the real error.

[schmidt-sebastian]

We have gotten a lot of feedback about our SDK size, which is one of the reason we removed API input validation from our surface. We only want to show API usage errors that TypeScript can't tell you about. @augustkimo if you use TypeScript it would help to know whether you saw this error or not.

We could also add an additional build target to our release that does not minify error messages away (if there is interest)/

[AshleyMedway]

We could also add an additional build target to our release that does not minify error messages away (if there is interest)/

Yes please! We are using an ionic application and a larger binary is acceptable for additional information.

[augustmuir]

We have gotten a lot of feedback about our SDK size, which is one of the reason we removed API input validation from our surface. We only want to show API usage errors that TypeScript can't tell you about. @augustkimo if you use TypeScript it would help to know whether you saw this error or not.

We could also add an additional build target to our release that does not minify error messages away (if there is interest)/

Good to know that Typescript would've caught this, I was using Javascript when I got the error.

It would have saved me from some confusion if the error was a generic Firebase API Error: Error messages stripped to reduce package size instead of the nondescript message. Not a huge deal to me personally, after tons of Firebase Javascript usage I only ran into this error once.

[Animus-Surge]

I'm having similar problems to the ones in this thread in version 9.6.7. I get several INTERNAL ASSERTION FAILED: Unexpected state errors bound to this line in a script: const news = await getDocs(q); where q is a query: const q = query(newsData, orderBy("date", "dsc"), limit(2)); I have not encountered this error before, and I don't know what I was doing wrong.

[augustmuir]

@Animus-Surge you made a small syntax mistake on the query. It should be orderBy("date", "desc") not orderBy("date", "dsc"). See my original comment on this issue.

[Animus-Surge]

@Animus-Surge you made a small syntax mistake on the query. It should be orderBy("date", "desc") not orderBy("date", "dsc"). See my original comment on this issue.

Oops, I guess I have. Sorry about that

[AshleyMedway]

We could also add an additional build target to our release that does not minify error messages away (if there is interest)/

@schmidt-sebastian Any update on this? We have upgraded through firebase V7 - V9 and still getting seemingly random INTERNAL ASSERTION FAILED: Unexpected state but only from iOS devices. I believe the issue is related to indexedDB.

Currently trying to produce 9.6.8 debug build to get more information

[schmidt-sebastian]

I talked to our JS Build team earlier this week. We will include a debug build with in either the next or the next next release.

[AshleyMedway]

With the debug build we see this error:

@firebase/firestore: Firestore (9.6.8): FIRESTORE (9.6.8) INTERNAL ASSERTION FAILED: AsyncQueue is already failed: Attempt to iterate a cursor that doesn't exist

Then looking back through the log this appears to be the original error:

@firebase/firestore: Firestore (9.6.8): AsyncQueue Initialization of query 'Query(target=Target(Databases/3f2ff7b5-70c1-4cff-81d5-222b0174ef98/Employees, orderBy: [__name__ (asc)]); limitType=F)' failed: UnknownError: Attempt to iterate a cursor that doesn't exist

Also

@firebase/firestore: Firestore (9.6.8): INTERNAL UNHANDLED ERROR: Attempt to iterate a cursor that doesn't exist

It might also be worth noting this issue only occurs on iOS devices.

Additionally, when we detect any error with a firestore query we terminate, initialise and rerun the query, but that does not seem to help

[AshleyMedway]

Any update on the above errors? This is still an on going issues for our iOS users. Seems like the connection to the IndexedDB will randomly fail and the only way to resolve is to quit and reopen the application.

[cherylEnkidu]

Hi @AshleyMedway,

1. If you are running in web browser, Safari IndexedDb implementation has been having issues for years. At this point we don’t have a solution. If your app do not need offline or multi-tab support, you can try is to use memory persistence instead.

2. If you build the iOS App on React Native, you can try turn on long polling :

   const db = initializeFirestore(app, {
   experimentalAutoDetectLongPolling: true
   })

[joker7391]

I have the same error