FIRESTORE INTERNAL ASSERTION FAILED: Failed to open DB

[lammertw]

[REQUIRED] Step 1: Describe your environment

• Xcode version: 11.4.1
• Firebase SDK version: 6.21.0
• Firebase Component: Firestore
• Component version: 6.21.0
• Installation method: CocoaPods

[REQUIRED] Step 2: Describe the problem

My app is crashing with a fatal exception due to Firestore.

I know several of these crashes has been reported before but since they are locked I could not comment on them.

So far this issue is only happening in production but it's happening in about 1% of all cases (about 2k crashes so far) so it's a pretty big problem for us now.

Fatal Exception: NSInternalInconsistencyException
FIRESTORE INTERNAL ASSERTION FAILED: Failed to open DB: Internal: Failed to open LevelDB database at /var/mobile/Containers/Data/Application/2211CDC3-4C78-4450-86AE-A1AC24D902BD/Library/Application Support/firestore/__FIRAPP_DEFAULT/eneco-360/main: LevelDB error: IO error: /var/mobile/Containers/Data/Application/2211CDC3-4C78-4450-86AE-A1AC24D902BD/Library/Application Support/firestore/__FIRAPP_DEFAULT/eneco-360/main/LOCK: Operation not permitted (expected created.ok())

Fatal Exception: NSInternalInconsistencyException
0  CoreFoundation                 0x1acd22a48 __exceptionPreprocess
1  libobjc.A.dylib                0x1aca49fa4 objc_exception_throw
2  CoreFoundation                 0x1acc24e88 +[_CFXNotificationTokenRegistration keyCallbacks]
3  Foundation                     0x1ad0a9044 -[NSAssertionHandler handleFailureInFunction:file:lineNumber:description:]
4  Eneco                          0x100581828 firebase::firestore::util::ObjcThrowHandler(firebase::firestore::util::ExceptionType, char const*, char const*, int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) + 59 (exception_apple.mm:59)
5  Eneco                          0x1005814dc firebase::firestore::util::Throw(firebase::firestore::util::ExceptionType, char const*, char const*, int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) + 90 (exception.cc:90)
6  Eneco                          0x1005cc7d0 firebase::firestore::util::internal::FailAssertion(char const*, char const*, int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, char const*) + 42 (hard_assert.cc:42)
7  Eneco                          0x1005cc894 firebase::firestore::util::internal::FailAssertion(char const*, char const*, int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, char const*) + 49 (hard_assert.cc:49)
8  Eneco                          0x1005a1a30 firebase::firestore::core::FirestoreClient::Initialize(firebase::firestore::auth::User const&, firebase::firestore::api::Settings const&) + 177 (firestore_client.cc:177)
9  Eneco                          0x100562724 firebase::firestore::util::AsyncQueue::ExecuteBlocking(std::__1::function<void ()> const&) + 957 (atomic:957)
10 Eneco                          0x100582854 firebase::firestore::util::internal::DispatchAsync(NSObject<OS_dispatch_queue>*, std::__1::function<void ()>&&)::$_0::__invoke(void*)
11 libdispatch.dylib              0x1ac9ef184 _dispatch_client_callout
12 libdispatch.dylib              0x1ac9cc710 _dispatch_lane_serial_drain$VARIANT$armv81
13 libdispatch.dylib              0x1ac9cd128 _dispatch_lane_invoke$VARIANT$armv81
14 libdispatch.dylib              0x1ac9d643c _dispatch_workloop_worker_thread
15 libsystem_pthread.dylib        0x1aca3eb88 _pthread_wqthread
16 libsystem_pthread.dylib        0x1aca41760 start_wqthread

[morganchen12]

Are you setting any file protection policies on the database directory?

[wilhuff]

Hi and sorry for your troubles.

What this stack says is when Firestore is trying to open its underlying database it's failing with an "Operation not permitted" error (EPERM).

We've seen this happen before when apps are in the background and the filesystem is locked, most notably in #2846, wherein Firestore's own idle activity could cause the failure while an application was in the background (this has since been fixed).

You may be running into a similar issue, wherein your application is taking some action that's trying to use Firestore while in the background. There are a few ways this can happen:

• Actually implementing a background task
• Responding push notification

If you've changed the com.apple.developer.default-data-protection entitlement you'll definitely see this error. Even if you haven't, it's possible to observe it if you try to run before the user has unlocked their device.

[lammertw]

Thanks for the answer.

We're not setting or changing the com.apple.developer.default-data-protection entitlement.

We do also use FCM and we also have background fetch enabled so background tasks could be running. However, according to the Crashlytics dashboard only 12% of the crashes occur when the device is in background.

Is there any other info you'd like me to send or anything you'd like me to try on our side?

[wilhuff]

In my experience the "background" state reflected in Crashlytics isn't terribly accurate. If you can observe some crashes in the background then it's worth starting down the path of trying to figure that out. If, after fixing that there are still non-background crashes going on that would be a follow-up.

Firestore is incompatible with locked filesystems. In the future we'll make this better by failing cleanly, but that will just push the failure out as an error out to your code--the failure will still be there.

So you have essentially two paths forward:

• Identify code paths in your system that might try to access Firestore in response to a push notification or background task and avoid making those calls, or
• Reduce the file protection level for Firestore's backing data (as outlined in https://github.com/firebase/firebase-ios-sdk/issues/2846#issuecomment-484291699) (though note that since #4609, Firestore now stores its data in .applicationSupportDirectory, not .documentsDirectory). This is only valid if your local data is not sensitive.

[lammertw]

Since the data in Firestore is public data I'll use the approach of reducing the file protection level for now. I'll report back later to let you know if that helped or not.

[google-oss-bot]

Hey @lammertw. We need more information to resolve this issue but there hasn't been an update in 5 weekdays. I'm marking the issue as stale and if there are no new updates in the next 5 days I will close it automatically.

If you have more information that will help us get to the bottom of this, just add a comment!

[lammertw]

We just released a new version of our app a couple of days ago that reduces the file protection level for Firestore's backing data. Unfortunately we're still getting the same crash just as much as before.

[vizaiapp]

I'm experiencing the same crash with a brand new macOS project using the (unofficial) Firebase pods. I first configure Firebase in the application delegate, then fetch a document from Firestore if the current user is set. All of this happens on the main thread and without any modification to the entitlements.

[elai950]

Receiving this error too:

*** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'FIRESTORE INTERNAL ASSERTION FAILED: Failed to open DB: Internal: Failed to open LevelDB database at /Users/user/Library/Developer/CoreSimulator/Devices/75E41D4A-1A35-4B35-A073-9591F9932BDF/data/Containers/Data/Application/F9074D72-BDD3-4A9F-8AB6-6E77DA2571C3/Library/Application Support/firestore/__FIRAPP_DEFAULT/project-25f32/main: LevelDB error: IO error: lock /Users/user/Library/Developer/CoreSimulator/Devices/75E41D4A-1A35-4B35-A073-9591F9932BDF/data/Containers/Data/Application/F9074D72-BDD3-4A9F-8AB6-6E77DA2571C3/Library/Application Support/firestore/__FIRAPP_DEFAULT/project-25f32/main/LOCK: Resource temporarily unavailable (expected created.ok())'
terminating with uncaught exception of type NSException
CoreSimulator 757.3 - Device: iPhone 12 (75E41D4A-1A35-4B35-A073-9591F9932BDF) - Runtime: iOS 14.4 (18D46) - DeviceType: iPhone 12

It crash on the very beginning, right after the launch screen disappears. Running on the new M1 MacBook Pro, this wasn't shown on my previous MacBook Pro using the Intel chip. I need to clean the build folder, restart the simulator, sometimes even the system needs to reboot, to run the app on the simulator.

[omerozk]

Exactly same issue than @elai950, with same iPhone 12 (simulator) iOS 14.4 version and also on M1 Pro. (maybe we should create a new issue 🤔

[tysonwil]

@vizaiapp any luck with this crash on macOS?

[Moriquendi]

This issue happens for me when I run the app in SwiftUI Preview, and then try to launch it. Xcode keeps the app's process alive - even after the Preview is no longer visible in Xcode. Having those two processes open at the same time leads to this error.

How to fix:

1. Open Activity Monitor app
2. Find process of your app
3. Kill it

It's annoying to deal with so if anyone from Firebase team can fix it, that'd be great :)

[lengocgiang]

You can using static library. Download firebase firestore at https://firebase.google.com/docs/ios/setup#frameworks

In Podfile add

target '{project_name}' do
  # Comment the next line if you don't want to use dynamic frameworks
  use_frameworks! :linkage => :static
end

references: https://firebase.google.com/docs/ios/link-firebase-static-dynamic

[santiotin]

@lammertw I have the same issue that you described, you were able to solve it? Thanks

[mastercodekw]

I have the same issue...

[bouraine]

Same issue with a swift share-extension. MBP M1 chip. i have never seen this error in my macbook intel chip. to solve the problem.

1- Clean derived data folder: /Users/XXXX/Library/Developer/Xcode/DerivedData 2 - Restart simulator/xcode

[coffmark]

I have the same issue, too...

[r-a-o]

Can this happen on real devices? if yes, how do I recover from it without asking users to delete/reinstall the app?

[Livinglist]

Same on real iOS device...

[courtneypresto]

Same on a real iOS device too.

It's happening ~ 1% of the time to our users. Would love an update!

[lammertw]

@lammertw I have the same issue that you described, you were able to solve it? Thanks

@santiotin No we're still having the issue

[iammike]

Also seeing this issue on an M1, but only in our UI testing scheme. No problem for anyone with an Intel chip.

[jhoanarango]

I've been running into this issue as well, I am on a M1 Max MacbookPro. I do noticed what @Moriquendi mentioned above. Which is when I run Activity Monitor, I can see the app still running in memory even if Xcode and the Simulator are closed.

[vishnunairvp]

I am getting the same crash when try to show unity window from iOS both using firebase dependencies. Can anyone please explain the cause and provide solution?

[davidseek]

Having the same issue randomly on my M1 macOS project. Using Swift Package manager. Latest versions. Cleared derived data and reset the cache of the package manager. Still not working.

FIRESTORE INTERNAL ASSERTION FAILED: Failed to open DB: Internal: Failed to open LevelDB
database at /Users/<ME>/Library/Application Support/firestore/__FIRAPP_DEFAULT/<APP_ID>/main: LevelDB error: 
IO error: lock /Users/<ME>/Library/Application Support/firestore/__FIRAPP_DEFAULT/<APP_ID>/main/LOCK: 
Resource temporarily unavailable (expected created.ok())

What helped me was to disable persistence. I started having the issue, when I started implementing Unit Tests. I assume, that the Tests created some kind of 2nd instance. Running for now without persistence, and will then figure out how to add it for production, but disable it for Unit Tests.

let settings = FirestoreSettings()
settings.isPersistenceEnabled = false

let firestore = Firestore.firestore()
firestore.settings = settings

[michzio]

I've got similar error with Firestore after unsintalling app on simulator and reinstalling it, and it happens all the time now. Before everything seems to work correctly :( I'm also using M1 Macbook it this have anything related to this issue. I've uninstall app again, make clean build, removed derived data, etc. and now it started to work again. But if it may crash it in production for significant number of users I am a little scared, and maybe I should turn off this local persistance?

[cherylEnkidu]

Hi, This is a known issue and a work around is using the memory persistence instead of disk persistence.

[ekoirsyad]

It does happen to me as well. After uninstalling app on my simulator and adding more collection to the firestore manually, it solved. Still wondering what happen tho.

[matthieudelaro]

I'm on Intel processor. Killing the app process (present even though the app was closed) in Activity Monitor solved the issue.

[pohy]

I have faced this issue using React Native Firebase library. What resolved the issue in the end was restarting my computer.🙃

[ilnur-nazmutdinov-spark]

We’ve found a way to reproduce it on iPhone. You will need an iPhone and an Apple Watch.

1. Schedule an actionable notification in your app.
2. Reboot your phone and don’t unlock it.
3. Wait for the notification to come on Apple Watch.
4. Press a notification action button on Apple Watch. The app will crash.

When iPhone is rebooted you can’t press action buttons on notifications, but you can do it using Apple Watch.

[dconeybe]

@ilnur-nazmutdinov-spark As mentioned in previous comments https://github.com/firebase/firebase-ios-sdk/issues/5423#issuecomment-617309806 and https://github.com/firebase/firebase-ios-sdk/issues/5423#issuecomment-617376885, Firestore will crash if persistence is enabled and the file system is locked. The reproduction steps you listed definitely result in Firestore attempting to open its database which the file system is locked. Unfortunately, we don't have a fix for this, and don't have any immediate plans to fix it (it's a complicated issue). Please check the workarounds outlined in those previous comments, or disable persistence if your app doesn't need it.

[jhoanarango]

Ok, so I am running into this issue more often. I do need persistence on. What is any other workaround other than turning off persistance?

[dconeybe]

@jhoanarango The other workaround is to avoid using Firestore when the file system is locked.

[jhoanarango]

@dconeybe I do not use Firestore when the file system is locked. Firestore is only used while the app is running.

[dconeybe]

@jhoanarango Could you post your stack trace then? Perhaps it is a different crash that looks similar.

[jesus-mg-ios]

@dconeybe how we know when the system is unlocked to use safely the persistence. Do you have something in mind?

[dconeybe]

If you have 10 minutes, take a read of https://github.com/firebase/firebase-ios-sdk/issues/2846, and old issue that documents a few potential workarounds.

The most notable workaround IMO would be downgrading from NSFileProtectionComplete to NSFileProtectionCompleteUntilFirstUserAuthentication (or another more permissive setting).

https://developer.apple.com/documentation/foundation/nsfileprotectioncomplete

[jesus-mg-ios]

@paulb777 I think that's possible to add a boolean for check if the Firestore db can be opened. Isn't it? I'm thinking on checking if the user keychain value can be retrieved. In this way, if firebase change the query to set/get this value we don't need to change a check logic, because the check logic is on your side and we use the boolean to retry it in a time or use no persistent storage.

CC: @unxavi

Thanks in advance

[DevboiDesigns]

This same issue is still occuring for me on an M1. Only happens on the Simulator not a real device.

[dconeybe]

@DevboiDesigns If you have a reproducible scenario, please open a new issue and mention my GitHub username @dconeybe and this issue. Make sure to mention your xcode version, iOS version on the simulator, and, if possible, an app that I could clone from GitHub to reproduce.

[jesus-mg-ios]

Any updates on this? For example, managing the crash and retrying instead of crashing? @ncooke3 @dconeybe

[jesus-mg-ios]

@ncooke3 isPersistenceEnabled is deprecated, which is the workaround with the new way?

[jesus-mg-ios]

And @ncooke3 is it possible that the Firebase instance adds the lock method and the app crashes at that moment, and then when the app initiates once more firebase thinks this is not available?

[wu-hui]

@ncooke3 isPersistenceEnabled is deprecated, which is the workaround with the new way?

Use this instead of isPersistenceEnabled: https://firebase.google.com/docs/reference/swift/firebasefirestore/api/reference/Classes/FirestoreSettings#cachesettings

And @ncooke3 is it possible that the Firebase instance adds the lock method and the app crashes at that moment, and then when the app initiates once more firebase thinks this is not available?

The SDK does not keep states between restarts, this issues happens when the OS tell the SDK you cannot access the filesystem, which we rely on for the SDK to function (with persistence cache turned on).

We will consider throw an exception instead of crashing, there is no timeline for this though, sorry.

[jesus-mg-ios]

It's weird @wu-hui in the last ios version I tried to disable persistence on my extension, but it seems to be crashing anyway for the same reason, Something weird with iOS 17.4.1 in all topics, but this makes me think about whether disabling persistence when persistence was enabled on an update before works as expected. And what happens if a transaction is on the air when the extension is closed or killed?

[wu-hui]

Do you mind sharing a minimum repro if possible?

[abdulrehman0796]

We have disabled persistence and still getting the crash. iOS version getting crashes: 17.5.1 and 17.4.1 Firestore version used: v10.18.0

How to fix this issue? Should we upgrade the plugin version. But in change logs their is no change related to this issue so looks unlikely it will fix on upgrade. We are disabling firestore all togethor until this issue is resolved.