[Realm compact] - io.realm.exceptions.RealmFileException: Unable to open a realm at path 'xxx.realm' Realm file initial open failed: Invalid streaming format cookie (0) Path:Exception backtrace: <backtrace not supported on this platform>

Oshuremy commented 1 year ago

How frequently does the bug occur?

Sometimes

Description

Hi,

I have the same issue than https://github.com/realm/realm-java/issues/4237 for the same same problem.

I have 32.000 crashes for 50 users (for 250.000 users in the month)

I don't find any solution, can you help me please ?

Stacktrace & log output

Caused by io.realm.exceptions.RealmFileException
Unable to open a realm at path '/data/user/0/com.app.androidfr/files/config.realm': Realm file initial open failed: Invalid streaming format cookie (0) Path:Exception backtrace: <backtrace not supported on this platform> Path: /data/user/0/com.app.androidfr/files/config.realm Exception backtrace: <backtrace not supported on this platform>. (Realm file initial open failed: Invalid streaming format cookie (0) Path: /data/user/0/com.app.androidfr/files/config.realm Exception backtrace: <backtrace not supported on this platform> Path: /data/user/0/com.app.androidfr/files/config.realm Exception backtrace: <backtrace not supported on this platform>) (/data/user/0/com.app.androidfr/files/config.realm) in /tmp/realm-java/realm/realm-library/src/main/cpp/io_realm_internal_OsSharedRealm.cpp line 107

io.realm.internal.OsSharedRealm.nativeGetSharedRealm (OsSharedRealm.java)
io.realm.internal.OsSharedRealm.<init> (OsSharedRealm.java:174)
io.realm.internal.OsSharedRealm.getInstance (OsSharedRealm.java:259)
io.realm.internal.OsSharedRealm.getInstance (OsSharedRealm.java:247)
io.realm.BaseRealm.compactRealm (BaseRealm.java:814)
io.realm.Realm.compactRealm (Realm.java:1954)
com.app.helper.realm.ConfigModule$Companion.compact (ConfigModule.kt:47)
com.app.App.initRealm (App.kt:70)
com.app.App.onCreate (App.kt:33)

Can you reproduce the bug?

No

Reproduction Steps

No response

Version

10.13.0

Are you using encryption?

No

Platform OS and version(s)

Android

Build environment

Android Build Tools version: 7.6-bin.zip Gradle version: 7.4.1

clementetb commented 1 year ago

This error looks like a corrupt file to some degree, are you using compaction? Are you doing any file manipulation like copying files or have backup enabled?

Oshuremy commented 1 year ago

I'm using compacting but not copying or nothing else

rorbech commented 1 year ago

@Oshuremy Are there any chance of getting hold of a realm file that throws this error?

rorbech commented 1 year ago

@Oshuremy Or can you see any issues/events report prior to getting into this state? It would greatly help to reproduce if we had some idea of the events leading up to this, e.g. migration, compation, out of storage, etc.

0neel commented 1 year ago

We've got a similar crash after updating from 10.8.0 to 10.13.2-transformer-api. I believe the most of issues we have with realm come from the device having not enough storage. But not in this case - users have plenty of storage available.

Our stacktrace is different though:

Fatal Exception: io.realm.exceptions.RealmFileException: Unable to open a realm at path '/data/user/0/<our_app_package_name>/files/default.realm': Realm file initial open failed: Invalid streaming format cookie (0) Path:Exception backtrace:
<backtrace not supported on this platform> Path: /data/user/0/<our_app_package_name>/files/default.realm
Exception backtrace:
<backtrace not supported on this platform>. (Realm file initial open failed: Invalid streaming format cookie (0) Path: /data/user/0/<our_app_package_name>/files/default.realm
Exception backtrace:
<backtrace not supported on this platform> Path: /data/user/0/<our_app_package_name>/files/default.realm
Exception backtrace:
<backtrace not supported on this platform>) (/data/user/0/<our_app_package_name>/files/default.realm) in /tmp/realm-java/realm/realm-library/src/main/cpp/io_realm_internal_OsSharedRealm.cpp line 107
       at io.realm.internal.OsSharedRealm.nativeGetSharedRealm(OsSharedRealm.java)
       at io.realm.internal.OsSharedRealm.<init>(OsSharedRealm.java:174)
       at io.realm.internal.OsSharedRealm.getInstance(OsSharedRealm.java:259)
       at io.realm.BaseRealm.<init>(BaseRealm.java:142)
       at io.realm.BaseRealm.<init>(BaseRealm.java:109)
       at io.realm.Realm.<init>(Realm.java:161)
       at io.realm.Realm.createInstance(Realm.java:535)
       at io.realm.RealmCache.createInstance(RealmCache.java:508)
       at io.realm.RealmCache.doCreateRealmOrGetFromCache(RealmCache.java:461)
       at io.realm.RealmCache.createRealmOrGetFromCache(RealmCache.java:422)
       at io.realm.Realm.getDefaultInstance(Realm.java:443)
       at <our_app_package_name>.MainActivity.onStart(MainActivity.kt:139)

Oshuremy commented 1 year ago

Are there any chance of getting hold of a realm file that throws this error?

I have no idea how to reproduce this case..

Or can you see any issues/events report prior to getting into this state? It would greatly help to reproduce if we had some idea of the events leading up to this, e.g. migration, compation, out of storage, etc.

Honestly I'm a bit lost with this crash.

I have no migration : deleteRealmIfMigrationNeeded()
I had a crash for compactRealm so I let him optionnal with a try/catch but now It's crashing just below when I want to delete some realm Item in my configuration
I don't think it's a storage issu, Firebase Crashlytics told me that user crash even if there is 1Go Ram free and 25Go Disk free.

This problem doesn't touch every one just a little part of my users but they can crash 10.000 time a day for 20 users

I have an idea for that. A try/catch of the first use of Realm, on the catch I remove Realm files entirely and I restart th init from 0. What do you think about that ?

Oshuremy commented 1 year ago

up @rorbech

sipersso commented 1 year ago

I am running into the same issue here.

sipersso commented 1 year ago

The strange thing is that the realm version is unchanged from the last version of my app 10.13.0. But it definitely started only for the latest version, so it must be that something else in the toolchain has caused compatibility issues.

EDIT: The above info is incorrect. I was on staged rollouts so had low usage of the previous versions. I just checked my crash logs and it looks like this change started happening after upgrading from Realm 10.12.0 to 10.13.0.

rorbech commented 1 year ago

@Oshuremy It is a valid approach to delete the realm file if failing to load and the reinitialize data in it, if that is an option.

Do you happen to have a trace of the crash during the compactRealm? Because I haven't been able to get insight or information about how to provoke this from the core team.

sipersso commented 1 year ago

@rorbech for me, deleting the realm file means lost data, so is not an option. Users will have to do that anyway now, but it does mean data loss.

Oshuremy commented 1 year ago

@rorbech I launched a new version yesterday and it this to work because Realm data are optionnal on our app. I will confirm that after several days.

Here you can find the crash report for compactRealm :

Fatal Exception: io.realm.exceptions.RealmFileException
Unable to open a realm at path '/data/user/0/com.x.x/files/helper.realm': Realm file initial open failed: Invalid streaming format cookie (0) Path:Exception backtrace: <backtrace not supported on this platform> Path: /data/user/0/com.x.x/files/helper.realm Exception backtrace: <backtrace not supported on this platform>. (Realm file initial open failed: Invalid streaming format cookie (0) Path: /data/user/0/com.x.x/files/helper.realm Exception backtrace: <backtrace not supported on this platform> Path: /data/user/0/com.x.x/files/helper.realm Exception backtrace: <backtrace not supported on this platform>) (/data/user/0/com.x.x/files/helper.realm) in /tmp/realm-java/realm/realm-library/src/main/cpp/io_realm_internal_OsSharedRealm.cpp line 107

io.realm.internal.OsSharedRealm.nativeGetSharedRealm (OsSharedRealm.java)
io.realm.internal.OsSharedRealm.<init> (OsSharedRealm.java:174)
io.realm.internal.OsSharedRealm.getInstance (OsSharedRealm.java:259)
io.realm.internal.OsSharedRealm.getInstance (OsSharedRealm.java:247)
io.realm.BaseRealm.compactRealm (BaseRealm.java:814)
io.realm.Realm.compactRealm (Realm.java:1954)
com.x.helper.realm.HelperModule$Companion.compact (HelperModule.kt:35)
com.x.App.initRealm (App.kt:74)
com.x.App.onCreate (App.kt:35)
android.app.Instrumentation.callApplicationOnCreate (Instrumentation.java:1193)
android.app.ActivityThread.handleBindApplication (ActivityThread.java:6731)
android.app.ActivityThread.access$1300 (ActivityThread.java:238)
android.app.ActivityThread$H.handleMessage (ActivityThread.java:1914)
android.os.Handler.dispatchMessage (Handler.java:106)
android.os.Looper.loop (Looper.java:223)
android.app.ActivityThread.main (ActivityThread.java:7698)
java.lang.reflect.Method.invoke (Method.java)
com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run (RuntimeInit.java:592)
com.android.internal.os.ZygoteInit.main (ZygoteInit.java:952)

@sipersso Don't you have a backend server and a real database to save those data ?

sipersso commented 1 year ago

@rorbech I launched a new version yesterday and it this to work because Realm data are optionnal on our app.

I will confirm that after several days.

Here you can find the crash report for compactRealm :


Fatal Exception: io.realm.exceptions.RealmFileException

Unable to open a realm at path '/data/user/0/com.x.x/files/helper.realm': Realm file initial open failed: Invalid streaming format cookie (0) Path:Exception backtrace: <backtrace not supported on this platform> Path: /data/user/0/com.x.x/files/helper.realm Exception backtrace: <backtrace not supported on this platform>. (Realm file initial open failed: Invalid streaming format cookie (0) Path: /data/user/0/com.x.x/files/helper.realm Exception backtrace: <backtrace not supported on this platform> Path: /data/user/0/com.x.x/files/helper.realm Exception backtrace: <backtrace not supported on this platform>) (/data/user/0/com.x.x/files/helper.realm) in /tmp/realm-java/realm/realm-library/src/main/cpp/io_realm_internal_OsSharedRealm.cpp line 107

io.realm.internal.OsSharedRealm.nativeGetSharedRealm (OsSharedRealm.java)

io.realm.internal.OsSharedRealm.<init> (OsSharedRealm.java:174)

io.realm.internal.OsSharedRealm.getInstance (OsSharedRealm.java:259)

io.realm.internal.OsSharedRealm.getInstance (OsSharedRealm.java:247)

io.realm.BaseRealm.compactRealm (BaseRealm.java:814)

io.realm.Realm.compactRealm (Realm.java:1954)

com.x.helper.realm.HelperModule$Companion.compact (HelperModule.kt:35)

com.x.App.initRealm (App.kt:74)

com.x.App.onCreate (App.kt:35)

android.app.Instrumentation.callApplicationOnCreate (Instrumentation.java:1193)

android.app.ActivityThread.handleBindApplication (ActivityThread.java:6731)

android.app.ActivityThread.access$1300 (ActivityThread.java:238)

android.app.ActivityThread$H.handleMessage (ActivityThread.java:1914)

android.os.Handler.dispatchMessage (Handler.java:106)

android.os.Looper.loop (Looper.java:223)

android.app.ActivityThread.main (ActivityThread.java:7698)

java.lang.reflect.Method.invoke (Method.java)

com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run (RuntimeInit.java:592)

com.android.internal.os.ZygoteInit.main (ZygoteInit.java:952)

@sipersso Don't you have a backend server and a real database to save those data ?

I only use realm as a local database for now, but this isn't relevant to this issue. Realm IS a real database and I should be able to trust that the data doesn't get randomly corrupted, regardless of what server solution I may use.

sipersso commented 1 year ago

FIY: I just downgraded my app from Realm 10.13.0 to 10.12.0 and the issues are now gone. So this is definitely bug that was introduced in 10.13.0.

@Oshuremy 10.12.0 use the same file format, so downgrading until this has been resolved is a possible workaround.

sipersso commented 1 year ago

Looks like the core version was updated in the 10.14.0 and contains some compaction fixes. Have you tried upgrading @Oshuremy? I have been told it might fix the issue even though Realm told me that they can't know for sure.

cmelchior commented 1 year ago

Note that Realm Java 10.7.0 introduced a new backup system when upgrading the Realm file format, like with 10.14.0. We will back up the current file and keep it for 3 months. If you then downgrade, that backup file will be used again. You can see more here: https://github.com/realm/realm-core/blob/master/CHANGELOG.md#1100-beta3-release-notes

So upgrading should be safe and there is a path for downgrading if needed.

sipersso commented 1 year ago

@cmelchior I am not sure how that helps to be honest. If I upgrade, the user adds new data to the new upgraded realm file, then gets an error that corrupts the new database, they will lose all data that has been added to the database since the upgrade to the new file format right? This essentially makes it pointless to use the backup file. Or am I missing something here?

EDIT: I understand that this will help a lot once I am actually on Atlas, but not for local realms.

cmelchior commented 1 year ago

Yes, that is correct. But the bugs in question here should have been addressed in the last version (10.14.0), so it is a bit unclear exactly what you are looking for?

sipersso commented 1 year ago

I was looking for the answer you just gave me. That the bug should have been addressed in the last version. The comments in https://github.com/realm/realm-java/issues/7779 didn't make me confident that this was the case.

EDIT: And the ticket is still open, so I just assumed that it wasn't confirmed that is was fixed.

joaodevsantos commented 11 months ago

I'm also facing this issue

realm / realm-java