Issue: java.lang.IllegalArgumentException: Illegal Argument: Invalid format of Realm file.

[ghost]

We have seen this crash alot recently , 15 users 2500 times it crashed :(

Devices: Lenovo, Samsungs, Blackberry

Realm version : io.realm:realm-gradle-plugin:1.1.0

We are not using compact ..

Code to initialize: Realm.getInstance(new RealmConfiguration.Builder(GlobalInstance.getContext()) .deleteRealmIfMigrationNeeded() .build())

Stack trace: Caused by: java.lang.IllegalArgumentException: Illegal Argument: Invalid format of Realm file. at io.realm.internal.SharedGroup.createNativeWithImplicitTransactions(Native Method) at io.realm.internal.SharedGroup.(SharedGroup.java:64) at io.realm.internal.SharedGroupManager.(SharedGroupManager.java:49) at io.realm.BaseRealm.(BaseRealm.java:76) at io.realm.Realm.(Realm.java:138) at io.realm.Realm.createAndValidate(Realm.java:268) at io.realm.Realm.createInstance(Realm.java:248) at io.realm.RealmCache.createRealmOrGetFromCache(RealmCache.java:114) at io.realm.Realm.getInstance(Realm.java:213) at x.y.z.presistance.RealmManager.getDefaultInstance(RealmManager.java:20) at x.y.z.presistance.RealmRepository.(RealmRepository.java:17) at x.y.z.presistance.RealmLocationRepository.(RealmLocationRepository.java:26)

Any more detail required please tell thanks

[Zhuinden]

Do you use encryption?

[kneth]

If you have an invalid Realm file, you are welcome to send it to help@realm.io, Then we can examine it.

[ghost]

No encryption and, i am not clear about realm file, plus we don't have access to the 15 devices facing it as its happening for them on Prod and we are unable to reproduce it at our end

[beeender]

Is it possible that you are catching some exceptions thrown by Realm and not properly handle them?

[ghost]

No, there is no try catch in our realm manager plus other realm related files...

[beeender]

So it happens only after the Realm updated to 1.1.0? How about before? Can you share your crashylitics link with us? to help@realm.io if you want to share it privately.

[ghost]

it existed even before but the frequency was very less, we were on 0.86 then jumped to 0.88, then to 1.0.1 then to 1.1.0

the frequency has skyrocketed after 1.0.1 to 1.1.0 upgrade... before that 35 users had faced it around 500 times only but this last jump has made this one crazy :/

7.2.0(7206) is the last build with 1.1.0 realm shipped about 36 hours ago Version Occurrences (Last 30 Days) 7.2.0 (7206)
2569 6.9.0 (6909) 194 6.8.0 (6803) 101 7.0.0 (7006) 92 7.1.0 (7109) 37 6.7.0 (6701) 35 6.5.1 (6510) 33 6.1.2 (6120) 12 6.7.1 (6712) 2

THANKS for all the help and timely responses, BLESS you :)

[beeender]

I saw you mentioned blackberry. There is an known issue with blackberry -- two system calls don't behave correctly which could potentially damage the db. See https://github.com/realm/realm-java/issues/3178#issuecomment-235500240

The above issue has been fixed in 1.1.1

But I am not quite sure about the problems on other devices, maybe they have similar issues with system calls? Can you please also list those devices I will try if I can find any of them.

I suggest you update it to 1.1.1 and also, if the db is corrupted already, updating won't fix it. Maybe you can consider to catch the "invalid db" exception at the first time open the Realm and delete the Realm there.

[ghost]

Device List attached via screenshots:

Can you shed more light on Catching the exception ? which exception and when ? THanks alot for the instant help.

[ghost]

Ok you mean catching this Exception ?

java.lang.IllegalArgumentException: Illegal Argument: Invalid format of Realm file. right ?

[kneth]

@baboo86 Yes, that's the one. But of course, only if your app can recreate the data e.g., if you use Realm for caching something.

[carlbenson]

I get this only on release builds on Samsung S7. WIth or without encryption.

[kneth]

@carlbenson Thanks for sharing that observation. Can you reproduce it consistently? We would be very happy if you can reproduce it - and are able to share either source code of your app or an apk (you can send it privately to help@realm.io).

[carlbenson]

I can reproduce it with 100 % accuracy. If I create a minimal app creating only a basic realm, the crash does not occur. I will send a debug and a release build with encryption turned off since it does not seem to affect anything.

[carlbenson]

I also just circumvented the error by catching the exception, deleting the DB with the same configuration, and then creating a new identical configuration.

basically

override fun onStart() {
    super.onStart()
    try {
        presenter.onStart(this) // this is where realm.getDefaultInstance() is called
    } catch (e: IllegalArgumentException) {
        Realm.deleteRealm(realmConfig)

        val realmConfig = RealmConfiguration.Builder(this) // this is identical with the prior config

        with(realmConfig) {
            schemaVersion(9)
            migration(DreamsRealmMigration())
            if (!BuildConfig.DEBUG) {
                encryptionKey(getKeyPair(this@StartActivity, "dreams").public.encoded.copyOfRange(0, 64))
            }
        }

        Realm.setDefaultConfiguration(realmConfig.build())
    }
    presenter.onStart(this) // works without exception.
    presenter.getConfiguration()
}

[carlbenson]

The exception occurs again on app restart, so I would have to delete the DB every time (or do some backup/restore mechanism which feels cumbersome).

[kneth]

@carlbenson Thanks for the details. We will find/borrow a Samsung S7 so we can investigate it further.

[Qubitium]

@kneth @beeender I am literally praying that @carlbenson found a reproduceable case. We have seen these error since last year so there needs to be a way to expose more stats from core at the point of this error. For example, at least have the core spit out the specifics of of the invalid format in the error string as string or numeric error code. Was it caused by bad block header, block data, version header, row header, row data, etc. I am just creating phantom error types there to show that the current catch call-all generic exception message is helping no one to expedite the debugging of this bug.

[beeender]

Internal notes: APK can be found https://secure.helpscout.net/conversation/237660564/5891/?folderId=366141

[carlbenson]

Unfortunately it was reproducible at 100 % with the release build of our app, but it seemed it was tied to two of my co-worker's Samsung S7 phones. We had at least one known customer who used the app without this problem, and it suddenly magically disappeared on my co-worker's phones too.

[kneth]

@carlbenson It sounds like a subtle timing issue which will be very hard to reproduce. Did you co-worker upgrade the S7 phones in any way?

[Qubitium]

@carlbenson Did you also upgrade dexguard/proguard in any way? We have found that proguard/dexguard changes can also drastically change the timing of executed codes.

[kneth]

@diegomontoya It is possible to share a few example of which proguard/dexguard changes lead to timing issue?

[Qubitium]

@kneth we have seen it when dexguard's anti-tampering/anti-debugging apis are used within the codebase. Proguard shouldn't have this issue.

[andrewheckford]

@kneth I stumbled across this thread as I've had one of our customers report a similar problem with our Android app running Realm 1.0.0. Our ACRA crash logs reports a stack as follows:

java.lang.IllegalArgumentException: Illegal Argument: Invalid format of Realm file. at io.realm.internal.SharedGroup.createNativeWithImplicitTransactions(Native Method) at io.realm.internal.SharedGroup.<init>(SharedGroup.java:64) at io.realm.internal.SharedGroupManager.<init>(SharedGroupManager.java:49) at io.realm.BaseRealm.<init>(BaseRealm.java:79) at io.realm.Realm.<init>(Realm.java:138) at io.realm.Realm.createAndValidate(Realm.java:248) at io.realm.Realm.createInstance(Realm.java:228) at io.realm.RealmCache.createRealmOrGetFromCache(RealmCache.java:126) at io.realm.Realm.getDefaultInstance(Realm.java:174)

This is running on a Moto E2 handset with Lollipop 5.0.2. This has never shown up during our own internal product testing. I've obtained the .realm file from the handset and attempted to view it using Realm Browser. However, Realm browser seems to think it is encrypted and asks for the decrypt key. Which is strange as we don't use encryption in the app! Assuming I can get clearance from our customer, would you be interested in taking a look at the .realm file for diagnosis? Should I send that to help@realm.io?

[Zhuinden]

You should definitely send the Realm file to help@realm.io

[andrewheckford]

@Zhuinden Thanks. Once I've got clearance, I'll email the file.

[kneth]

@andrewheckford If the browser thinks the Realm file is encrypted, and you know it isn't, it is corrupted. It will be of great interest to get a copy of the file to see what the corruption is. Do you give us more details on what happened before? I mean, did you compact the file? Did the app crash and restart?

[andrewheckford]

@kneth We don't have much to go on, I'm afraid. To give you some background, our app is shipped pre-installed on phones in rugged cases sent to our customers. The phone was returned to me via our customer support team with the app failing to start showing the "Invalid format of Realm file" error message. I'll ask our tech support team to go back and try and get some more background. What I can tell you is that we compact the database after ever sync with our cloud service which runs hourly. In the meantime I've email you the file to help@realm.io (subject: Corrupted realm file for https://github.com/realm/realm-java/issues/3264). I'll be very interested to understand if we're doing something wrong on our side with our usage of Realm. Thanks!

[kneth]

Internal note: we have received files from @andrewheckford and @finnschiermer will have a look.

[finnschiermer]

@andrewheckford The file is not encrypted but appears to have been truncated. Otherwise the file looks intact. The file is not on the form generated by compaction, so it is likely to have been generated by a commit during normal operation. Possibly causes could be an error on our side, but it could also be that the file is written from somewhere else while already opened. This leads me to ask: how do you coordinate the "sync with our cloud service" you mention above. Is there any chance the file may be updated by your sync while already in use as a Realm file?

[andrewheckford]

@finnschiermer There shouldn't be anything else writing to the Db during sync (there may be reads). However, we did introduce a feature recently that results in both a background service and activity classes will be writing to the Db during a logged-in user session. We don't hold open database connections for either services or activities: I get a realm instance, do the write and then immediately close it. Is there a risk of some sort of race condition if two different threads could be writing the Db at the same time?

[finnschiermer]

@andrewheckford : When you write "there may be reads", do you mean that another process (or thread) may be reading from the realm file (e.g. to copy it elsewhere) while it is still open through one or more Realm objects?

With regard to your question: As long as every write to the realm file goes through a Realm object as part of a write transaction, we ensure full transaction separation.

[Zhuinden]

@andrewheckford You can't have multiple transactions (writes) at the same time with transactions started within the same process.

[andrewheckford]

@finnschiermer Every write does indeed go through a Realm object as part of a transaction. So it sounds like we should be safe there.

When I say "there may be reads", I mean we may have the UI thread reading from Realm whilst the sync adapter is writing. For reads we follow this pattern: get the default instance, iterate through RealmResults, copy the individual properties from the managed Realm objects into our own POJOs (which are un-managed and don't implement RealmModel) and then immediately close the instance before the screen is drawn. We never hold open the instance connection and the UI doesn't use managed Realm objects directly.

All access to the Realm file is done via the Realm API, we don't touch the file on disk directly.

[finnschiermer]

Ok. No obvious explanation then. It is likely we'll need a repro-case to make further progress.

[andrewheckford]

@finnschiermer Ok. For the time being I'll trap the exception and regenerate the db as other folks have. We don't have any mission critical data in there at the moment so we can go from an empty clean db. In time, we are hoping to move more of our data out of our old flat-file db into Realm. I'm guessing we could also use Realm.writeCopyTo() to make periodic backups and then restore from one of these if the crash happens?

[kneth]

Backing up Realms is a FAQ: https://realm.io/docs/java/latest/#how-to-back-up-and-restore-realm

But yes, writeCopyTo() is the core of doing backup :-)

[kneth]

I'll close the issue. Don't hesitate to reopen or create new issue if you see the problem again or have any updates.

[imidhuntv]

We have a GCM Receiver which extends From FirebaseMessagingService like below.

`public class GcmReceiver extends FirebaseMessagingService {

private static String TAG = "GcmReceiver";

@Inject
PubNubObserverDelegate mDelegate;

@Override
public void onMessageReceived(RemoteMessage message) {
    String from = message.getFrom();
    Map data = message.getData();
    if (!from.equals(BuildConfig.GCM_SENDER_ID))
    {
        return;
    }

    App.getComponent().inject(this);
    JSONObject json = new JSONObject();
    Set<String> keys = data.keySet();
    for (String key : keys) {
        try {
            // json.put(key, bundle.get(key)); see edit below
            json.put(key, data.get(key));
        } catch(JSONException e) {
            //Handle exception here
            e.printStackTrace();
        }
    }
    String channel = null;
    if (data.containsKey("channelId"))
        channel = data.get("channelId").toString();
    if (channel != null)
        mDelegate.showChatNotification(channel, json);
    else
        Log.v(TAG, "Ignore non chat message");
}

}`

In mDelegate.showChatNotification(channel, json) method we get realm instance and write the message to realm. We haven't specified any process for GCM receiver. Could this be the problem in anyways? Is GCM receiving the message in another process and is our crash happening while we open the app while GCM is writing to realm? Any help?

[imidhuntv]

This thread increases my suspicion

[Zhuinden]

Well the question is if your Realm is encrypted.

Multiprocess access does work for unencrypted Realms since 2.0.x. It does not work for encrypted Realm's.

[imidhuntv]

Our Realm is not encrypted. Hope 2.0.x fixes my issue.

[morder]

We got an error java.lang.IllegalArgumentException: Illegal Argument: Invalid format of Realm file This will happen after implementing compact and closing realm instances in all threads (we have an error with this) https://gyazo.com/6b15101f7ddf00ff45df5c5c77dce166

Well, we have got DB files from several users and we can reproduce this error.

io.realm:realm-gradle-plugin:1.2.0

[kneth]

@morder If you can send one of the corrupt Realm files to help@realm.io we will be very interested. Then we can learn how it is corrupted which might give us a clue how it happened.

[morder]

@kneth just sent, wait for answer from realm team :)