Illegal Argument Exception encountered when compressing a Realm file created by version 0.83.0

[creativepsyco]

Hi peeps,

I am currently running into an IllegalArgumentException indicating that the format of the Realm file is invalid when I am trying to call Realm.compactRealm(realmConfiguration...) before I use the realm file.

I have a sample project to demonstrate the crash here: https://github.com/creativepsyco/sample-realm-crash/

Essentially I am trying to compact the realm file before it is ever used in the app.

Steps to reproduce:

1. Clone the project from https://github.com/creativepsyco/sample-realm-crash/
2. Compile and run. It will not crash because realm version is 0.82.2
3. Now in build.gradle edit the version to 0.83.0
4. Compile and run the app.
5. It crashes with the stack trace:

 FATAL EXCEPTION: main
 Process: mohitkanwal.com.examplerealm, PID: 28687
 java.lang.RuntimeException: Unable to start activity
ComponentInfo{mohitkanwal.com.examplerealm/mohitkanwal.com.examplerealm.MainActivity}:\
java.lang.IllegalArgumentException: Illegal Argument: Invalid format of Realm file.
     at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:3119)
     at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:3218)
     at android.app.ActivityThread.access$1000(ActivityThread.java:198)
     at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1676)
     at android.os.Handler.dispatchMessage(Handler.java:102)
     at android.os.Looper.loop(Looper.java:145)
     at android.app.ActivityThread.main(ActivityThread.java:6837)
     at java.lang.reflect.Method.invoke(Native Method)
     at java.lang.reflect.Method.invoke(Method.java:372)
     at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1404)
     at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1199)
  Caused by: java.lang.IllegalArgumentException: Illegal Argument: Invalid format of Realm file.
     at io.realm.internal.SharedGroup.createNativeWithImplicitTransactions(Native Method)
     at io.realm.internal.SharedGroup.<init>(SharedGroup.java:68)
     at io.realm.internal.SharedGroupManager.<init>(SharedGroupManager.java:49)
     at io.realm.BaseRealm.<init>(BaseRealm.java:87)
     at io.realm.Realm.<init>(Realm.java:151)
     at io.realm.Realm.createAndValidate(Realm.java:279)
     at io.realm.Realm.create(Realm.java:247)
     at io.realm.Realm.getInstance(Realm.java:219)
     at mohitkanwal.com.examplerealm.ContactHelper.getMeARealmFile(ContactHelper.java:53)
     at mohitkanwal.com.examplerealm.ContactHelper.getAllContacts(ContactHelper.java:64)
     at mohitkanwal.com.examplerealm.MainActivity.onCreate(MainActivity.java:25)
     at android.app.Activity.performCreate(Activity.java:6500)
     at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1120)
     at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:3072)
     at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:3218) 

Is the SharedGroupManager.compact() implementation compatible with the new format of the Realm File? Right now it seems to be unusable with any Realm created by v0.83 or v0.82.2

Regards Mohit

[dalinaum]

Hello @creativepsyco I can reproduce this probem too. We are investigating this.

[creativepsyco]

:+1: Sounds good.

[kneth]

@creativepsyco When you don't call Realm.compactRealm() everything is fine, right? We are currently investigating the underlying method in the storage engine. My recommendation is to disable compaction. I hope we will be able to release a patch to fix it.

[creativepsyco]

Yes everything is fine if I don't call Realm.compactRealm(). I suppose some format related issues with the core storage engine. Thanks for the prompt reply :+1:

[cmelchior]

0.83.1 was released today with a fix for this.

[creativepsyco]

:+1: Verified and it works.

[erichkleung]

Hi, I'm running into this problem with 0.86.0. Old Realm was dropped and new one was created on this version.

Fatal Exception: java.lang.RuntimeException: Unable to create application com.company.myapp.app: java.lang.IllegalArgumentException: Illegal Argument: Invalid format of Realm file.
       at android.app.ActivityThread.handleBindApplication(ActivityThread.java:5116)
       at android.app.ActivityThread.access$1600(ActivityThread.java:177)
       at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1509)
       at android.os.Handler.dispatchMessage(Handler.java:102)
       at android.os.Looper.loop(Looper.java:145)
       at android.app.ActivityThread.main(ActivityThread.java:5942)
       at java.lang.reflect.Method.invoke(Method.java)
       at java.lang.reflect.Method.invoke(Method.java:372)
       at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1399)
       at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1194)
Caused by java.lang.IllegalArgumentException: Illegal Argument: Invalid format of Realm file.
       at io.realm.internal.SharedGroup.nativeCreate(SharedGroup.java)
       at io.realm.internal.SharedGroup.<init>(SharedGroup.java:71)
       at io.realm.internal.SharedGroupManager.compact(SharedGroupManager.java:174)
       at io.realm.BaseRealm.compactRealm(BaseRealm.java:547)
       at io.realm.Realm.compactRealm(Realm.java:1095)
       at com.company.myapp.app.checkForMigration(app.java:122)
       at com.company.myapp.app.onCreate(app.java:91)
       at android.app.Instrumentation.callApplicationOnCreate(Instrumentation.java:1020)
       at android.app.ActivityThread.handleBindApplication(ActivityThread.java:5113)
       at android.app.ActivityThread.access$1600(ActivityThread.java:177)
       at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1509)
       at android.os.Handler.dispatchMessage(Handler.java:102)
       at android.os.Looper.loop(Looper.java:145)
       at android.app.ActivityThread.main(ActivityThread.java:5942)
       at java.lang.reflect.Method.invoke(Method.java)
       at java.lang.reflect.Method.invoke(Method.java:372)
       at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1399)
       at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1194)

[kneth]

The Invalid format of Realm file. message indicate that the Realm file is damaged in some way. Can you elaborate a bit on your setup?

[erichkleung]

We had a previous version of Realm installed, and decided it would be easier to drop the old Realm and create a new one.

This is what we do immediately prior to attempting to compact Realm.

        RealmConfiguration oldRealm = new RealmConfiguration.Builder(this)
                .name("default.realm")
                .build();

        boolean needToRebuildRealm = false;
        File f = new File(getFilesDir().getAbsolutePath(), "default.realm");
        if (f.exists()) {
            FSLog.log("old realm exists, will be rebuilding");
            needToRebuildRealm = true;
        }

        Realm.deleteRealm(oldRealm);

        RealmConfiguration.Builder builder = new RealmConfiguration.Builder(this)
                .name("cache")
                .migration(new MigrationToLatest())
                .schemaVersion(MigrationToLatest.LATEST_VERSION);

        RealmConfiguration realmConfiguration = builder.build();
        Realm.setDefaultConfiguration(realmConfiguration);

I'm not really sure how I can help elaborate on the setup, what exactly are you looking for?

[kneth]

The error happens when you call compactRealm(). How do you call that method?

[erichkleung]

Oh, immediately below the block above with this:

Realm.compactRealm(realmConfiguration);

[kneth]

@erichkleung I have tried to reproduce your case without luck. Please take a look at #1958 and tell me if my test differs from your code.

[LiuDeng]

@kneth i still get this problem in io.realm:realm-android:0.87.4 if i do not use compactRealm the realm file increase fast when will you fix this！

[cmelchior]

Hi @LiuDeng. This issue is unrelated to any file size increase. Please create a new issue with all relevant information instead.

[LiuDeng]

@cmelchior i mean if i do not use compactRealm the realm size increase fast

[dalinaum]

Hello, @LiuDeng. Could you create another issue for your problem? It would be helpful to handle your problem.

[puneetagarwal]

Hello, I am not using this method Realm.compactRealm() but still I am facing the same crash here.

Caused by: java.lang.IllegalArgumentException: Illegal Argument: Invalid format of Realm file. at io.realm.internal.SharedGroup.createNativeWithImplicitTransactions(Native Method) at io.realm.internal.SharedGroup.(Unknown Source) at io.realm.internal.SharedGroupManager.(Unknown Source) at io.realm.BaseRealm.(Unknown Source) at io.realm.Realm.(Unknown Source) at io.realm.Realm.createAndValidate(Unknown Source) at io.realm.Realm.create(Unknown Source) at io.realm.Realm.getDefaultInstance(Unknown Source)

I am using Realm 0.85.

Can you please help me with this. Thanks.

[kneth]

@puneetagarwal Many things can cause this exception. For example, if you have an encrypted Realm file, and open it with the wrong key. Can you share your configuration?

[puneetagarwal]

Hi Kneth, Here is the code for my realm configuration:-

`private void configureRealm() { byte[] key = null;

    if (Build.VERSION.SDK_INT < Build.VERSION_CODES.JELLY_BEAN_MR2) {
        SecurePreferences securePreferences = new SecurePreferences(this);

        if (!securePreferences.contains("DBAlias")) {
            // generate a key
            key = new byte[64];
            new SecureRandom().nextBytes(key);

            // store the key in the encrypted shared prefs
            securePreferences.edit().putString("DBAlias", Base64.encodeToString(key,  Base64.DEFAULT)).apply();
        } else {
            String keyString = securePreferences.getString("DBAlias", null);
            key = Base64.decode(keyString, Base64.DEFAULT);
        }
    } else {
        try {
            KeyStore keyStore = KeyStore.getInstance("AndroidKeyStore");
            keyStore.load(null);

            List<String> keyAliases = new ArrayList<>();
            Enumeration<String> aliases = keyStore.aliases();
            while (aliases.hasMoreElements()) {
                keyAliases.add(aliases.nextElement());
            }

            SharedPreferences sharedPreferences = getSharedPreferences("DBPrefs", Context.MODE_PRIVATE);

            if (!keyStore.containsAlias("DBAlias")) {
                // create a keystore alias
                Calendar start = Calendar.getInstance();
                Calendar end = Calendar.getInstance();
                end.add(Calendar.YEAR, 1);
                KeyPairGeneratorSpec spec = new KeyPairGeneratorSpec.Builder(this)
                        .setAlias("DBAlias")
                        .setSubject(new X500Principal("CN=Sample Name, O=Android Authority"))
                        .setSerialNumber(BigInteger.ONE)
                        .setStartDate(start.getTime())
                        .setEndDate(end.getTime())
                        .build();
                KeyPairGenerator generator = KeyPairGenerator.getInstance("RSA", "AndroidKeyStore");
                generator.initialize(spec);

                KeyPair keyPair = generator.generateKeyPair();
                RSAPublicKey publicKey = (RSAPublicKey) keyPair.getPublic();

                // generate a key
                key = new byte[64];
                new SecureRandom().nextBytes(key);

                // encrypt the new key
                Cipher input = Cipher.getInstance("RSA/ECB/PKCS1Padding", "AndroidOpenSSL");
                input.init(Cipher.ENCRYPT_MODE, publicKey);

                ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
                CipherOutputStream cipherOutputStream = new CipherOutputStream(outputStream, input);
                cipherOutputStream.write(key);
                cipherOutputStream.close();

                // store the encrypted key in the shared prefs
                sharedPreferences.edit().putString("DBAlias", Base64.encodeToString(outputStream.toByteArray(), Base64.DEFAULT)).apply();
            } else {
                String encryptedString = sharedPreferences.getString("DBAlias", null);

                if (encryptedString != null) {
                    KeyStore.PrivateKeyEntry privateKeyEntry = (KeyStore.PrivateKeyEntry) keyStore.getEntry("DBAlias", null);

                    Cipher output = Cipher.getInstance("RSA/ECB/PKCS1Padding");
                    output.init(Cipher.DECRYPT_MODE, privateKeyEntry.getPrivateKey());

                    CipherInputStream cipherInputStream = new CipherInputStream(new ByteArrayInputStream(Base64.decode(encryptedString, Base64.DEFAULT)), output);
                    List<Byte> values = new ArrayList<>();
                    int nextByte;
                    while ((nextByte = cipherInputStream.read()) != -1) {
                        values.add((byte) nextByte);
                    }

                    key = new byte[values.size()];
                    for (int i = 0; i < key.length; i++) {
                        key[i] = values.get(i);
                    }
                }
            }

        } catch (NoSuchAlgorithmException |
                KeyStoreException |
                CertificateException |
                IOException |
                NoSuchProviderException |
                InvalidAlgorithmParameterException |
                InvalidKeyException |
                UnrecoverableEntryException |
                NoSuchPaddingException e) {
            e.printStackTrace();
        }
    }

    RealmConfiguration config = null;
    if (key != null) {
        config = new RealmConfiguration.Builder(getApplicationContext())
                .encryptionKey(key)
                .schemaVersion(AppRealmMigration.REALM_VERSION) // Must be bumped when the schema changes
                .deleteRealmIfMigrationNeeded()
                .migration(new AppRealmMigration()) // Migration to run
                .build();
    } else {
        config = new RealmConfiguration.Builder(getApplicationContext())
                .schemaVersion(AppRealmMigration.REALM_VERSION) // Must be bumped when the schema changes
                .deleteRealmIfMigrationNeeded()
                .migration(new AppRealmMigration()) // Migration to run
                .build();
    }

    Realm.setDefaultConfiguration(config);
}`

I just added deleteRealmIfMigrationNeeded () method to resolve this issue in case if there is any issue with the DB migration. Does it helped you to understand if I am doing something wrong. Thanks.

[kneth]

deleteRealmIfMigrationIsNeeded will delete your Realm file, and all (Realm) data will be wiped.

If you can reproduce it on your local devices, it might be useful to log the key to see if you get the exact key every time.

[puneetagarwal]

I tried to reproduce this issue at our end and it's not crashing but it crashes in the play store version only on some devices.

Is this possible that if I use deleteRealmIfMigrationIsNeeded() and migration() methods at the same time while creating Realm configuration, it throws this exception "IllegalArgumentException: Illegal Argument: Invalid format of Realm file."

Also, if I add name() method to create a realm file, will it create a new Realm file with that name or it just renames the previous one. As I am not using name() method right now and Realm creates a file with a default name.

Hope to hear from you soon. Thanks.