backup and restore issue (?)

[lemenkov]

I've tried to backup data in an old phone (I guess Android pre-12 or something) - got a file named "externalBackup" (w/o quotes) which according to a file utility is a "Java serialization data, version 5". It looks valid - i've tried strings utility and can see a lot of strings inside.

Now I'm trying to restore on a new Android 13 (if it matters). It asks for the password which I provide and then does nothing. I've tried wrong password to see if it changes anything - it does complains about the wrong password.

Is it possible to restore data from "Java serialization data, version 5" on a modern Android and using FreeOTP ver. 2.0.2?

[lemenkov]

A little bit more details. I've tried to deserialize externalBackup file with the @NickstaDB's excellent SerializationDumper app. Theoretically I could convert it to a JSON and then export it to freeotpplus or aegis application. Unfortunately looks like the data is stored in an encrypted format. If only it is possible to export it to something more explicit (plain-text data in JSON-form) things could be more easy.

Speaking about exporting. I can export from the app - it creates an identical externalBackup file so maybe the problem is that it imports fine but cannot display OTP entries for unknown reason.

[justin-stephenson]

Is it possible to restore data from "Java serialization data, version 5" on a modern Android and using FreeOTP ver. 2.0.2?

Yes that is the intention, but it sounds like an unexpected issue during the restore operation which decrypts and restores the backup data. Could you check Logcat logs on both devices to see if there are any errors (especially on the restore-to device) when attempting the backup and restore? There is some logging https://github.com/freeotp/freeotp-android/blob/master/mobile/src/main/java/org/fedorahosted/freeotp/TokenPersistence.java#L146 FreeOTP does which might help to analyze logging also.

I tried to reproduce a similar issue but was unable to, if you have any more data you can share that would be great.

[lemenkov]

@justin-stephenson I've just managed to restore almost all tokens. Turned out it does throws exceptions during the restore process. I've wrapped a few lines of code like that:

-            SecretKey skKey = ekKey.decrypt(sk);
-
-            // Deserialize token
-            Token token = Token.deserialize(tokenData);
-
-            bkp.key = skKey;
-            bkp.token = token;
-            bkp.uuid = uuid;
-            Log.i(LOGTAG, String.format("Added [%s] token to backup list", uuid));
-            tokensList.add(bkp);
+            try {
+                SecretKey skKey = ekKey.decrypt(sk);
+
+                // Deserialize token
+                Token token = Token.deserialize(tokenData);
+
+                bkp.key = skKey;
+                bkp.token = token;
+                bkp.uuid = uuid;
+                Log.i(LOGTAG, String.format("Added [%s] token [%s] to backup list", uuid, encodedKey));
+                tokensList.add(bkp);
+            } catch (javax.crypto.AEADBadTagException e) {
+                Log.e(LOGTAG, "Exception", e);
+            }

It throws the following exception every time it tries to restore a token containing "counter" parameter:

2023-08-30 17:22:04.112 20013-20013 TokenPersistence        org.fedorahosted.freeotp             E  Exception
                                                                                                    javax.crypto.AEADBadTagException: error:1e000065:Cipher functions:OPENSSL_internal:BAD_DECRYPT
                                                                                                        at java.lang.reflect.Constructor.newInstance0(Native Method)
                                                                                                        at java.lang.reflect.Constructor.newInstance(Constructor.java:343)
                                                                                                        at com.android.org.conscrypt.OpenSSLAeadCipher.throwAEADBadTagExceptionIfAvailable(OpenSSLAeadCipher.java:320)
                                                                                                        at com.android.org.conscrypt.OpenSSLAeadCipher.doFinalInternal(OpenSSLAeadCipher.java:371)
                                                                                                        at com.android.org.conscrypt.OpenSSLCipher.engineDoFinal(OpenSSLCipher.java:374)
                                                                                                        at javax.crypto.Cipher.doFinal(Cipher.java:2056)
                                                                                                        at org.fedorahosted.freeotp.encryptor.EncryptedKey.decrypt(EncryptedKey.java:59)
                                                                                                        at org.fedorahosted.freeotp.TokenPersistence.restore(TokenPersistence.java:205)
                                                                                                        at org.fedorahosted.freeotp.main.Adapter.restoreTokens(Adapter.java:265)
                                                                                                        at org.fedorahosted.freeotp.main.Activity$5.onClick(Activity.java:413)
                                                                                                        at androidx.appcompat.app.AlertController$ButtonHandler.handleMessage(AlertController.java:167)
                                                                                                        at android.os.Handler.dispatchMessage(Handler.java:106)
                                                                                                        at android.os.Looper.loopOnce(Looper.java:205)
                                                                                                        at android.os.Looper.loop(Looper.java:294)
                                                                                                        at android.app.ActivityThread.main(ActivityThread.java:8177)
                                                                                                        at java.lang.reflect.Method.invoke(Native Method)
                                                                                                        at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:552)
                                                                                                        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:971)

Not sure how it is related but only those three tokens which have "counter" parameter cannot be restored. Namely - from @github, @binance and a corporate one from @redhat.

The first two are TOTP so technically they shouldn't contain this parameter at all. The latter is a legit HOTP one so it has to have it.

[justin-stephenson]

@justin-stephenson I've just managed to restore almost all tokens.

Initially in your comment no tokens were restored, is that right or do you see some consistency in results now?

Turned out it does throws exceptions during the restore process. I've wrapped a few lines of code like that:

-            SecretKey skKey = ekKey.decrypt(sk);
-
-            // Deserialize token
-            Token token = Token.deserialize(tokenData);
-
-            bkp.key = skKey;
-            bkp.token = token;
-            bkp.uuid = uuid;
-            Log.i(LOGTAG, String.format("Added [%s] token to backup list", uuid));
-            tokensList.add(bkp);
+            try {
+                SecretKey skKey = ekKey.decrypt(sk);
+
+                // Deserialize token
+                Token token = Token.deserialize(tokenData);
+
+                bkp.key = skKey;
+                bkp.token = token;
+                bkp.uuid = uuid;
+                Log.i(LOGTAG, String.format("Added [%s] token [%s] to backup list", uuid, encodedKey));
+                tokensList.add(bkp);
+            } catch (javax.crypto.AEADBadTagException e) {
+                Log.e(LOGTAG, "Exception", e);
+            }

It throws the following exception every time it tries to restore a token containing "counter" parameter:

2023-08-30 17:22:04.112 20013-20013 TokenPersistence        org.fedorahosted.freeotp             E  Exception
                                                                                                    javax.crypto.AEADBadTagException: error:1e000065:Cipher functions:OPENSSL_internal:BAD_DECRYPT
                                                                                                      at java.lang.reflect.Constructor.newInstance0(Native Method)
                                                                                                      at java.lang.reflect.Constructor.newInstance(Constructor.java:343)
                                                                                                      at com.android.org.conscrypt.OpenSSLAeadCipher.throwAEADBadTagExceptionIfAvailable(OpenSSLAeadCipher.java:320)
                                                                                                      at com.android.org.conscrypt.OpenSSLAeadCipher.doFinalInternal(OpenSSLAeadCipher.java:371)
                                                                                                      at com.android.org.conscrypt.OpenSSLCipher.engineDoFinal(OpenSSLCipher.java:374)
                                                                                                      at javax.crypto.Cipher.doFinal(Cipher.java:2056)
                                                                                                      at org.fedorahosted.freeotp.encryptor.EncryptedKey.decrypt(EncryptedKey.java:59)
                                                                                                      at org.fedorahosted.freeotp.TokenPersistence.restore(TokenPersistence.java:205)
                                                                                                      at org.fedorahosted.freeotp.main.Adapter.restoreTokens(Adapter.java:265)
                                                                                                      at org.fedorahosted.freeotp.main.Activity$5.onClick(Activity.java:413)
                                                                                                      at androidx.appcompat.app.AlertController$ButtonHandler.handleMessage(AlertController.java:167)
                                                                                                      at android.os.Handler.dispatchMessage(Handler.java:106)
                                                                                                      at android.os.Looper.loopOnce(Looper.java:205)
                                                                                                      at android.os.Looper.loop(Looper.java:294)
                                                                                                      at android.app.ActivityThread.main(ActivityThread.java:8177)
                                                                                                      at java.lang.reflect.Method.invoke(Native Method)
                                                                                                      at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:552)
                                                                                                      at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:971)

This traceback is similarly seen in https://github.com/freeotp/freeotp-android/issues/283

Not sure how it is related but only those three tokens which have "counter" parameter cannot be restored. Namely - from @github, @binance and a corporate one from @redhat.

I'm not sure what is the relationship with the failure and tokens with the counter parameter, I am often testing with HOTP tokens from https://freeotp.github.io/qrcode.html containing counter=0 as an OTP auth parameter.

The first two are TOTP so technically they shouldn't contain this parameter at all. The latter is a legit HOTP one so it has to have it.

I suspect the issue is more related to an unexpected issue with backing up from an old Android pre-12 device and restoring to a newer device. I wonder if comparing the exported backup file from an old and new device (with the same set of tokens) would show something relevant.

Assuming you can recover any lost FreeOTP tokens on your old device, can you try creating a backup, deleting these tokens, and restoring them on this same device to see if it fails?

[lemenkov]

Unfortunately I cannot use my old device anymore. All I've got is the backup file.

I've tried to restore it in a previuous Android versions, namely 8.0 and 8.1 which my lost phone was using and stumbled upon this stacktrace:

                                                                                                    javax.crypto.AEADBadTagException: mac check in GCM failed
                                                                                                        at java.lang.reflect.Constructor.newInstance0(Native Method)
                                                                                                        at java.lang.reflect.Constructor.newInstance(Constructor.java:334)
                                                                                                        at com.android.org.bouncycastle.jcajce.provider.symmetric.util.BaseBlockCipher$AEADGenericBlockCipher.doFinal(BaseBlockCipher.java:1406)
                                                                                                        at com.android.org.bouncycastle.jcajce.provider.symmetric.util.BaseBlockCipher.engineDoFinal(BaseBlockCipher.java:1139)
                                                                                                        at javax.crypto.Cipher.doFinal(Cipher.java:1741)
                                                                                                        at org.fedorahosted.freeotp.encryptor.EncryptedKey.decrypt(EncryptedKey.java:64)
                                                                                                        at org.fedorahosted.freeotp.TokenPersistence.restore(TokenPersistence.java:205)
                                                                                                        at org.fedorahosted.freeotp.main.Adapter.restoreTokens(Adapter.java:265)
                                                                                                        at org.fedorahosted.freeotp.main.Activity$5.onClick(Activity.java:413)
                                                                                                        at androidx.appcompat.app.AlertController$ButtonHandler.handleMessage(AlertController.java:167)
                                                                                                        at android.os.Handler.dispatchMessage(Handler.java:106)
                                                                                                        at android.os.Looper.loop(Looper.java:164)
                                                                                                        at android.app.ActivityThread.main(ActivityThread.java:6494)
                                                                                                        at java.lang.reflect.Method.invoke(Native Method)
                                                                                                        at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:438)
                                                                                                        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:807)

[lemenkov]

Ok for having everything in one place I am closing this as a duplicate of #283