Unable to restore encrypted backup - Bad MAC

mike-edel commented 5 years ago

[x] I have searched open and closed issues for duplicates
[x] I am submitting a bug report for existing functionality that does not work as intended
[x] I have read https://github.com/signalapp/Signal-Android/wiki/Submitting-useful-bug-reports
[x] This isn't a feature request or a discussion topic

Bug description

Unable to restore signal backup on new phone. Similar to #7637 but I don't have xposed installed. Signal stable closing without error after reading 11401 messages. Signal beta giving error message about incorrect passphrase after reading 11401 messages. I am certain the passphrase and the backup are correct as they are working fine using https://github.com/xeals/signal-back

Steps to reproduce

copy signal backup to new phone
open signal
enter passphrase
wait for messages to import

Actual result:

Signal just closing (stable)
Error message stating the passphrase is incorrect (beta).

Expected result: Backup is successfully restored.

Screenshots

beta error (in German due to phone language) _20181110_121218

Device info

Device: Nokia 7 Plus Dual-Sim Android version: 9 Kernel version: 4.4.146-perf+ Signal version: Beta: 4.30.2 also tested on stable: 4.29.7

Link to debug log

Working on grabbing a logcat log but struggling with it. Will add through edit. Logcat for version 4.29.7:

--------- beginning of crash
11-08 22:32:10.330  4472  6987 E AndroidRuntime: FATAL EXCEPTION: AsyncTask #3
11-08 22:32:10.330  4472  6987 E AndroidRuntime: Process: org.thoughtcrime.securesms, PID: 4472
11-08 22:32:10.330  4472  6987 E AndroidRuntime: java.lang.RuntimeException: An error occurred while executing doInBackground()
11-08 22:32:10.330  4472  6987 E AndroidRuntime:    at android.os.AsyncTask$3.done(AsyncTask.java:354)
11-08 22:32:10.330  4472  6987 E AndroidRuntime:    at java.util.concurrent.FutureTask.finishCompletion(FutureTask.java:383)
11-08 22:32:10.330  4472  6987 E AndroidRuntime:    at java.util.concurrent.FutureTask.setException(FutureTask.java:252)
11-08 22:32:10.330  4472  6987 E AndroidRuntime:    at java.util.concurrent.FutureTask.run(FutureTask.java:271)
11-08 22:32:10.330  4472  6987 E AndroidRuntime:    at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:245)
11-08 22:32:10.330  4472  6987 E AndroidRuntime:    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
11-08 22:32:10.330  4472  6987 E AndroidRuntime:    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
11-08 22:32:10.330  4472  6987 E AndroidRuntime:    at java.lang.Thread.run(Thread.java:764)
11-08 22:32:10.330  4472  6987 E AndroidRuntime: Caused by: java.lang.NegativeArraySizeException: -1884239691
11-08 22:32:10.330  4472  6987 E AndroidRuntime:    at org.thoughtcrime.securesms.backup.FullBackupImporter$BackupRecordInputStream.readFrame(FullBackupImporter.java:310)
11-08 22:32:10.330  4472  6987 E AndroidRuntime:    at org.thoughtcrime.securesms.backup.FullBackupImporter$BackupRecordInputStream.readFrame(FullBackupImporter.java:252)
11-08 22:32:10.330  4472  6987 E AndroidRuntime:    at org.thoughtcrime.securesms.backup.FullBackupImporter.importFile(FullBackupImporter.java:81)
11-08 22:32:10.330  4472  6987 E AndroidRuntime:    at org.thoughtcrime.securesms.RegistrationActivity$2.doInBackground(RegistrationActivity.java:394)
11-08 22:32:10.330  4472  6987 E AndroidRuntime:    at org.thoughtcrime.securesms.RegistrationActivity$2.doInBackground(RegistrationActivity.java:386)
11-08 22:32:10.330  4472  6987 E AndroidRuntime:    at android.os.AsyncTask$2.call(AsyncTask.java:333)
11-08 22:32:10.330  4472  6987 E AndroidRuntime:    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
11-08 22:32:10.330  4472  6987 E AndroidRuntime:    ... 4 more

logcat for version 4.30.2:

11-10 13:15:20.687 27588 31822 W RegistrationActivity: null
11-10 13:15:20.687 27588 31822 W RegistrationActivity: java.io.IOException: Bad MAC
11-10 13:15:20.687 27588 31822 W RegistrationActivity:  at org.thoughtcrime.securesms.backup.FullBackupImporter$BackupRecordInputStream.readAttachmentTo(FullBackupImporter.java:298)
11-10 13:15:20.687 27588 31822 W RegistrationActivity:  at org.thoughtcrime.securesms.backup.FullBackupImporter.processAttachment(FullBackupImporter.java:140)
11-10 13:15:20.687 27588 31822 W RegistrationActivity:  at org.thoughtcrime.securesms.backup.FullBackupImporter.importFile(FullBackupImporter.java:87)
11-10 13:15:20.687 27588 31822 W RegistrationActivity:  at org.thoughtcrime.securesms.RegistrationActivity$2.doInBackground(RegistrationActivity.java:396)
11-10 13:15:20.687 27588 31822 W RegistrationActivity:  at org.thoughtcrime.securesms.RegistrationActivity$2.doInBackground(RegistrationActivity.java:388)
11-10 13:15:20.687 27588 31822 W RegistrationActivity:  at android.os.AsyncTask$2.call(AsyncTask.java:333)
11-10 13:15:20.687 27588 31822 W RegistrationActivity:  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
11-10 13:15:20.687 27588 31822 W RegistrationActivity:  at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:245)
11-10 13:15:20.687 27588 31822 W RegistrationActivity:  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
11-10 13:15:20.687 27588 31822 W RegistrationActivity:  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
11-10 13:15:20.687 27588 31822 W RegistrationActivity:  at java.lang.Thread.run(Thread.java:764)

bepaald commented 5 years ago

@mk121212 I think this was 99.9% certainly a case of wrong password, any chance of that? Otherwise, if corruption did occur in the very first bytes of the backup, your data could never have been saved anyway.

mk121212 commented 5 years ago

@bepaald That's what I thought several times. However, I am sure that it was the correct password. I did not only write the password down but also took a picture of Signal showing it on the old phone. So I was able to check carefully.

With the old password I made backups several times a day for a couple of days. Sometimes I deleted the former backup on the old phone, sometimes I didn't. This way I was sure that the new file was written a new memory address and (to some extend) that the corruptions does not come from the hardware. The result was always like described in my post above. After disabling and re-enabling backups, the first backup with the new password worked perfectly fine (on the new phone and with the tools).

Still, I don't know whether this helps with the bad MAC problematic described in this issue but maybe it does point to other code parts of Signals backup functionality that could help solving this issue.

SelfSealShelf commented 5 years ago

Is this safe to do? It seems like bad practice to decrypt your backup outside of signal. If the backup contains key material that would expose it.

bepaald commented 5 years ago

@AbstractThought I'm not sure what you mean. In normal operation, all (unencrypted) data only lives in memory, but I suppose you could imagine some malware exists that gets to it, but I'd say it's far more likely that type of malware exists for android and is looking for the signal app, than for PC looking for my program. Personally, I actually trust my PC more than I do my phone.

@santa-klaus I have some initial backup merging support. It works on the small files I tried, but definitely needs testing. Are you interested in trying? The method is a little involved as it actually turned out to be quite a difficult procedure.

Example, click to open if you dare... ;)

Assuming a current backup `current.backup` and an old one `source.backup`. First you need to get a list of threads from the old backup: ``` [~/programming/signalbackup-tools] $ ./signalbackup-tools --listthreads source.backup 871668681636341580140408145422 IV: (hex:) e3 1c c6 5c 46 e3 b6 62 ff a7 95 40 e6 99 f9 eb (size: 16) SALT: (hex:) 35 6a 55 bc 80 e9 da dd 2c ed 28 e2 da 22 f4 b8 3b e3 5d 44 2b 94 18 3e d0 6a 6f e3 79 d9 5d 1f (size: 32) BACKUPKEY: (hex:) 3e 91 7a 80 66 9f 6b 1f a9 77 b1 3e fb 10 ae 15 b2 b4 69 7f 99 ad 40 16 57 61 00 ce cc 8e d0 cc (size: 32) CIPHERKEY: (hex:) 69 67 67 1b d7 51 41 84 33 88 6b 1e 3b 21 e9 96 6e f7 f0 0d 61 ec 64 6e 44 29 97 3e df 49 83 26 (size: 32) MACKEY: (hex:) 0b 30 c9 d3 56 6b 8c 81 f3 f5 bb cf 1d b4 ed 57 66 dc c8 72 11 58 04 42 07 db b8 a1 61 62 2e c3 (size: 32) COUNTER: 3810313820 Reading backup file... FRAME 93 (100.0%)... Read entire backup file... done! ---------------------------------------------------------------------------------------------------------- | _id | recipient_ids | snippet | COALESCE(recipient_prefer[...] | ---------------------------------------------------------------------------------------------------------- | 1 | +161765XXXXX | <#> Your Signal verificat[...] | (NULL) | | 2 | __textsecure_group__!a2c3[...] | Ok | devgroup | | 3 | +316474XXXXX | Last msg | Master Phone | | 4 | +316836XXXXX | Ok | Devphone Red | ---------------------------------------------------------------------------------------------------------- ``` Then, you can import a selection of threads from the source file into your current backup and export to a new backup file. Expect tons of output (I really need to clean that up sometime): ``` [~/programming/signalbackup-tools] $ ./signalbackup-tools --importthreads 2,3,4 --source source.backup --sourcepassword 871668681636341580140408145422 --output merged.backup --opassword 000000000000000000000000000000 current.backup 420676745407910020904427069666 IV: (hex:) e2 dd c7 b0 d7 c1 81 01 6b db f8 24 47 98 5c 35 (size: 16) SALT: (hex:) 47 a8 83 be 1f 9f d7 a4 db 6c 82 bd c4 d2 e9 4b 5e 90 d7 fd a4 98 81 4a 62 f1 0e d6 e5 52 f7 ee (size: 32) BACKUPKEY: (hex:) b1 59 c2 ec ce cf dc de 37 6f bd af 15 79 06 c7 30 c4 56 3f 5f 60 f8 74 67 34 90 7b a5 c5 44 2b (size: 32) CIPHERKEY: (hex:) 69 12 52 65 c2 5d 96 e0 26 dc 46 6c 95 92 18 f0 e7 69 31 7d 07 f7 ce 7e 4d 74 76 10 d1 78 da d8 (size: 32) MACKEY: (hex:) 37 4c bc 18 c2 93 47 60 67 63 0d 81 24 65 9e ab 55 a3 c6 17 fb 95 26 2e 4a 68 e8 aa 5c a5 0b 7e (size: 32) COUNTER: 3806185392 Reading backup file... FRAME 88 (100.0%)... Read entire backup file... done! Importing thread 2 from source file: source.backup IV: (hex:) e3 1c c6 5c 46 e3 b6 62 ff a7 95 40 e6 99 f9 eb (size: 16) SALT: (hex:) 35 6a 55 bc 80 e9 da dd 2c ed 28 e2 da 22 f4 b8 3b e3 5d 44 2b 94 18 3e d0 6a 6f e3 79 d9 5d 1f (size: 32) BACKUPKEY: (hex:) 3e 91 7a 80 66 9f 6b 1f a9 77 b1 3e fb 10 ae 15 b2 b4 69 7f 99 ad 40 16 57 61 00 ce cc 8e d0 cc (size: 32) CIPHERKEY: (hex:) 69 67 67 1b d7 51 41 84 33 88 6b 1e 3b 21 e9 96 6e f7 f0 0d 61 ec 64 6e 44 29 97 3e df 49 83 26 (size: 32) MACKEY: (hex:) 0b 30 c9 d3 56 6b 8c 81 f3 f5 bb cf 1d b4 ed 57 66 dc c8 72 11 58 04 42 07 db b8 a1 61 62 2e c3 (size: 32) COUNTER: 3810313820 Reading backup file... FRAME 93 (100.0%)... Read entire backup file... done! Deleting messages not belonging to requested thread(s) from 'sms' Deleting messages not belonging to requested thread(s) from 'mms' Deleting attachment entries from 'part' not belonging to remaining mms entries Deleting other threads from 'thread'... Dealing with thread id: 2 Updating msgcount Setting last msg date Updating snippet Updating snippet type Deleting removed groups... Delete others from 'identities' Deleting group receipts entries from deleted messages... Deleting drafts from deleted threads... Adjusting indexes in tables... Compacting table: sms Compacting table: mms Compacting table: part Compacting table: recipient_preferences Compacting table: groups Compacting table: identities Compacting table: group_receipts Compacting table: drafts Found existing thread for this recipient in target database, merging into thread 5 Importing statements from source table 'sms'...4 entries... Importing statements from source table 'mms'...3 entries... Importing statements from source table 'part'...0 entries... Importing statements from source table 'drafts'...0 entries... Importing statements from source table 'push'...0 entries... Importing statements from source table 'group_receipts'...6 entries... Importing statements from source table 'sticker'...0 entries... Importing statements from source table 'job_spec'...0 entries... Importing statements from source table 'constraint_spec'...0 entries... Importing statements from source table 'dependency_spec'...0 entries... Importing thread 3 from source file: source.backup IV: (hex:) e3 1c c6 5c 46 e3 b6 62 ff a7 95 40 e6 99 f9 eb (size: 16) SALT: (hex:) 35 6a 55 bc 80 e9 da dd 2c ed 28 e2 da 22 f4 b8 3b e3 5d 44 2b 94 18 3e d0 6a 6f e3 79 d9 5d 1f (size: 32) BACKUPKEY: (hex:) 3e 91 7a 80 66 9f 6b 1f a9 77 b1 3e fb 10 ae 15 b2 b4 69 7f 99 ad 40 16 57 61 00 ce cc 8e d0 cc (size: 32) CIPHERKEY: (hex:) 69 67 67 1b d7 51 41 84 33 88 6b 1e 3b 21 e9 96 6e f7 f0 0d 61 ec 64 6e 44 29 97 3e df 49 83 26 (size: 32) MACKEY: (hex:) 0b 30 c9 d3 56 6b 8c 81 f3 f5 bb cf 1d b4 ed 57 66 dc c8 72 11 58 04 42 07 db b8 a1 61 62 2e c3 (size: 32) COUNTER: 3810313820 Reading backup file... FRAME 93 (100.0%)... Read entire backup file... done! Deleting messages not belonging to requested thread(s) from 'sms' Deleting messages not belonging to requested thread(s) from 'mms' Deleting attachment entries from 'part' not belonging to remaining mms entries Deleting other threads from 'thread'... Dealing with thread id: 3 Updating msgcount Setting last msg date Updating snippet Updating snippet type Deleting removed groups... Delete others from 'identities' Deleting group receipts entries from deleted messages... Deleting drafts from deleted threads... Adjusting indexes in tables... Compacting table: sms Compacting table: mms Compacting table: part Compacting table: recipient_preferences Compacting table: groups Compacting table: identities Compacting table: group_receipts Compacting table: drafts Found existing thread for this recipient in target database, merging into thread 6 Importing statements from source table 'sms'...5 entries... Importing statements from source table 'mms'...1 entries... Importing statements from source table 'part'...1 entries... Importing statements from source table 'drafts'...0 entries... Importing statements from source table 'push'...0 entries... Importing statements from source table 'group_receipts'...0 entries... Importing statements from source table 'sticker'...0 entries... Importing statements from source table 'job_spec'...0 entries... Importing statements from source table 'constraint_spec'...0 entries... Importing statements from source table 'dependency_spec'...0 entries... Importing thread 4 from source file: source.backup IV: (hex:) e3 1c c6 5c 46 e3 b6 62 ff a7 95 40 e6 99 f9 eb (size: 16) SALT: (hex:) 35 6a 55 bc 80 e9 da dd 2c ed 28 e2 da 22 f4 b8 3b e3 5d 44 2b 94 18 3e d0 6a 6f e3 79 d9 5d 1f (size: 32) BACKUPKEY: (hex:) 3e 91 7a 80 66 9f 6b 1f a9 77 b1 3e fb 10 ae 15 b2 b4 69 7f 99 ad 40 16 57 61 00 ce cc 8e d0 cc (size: 32) CIPHERKEY: (hex:) 69 67 67 1b d7 51 41 84 33 88 6b 1e 3b 21 e9 96 6e f7 f0 0d 61 ec 64 6e 44 29 97 3e df 49 83 26 (size: 32) MACKEY: (hex:) 0b 30 c9 d3 56 6b 8c 81 f3 f5 bb cf 1d b4 ed 57 66 dc c8 72 11 58 04 42 07 db b8 a1 61 62 2e c3 (size: 32) COUNTER: 3810313820 Reading backup file... FRAME 93 (100.0%)... Read entire backup file... done! Deleting messages not belonging to requested thread(s) from 'sms' Deleting messages not belonging to requested thread(s) from 'mms' Deleting attachment entries from 'part' not belonging to remaining mms entries Deleting other threads from 'thread'... Dealing with thread id: 4 Updating msgcount Setting last msg date Updating snippet Updating snippet type Deleting removed groups... Delete others from 'identities' Deleting group receipts entries from deleted messages... Deleting drafts from deleted threads... Adjusting indexes in tables... Compacting table: sms Compacting table: mms Compacting table: part Compacting table: recipient_preferences Compacting table: groups Compacting table: identities Compacting table: group_receipts Compacting table: drafts Importing statements from source table 'sms'...4 entries... Importing statements from source table 'mms'...0 entries... Importing statements from source table 'part'...0 entries... Importing statements from source table 'thread'...1 entries... Importing statements from source table 'identities'...0 entries... Importing statements from source table 'drafts'...0 entries... Importing statements from source table 'push'...0 entries... Importing statements from source table 'groups'...0 entries... Importing statements from source table 'recipient_preferences'...0 entries... Importing statements from source table 'group_receipts'...0 entries... Importing statements from source table 'sticker'...0 entries... Importing statements from source table 'job_spec'...0 entries... Importing statements from source table 'constraint_spec'...0 entries... Importing statements from source table 'dependency_spec'...0 entries... Exporting backup to 'merged.backup' Writing HeaderFrame... Writing DatabaseVersionFrame... Writing SqlStatementFrame(s)... Dealing with table 'sms'... 25/25 entries...done Dealing with table 'mms'... 7/7 entries...done Dealing with table 'part'... 2/2 entries...done Dealing with table 'thread'... 7/7 entries...done Dealing with table 'identities'... 3/3 entries...done Dealing with table 'drafts'... 0/0 entries... Dealing with table 'push'... 0/0 entries... Dealing with table 'groups'... 1/1 entries...done Dealing with table 'recipient_preferences'... 4/4 entries...done Dealing with table 'group_receipts'... 8/8 entries...done Dealing with table 'sticker'... 0/0 entries... Dealing with table 'job_spec'... 1/1 entries...done Dealing with table 'constraint_spec'... 0/0 entries... Dealing with table 'dependency_spec'... 0/0 entries... Writing SharedPrefFrame(s)... Writing EndFrame... Done! ``` The program automatically tries to determine into which thread of the current db the old messages should be inserted. This might fail if contact have changed numbers or if one of the backups has a contact with a country code (+316.....) and the other omits it (06....).

oneof3holes commented 5 years ago

@bepaald been fighting with coinbase to release my dust without me sending in a pic of my ID. Finally happened, didn't want you to think I forgot about you. Thank you again.

bepaald commented 5 years ago

@oneof3holes O wow, thanks! I thought you had just decided against it (which would have been fine of course), but this is cool. I really appreciate it. Thanks again!

santa-klaus commented 5 years ago

@bepaald I am sorry, I got drowned in work. I might have some more time to try things in October and would like to do it. Also, my phone broke, so I will probably have a good reason to merge backups soon. I'll let you know when I get around to do some testing.

bepaald commented 4 years ago

@santa-klaus Absolutely not a problem of course. However, just a heads up: I was looking at the source of the 4.48 beta that is currently out and it will change the internal sql database in such a way that it will certainly break my current merging code.

I do plan to rewrite the code to be able to merge backups, but there is no telling how long it will take me. Also, it might take even longer (if ever) to get to a point where I can merge an old style db into a new one which would probably also be a use case after your current install updates to 4.48.

Obviously, you don't have to do it if you don't want to or don't have time, but I just wanted to warn you that the window of opportunity might be closing.

santa-klaus commented 4 years ago

Oh, that is really good to know, thank you for the heads up! To be honest, for me it doesn't sound like rewriting stuff to be able to merge old and new backups is worth the effort, since this will have some, but very limited use cases...

I'll definitely write if I manage to look into this!

Am 25.09.2019 20:39 schrieb bepaald:

@santa-klaus [1] Absolutely not a problem of course. However, just a heads up: I was looking at the source of the 4.48 beta that is currently out and it will change the internal sql database in such a way that it will certainly break my current merging code.

I do plan to rewrite the code to be able to merge backups, but there is no telling how long it will take me. Also, it might take even longer (if ever) to get to a point where I can merge an old style db into a new one which would probably also be a use case after your current install updates to 4.48.

Obviously, you don't have to do it if you don't want to or don't have time, but I just wanted to warn you that the window of opportunity might be closing.

-- You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub [2], or mute the thread [3].

Links:

[1] https://github.com/santa-klaus [2] https://github.com/signalapp/Signal-Android/issues/8355?email_source=notifications&email_token=ACTFAB5LXJRUNB2334ODHLLQLOV4TA5CNFSM4GC73FNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7S5I7Q#issuecomment-535155838 [3] https://github.com/notifications/unsubscribe-auth/ACTFAB3ATVCOA66YK2JLMNTQLOV4TANCNFSM4GC73FNA

alpha1 commented 4 years ago

Same issue, android 10. Adding my info and a screen cast in case it helps any.

Actual result: I have followed Signal's instructions on backing up and restoring a backup. I am moving from a OnePlus 5t to a OnePlus 7t, both using Signal beta, latest. When I type in the wrong backup code, it fails to restore the backup, which is expected, but when I type in the right code, it proceeds to loop through all messages and then throw a toast error saying "Incorrect backup passphrase", and does not import the backup.

Expected result: Backup would restore messages