Unable to restore encrypted backup - Bad MAC

[mike-edel]

• [x] I have searched open and closed issues for duplicates
• [x] I am submitting a bug report for existing functionality that does not work as intended
• [x] I have read https://github.com/signalapp/Signal-Android/wiki/Submitting-useful-bug-reports
• [x] This isn't a feature request or a discussion topic

---

Bug description

Unable to restore signal backup on new phone. Similar to #7637 but I don't have xposed installed. Signal stable closing without error after reading 11401 messages. Signal beta giving error message about incorrect passphrase after reading 11401 messages. I am certain the passphrase and the backup are correct as they are working fine using https://github.com/xeals/signal-back

Steps to reproduce

• copy signal backup to new phone
• open signal
• enter passphrase
• wait for messages to import

Actual result:

• Signal just closing (stable)
• Error message stating the passphrase is incorrect (beta).

Expected result: Backup is successfully restored.

Screenshots

beta error (in German due to phone language)

Device info

Device: Nokia 7 Plus Dual-Sim Android version: 9 Kernel version: 4.4.146-perf+ Signal version: Beta: 4.30.2 also tested on stable: 4.29.7

Link to debug log

Working on grabbing a logcat log but struggling with it. Will add through edit. Logcat for version 4.29.7:

--------- beginning of crash
11-08 22:32:10.330  4472  6987 E AndroidRuntime: FATAL EXCEPTION: AsyncTask #3
11-08 22:32:10.330  4472  6987 E AndroidRuntime: Process: org.thoughtcrime.securesms, PID: 4472
11-08 22:32:10.330  4472  6987 E AndroidRuntime: java.lang.RuntimeException: An error occurred while executing doInBackground()
11-08 22:32:10.330  4472  6987 E AndroidRuntime:    at android.os.AsyncTask$3.done(AsyncTask.java:354)
11-08 22:32:10.330  4472  6987 E AndroidRuntime:    at java.util.concurrent.FutureTask.finishCompletion(FutureTask.java:383)
11-08 22:32:10.330  4472  6987 E AndroidRuntime:    at java.util.concurrent.FutureTask.setException(FutureTask.java:252)
11-08 22:32:10.330  4472  6987 E AndroidRuntime:    at java.util.concurrent.FutureTask.run(FutureTask.java:271)
11-08 22:32:10.330  4472  6987 E AndroidRuntime:    at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:245)
11-08 22:32:10.330  4472  6987 E AndroidRuntime:    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
11-08 22:32:10.330  4472  6987 E AndroidRuntime:    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
11-08 22:32:10.330  4472  6987 E AndroidRuntime:    at java.lang.Thread.run(Thread.java:764)
11-08 22:32:10.330  4472  6987 E AndroidRuntime: Caused by: java.lang.NegativeArraySizeException: -1884239691
11-08 22:32:10.330  4472  6987 E AndroidRuntime:    at org.thoughtcrime.securesms.backup.FullBackupImporter$BackupRecordInputStream.readFrame(FullBackupImporter.java:310)
11-08 22:32:10.330  4472  6987 E AndroidRuntime:    at org.thoughtcrime.securesms.backup.FullBackupImporter$BackupRecordInputStream.readFrame(FullBackupImporter.java:252)
11-08 22:32:10.330  4472  6987 E AndroidRuntime:    at org.thoughtcrime.securesms.backup.FullBackupImporter.importFile(FullBackupImporter.java:81)
11-08 22:32:10.330  4472  6987 E AndroidRuntime:    at org.thoughtcrime.securesms.RegistrationActivity$2.doInBackground(RegistrationActivity.java:394)
11-08 22:32:10.330  4472  6987 E AndroidRuntime:    at org.thoughtcrime.securesms.RegistrationActivity$2.doInBackground(RegistrationActivity.java:386)
11-08 22:32:10.330  4472  6987 E AndroidRuntime:    at android.os.AsyncTask$2.call(AsyncTask.java:333)
11-08 22:32:10.330  4472  6987 E AndroidRuntime:    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
11-08 22:32:10.330  4472  6987 E AndroidRuntime:    ... 4 more

logcat for version 4.30.2:

11-10 13:15:20.687 27588 31822 W RegistrationActivity: null
11-10 13:15:20.687 27588 31822 W RegistrationActivity: java.io.IOException: Bad MAC
11-10 13:15:20.687 27588 31822 W RegistrationActivity:  at org.thoughtcrime.securesms.backup.FullBackupImporter$BackupRecordInputStream.readAttachmentTo(FullBackupImporter.java:298)
11-10 13:15:20.687 27588 31822 W RegistrationActivity:  at org.thoughtcrime.securesms.backup.FullBackupImporter.processAttachment(FullBackupImporter.java:140)
11-10 13:15:20.687 27588 31822 W RegistrationActivity:  at org.thoughtcrime.securesms.backup.FullBackupImporter.importFile(FullBackupImporter.java:87)
11-10 13:15:20.687 27588 31822 W RegistrationActivity:  at org.thoughtcrime.securesms.RegistrationActivity$2.doInBackground(RegistrationActivity.java:396)
11-10 13:15:20.687 27588 31822 W RegistrationActivity:  at org.thoughtcrime.securesms.RegistrationActivity$2.doInBackground(RegistrationActivity.java:388)
11-10 13:15:20.687 27588 31822 W RegistrationActivity:  at android.os.AsyncTask$2.call(AsyncTask.java:333)
11-10 13:15:20.687 27588 31822 W RegistrationActivity:  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
11-10 13:15:20.687 27588 31822 W RegistrationActivity:  at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:245)
11-10 13:15:20.687 27588 31822 W RegistrationActivity:  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
11-10 13:15:20.687 27588 31822 W RegistrationActivity:  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
11-10 13:15:20.687 27588 31822 W RegistrationActivity:  at java.lang.Thread.run(Thread.java:764)

[dorakas]

So basically now coming from #8347 it seems i evolved to this one here. With the latest 4.30.7 on Nexus 6P it finishes after 2 hours my backup and my Pixel3 - 4.30.7 Signal tells me that my password is wrong. Same problem as above. Tried https://github.com/xeals/signal-back and worked with the same file and same password. Since i reached from crashing Signal on Pixel with restoring backup, i think the next Level is succesfully restoring a backup. :D

Pixel 3 - https://debuglogs.org/8afd5e209b77295d13b983b38b06c71fd0cbe32f04e4521f5f5aad5b2f87a4d3

Nexus 6P - https://debuglogs.org/703329cfc7d727fab8c4cc4087552e37cde3d45abd8f88428ef92ea91d3ee583

[sarevok-anchev]

Same here. Running LineageOS 16.

Interestingly, with the wrong password specified, Signal will immediately claim that the password is incorrect.

With the right password, it reads around 2000 messages and then suddenly has a change of mind and claims that the password is incorrect after all. Seems legit.

*I tried both 4.26.2 and 4.30.8**.

4.26.2:

W RegistrationActivity:  at com.google.protobuf.CodedInputStream.pushLimit(CodedInputStream.java:651)
W RegistrationActivity:  at com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:307)
W RegistrationActivity:  at org.thoughtcrime.securesms.backup.BackupProtos$BackupFrame.<init>(BackupProtos.java:4637)
W RegistrationActivity:  at org.thoughtcrime.securesms.backup.BackupProtos$BackupFrame.<init>(BackupProtos.java:4514)
W RegistrationActivity:  at org.thoughtcrime.securesms.backup.BackupProtos$BackupFrame$1.parsePartialFrom(BackupProtos.java:4675)
W RegistrationActivity:  at org.thoughtcrime.securesms.backup.BackupProtos$BackupFrame$1.parsePartialFrom(BackupProtos.java:4670)
W RegistrationActivity:  at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:141)
W RegistrationActivity:  at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:176)
W RegistrationActivity:  at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:188)
W RegistrationActivity:  at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:193)
W RegistrationActivity:  at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
W RegistrationActivity:  at org.thoughtcrime.securesms.backup.BackupProtos$BackupFrame.parseFrom(BackupProtos.java:4937)
W RegistrationActivity:  at org.thoughtcrime.securesms.backup.FullBackupImporter$BackupRecordInputStream.readFrame(FullBackupImporter.java:328)
W RegistrationActivity:  at org.thoughtcrime.securesms.backup.FullBackupImporter$BackupRecordInputStream.readFrame(FullBackupImporter.java:252)
W RegistrationActivity:  at org.thoughtcrime.securesms.backup.FullBackupImporter.importFile(FullBackupImporter.java:81)
W RegistrationActivity:  at org.thoughtcrime.securesms.RegistrationActivity$2.doInBackground(RegistrationActivity.java:394)
W RegistrationActivity:  at org.thoughtcrime.securesms.RegistrationActivity$2.doInBackground(RegistrationActivity.java:386)
W RegistrationActivity:  at android.os.AsyncTask$2.call(AsyncTask.java:333)
W RegistrationActivity:  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
RegistrationActivity:  at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:245)
W RegistrationActivity:  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
W RegistrationActivity:  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
W RegistrationActivity:  at java.lang.Thread.run(Thread.java:764)

4.30.8:

W RegistrationActivity: null
 W RegistrationActivity: java.io.IOException: Bad MAC
W RegistrationActivity:  at org.thoughtcrime.securesms.backup.FullBackupImporter$BackupRecordInputStream.readFrame(FullBackupImporter.java:320)
W RegistrationActivity:  at org.thoughtcrime.securesms.backup.FullBackupImporter$BackupRecordInputStream.readFrame(FullBackupImporter.java:252)
W RegistrationActivity:  at org.thoughtcrime.securesms.backup.FullBackupImporter.importFile(FullBackupImporter.java:81)
W RegistrationActivity:  at org.thoughtcrime.securesms.RegistrationActivity$2.doInBackground(RegistrationActivity.java:396)
W RegistrationActivity:  at org.thoughtcrime.securesms.RegistrationActivity$2.doInBackground(RegistrationActivity.java:388)
W RegistrationActivity:  at android.os.AsyncTask$2.call(AsyncTask.java:333)
W RegistrationActivity:  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
W RegistrationActivity:  at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:245)
W RegistrationActivity:  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
W RegistrationActivity:  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
RegistrationActivity:  at java.lang.Thread.run(Thread.java:764)

I have restored backups (not this specific file) in the past a couple of times with no issues. The only change is that when it worked it was with Android 7.x and now I'm on 9.x. Probably not relevant though.

Clearly, the backup was a) generated by signal itself and b) in good health since signal-back can properly decode it.

[dorakas]

9.x. Probably not relevant though.

But maybe this is the relevant part

[mike-edel]

9.x. Probably not relevant though.

But maybe this is the relevant part

Indeed, seems to be a common thing...

[chainsawbeaver]

Not valid anymore! I had different Signal versions! Sorry for any confusion.

Same problem here with Android 8.1

Signal crashes after it reads the 4000+ Message. So perhaps it is not the same Issue? I cropped my conversations and deleted all media inside signal conversations, but that did not help. I also can open the backup with signal-back (only "umlaute" (ÄÜÖ) are displayed wrong).

[sarevok-anchev]

^^ There goes the 9.x theory.

[sarevok-anchev]

By the way, I did not see this issue restoring on LineageOS 15.1 (Android 8.1). Different backup file though (another account).

[sarevok-anchev]

Actually, it may be possible that the backup was not restored. This was a few weeks ago and my memory is flimsy. Sorry about the spam, not possible to edit comments anymore it seems..

[chainsawbeaver]

I had different Signal versions! Sorry for any confusion. So perhaps still a 9/pie issue?

[spospartan104]

Also experiencing this migrating from Nexus 6 Android 7.1.1 to Moto X4 Android 8.1.0

Decrypts messages (as it is counting the correct number to be restored) Then when it fails it throws Incorrect password. Log cat shows the Bad Mac error as well.

Has anyone found a work around?

Stock Roms (Nexus 6, Moto X4 Android One edition) Signal 4.30.8 on both as well

[sarevok-anchev]

Had the chance to day to try on Android 8.1 AOSP.

I zygote  : Deoptimizing void net.sqlcipher.database.SQLiteClosable.releaseReference() due to JIT inline cache
W RegistrationActivity: null
W RegistrationActivity: java.io.IOException: Bad MAC
W RegistrationActivity:  at org.thoughtcrime.securesms.backup.FullBackupImporter$BackupRecordInputStream.readFrame(FullBackupImporter.java:320)
W RegistrationActivity:  at org.thoughtcrime.securesms.backup.FullBackupImporter$BackupRecordInputStream.readFrame(FullBackupImporter.java:252)
W RegistrationActivity:  at org.thoughtcrime.securesms.backup.FullBackupImporter.importFile(FullBackupImporter.java:81)
W RegistrationActivity:  at org.thoughtcrime.securesms.RegistrationActivity$2.doInBackground(RegistrationActivity.java:396)
W RegistrationActivity:  at org.thoughtcrime.securesms.RegistrationActivity$2.doInBackground(RegistrationActivity.java:388)
W RegistrationActivity:  at android.os.AsyncTask$2.call(AsyncTask.java:333)
W RegistrationActivity:  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
W RegistrationActivity:  at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:245)
W RegistrationActivity:  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1162)
W RegistrationActivity:  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:636)
W RegistrationActivity:  at java.lang.Thread.run(Thread.java:764)

In one swift strike, it seems that we can rule out the culprit being Android 9 or LineageOS.

Perhaps @moxie0 has something to add at this stage?

[sarevok-anchev]

This is the very same backup file from last month by the way, which signal-back can perfectly read..

[edaubert]

Hi,

I also have this issue coming from an Android 7.1.2 and going to an Android 9.0.2

As far as I understand, this seems to be related to https://github.com/signalapp/Signal-Android/pull/8327

@greyson-signal @elkhadiy can maybe help us as they were the ones involved in https://github.com/signalapp/Signal-Android/pull/8327

I am using the 4.31.6 version which seems to be corresponding to the master version. Still I have the issue.

[elkhadiy]

Hi,

Just skimmed through this thread, I'll be back later on to help.

What I can say for now is that before 4.30.2, when restoring a backup, Signal was happily decrypting all messages without checking their message authentication code (MAC). #8327 fixes this check.

Now it seems we are dealing with some corrupt data in the backup. I'll hack together a tool to try and find the offending frames. (I'll keep you guys posted)

EDIT(2018-12-16):

So I'm a little puzzled by the fact that OP can decrypt the backup file with signal-back while he has a Bad MAC error. Are you sure the whole file is decrypted and that it didn't just decrypt the good frames? Could you use my tool : https://github.com/elkhadiy/debruitage ? I made sure to make it explicitly report all bad frames it encounters during the process.

EDIT(2018-12-18): Removed superfluous information irrelevant to the issue at hand.

[edaubert]

I ran your tool and got the following exception:

Traceback (most recent call last):
  File "/home/royalvein/bin/miniconda/envs/atom/bin/signal-bkp-decrypt", line 11, in <module>
    load_entry_point('signal-backup-manager==0.1.dev1', 'console_scripts', 'signal-bkp-decrypt')()
  File "/home/royalvein/bin/miniconda/envs/atom/lib/python2.7/site-packages/signal_backup_manager/cli.py", line 23, in run
    bkp = SignalBackup(args.backup_file, args.passphrase)
  File "/home/royalvein/bin/miniconda/envs/atom/lib/python2.7/site-packages/signal_backup_manager/signal_backup.py", line 22, in __init__
    + datetime.now().strftime("%Y-%m-%d--%H-%M-%S")
  File "/home/royalvein/bin/miniconda/envs/atom/lib/python2.7/site-packages/fs/osfs.py", line 311, in makedir
    _path = self.validatepath(path)
  File "/home/royalvein/bin/miniconda/envs/atom/lib/python2.7/site-packages/fs/base.py", line 1429, in validatepath
    else "paths must be str (not bytes)"
TypeError: paths must be unicode (not str)
Exception AttributeError: "SignalBackup instance has no attribute 'preference_file'" in <bound method SignalBackup.__del__ of <signal_backup_manager.signal_backup.SignalBackup instance at 0x7f77295e3200>> ignored

I do not know if it is the expected behavior... Basically, I clone your project, and follow the readme (pip install...).

However, I did not try signal-back before so i did it at the same time and I was not able to extract my messages and got:

error: failed to open backup file: failed to read headerLengthBytes: EOF

Maybe my issue is not the same after all even if the behavior seems similar (bad mac error + count of messages is good) to @spospartan104

So I try again to import my message and get the exact exception which is not the same as @sarevok-anchev (in term of the line numbers but maybe the version differs) Here are the logs:

W RegistrationActivity: null
12-19 23:45:39.169  4632  7965 W RegistrationActivity: java.io.IOException: Bad MAC
12-19 23:45:39.169  4632  7965 W RegistrationActivity:  at org.thoughtcrime.securesms.backup.FullBackupImporter$BackupRecordInputStream.readAttachmentTo(FullBackupImporter.java:298)
12-19 23:45:39.169  4632  7965 W RegistrationActivity:  at org.thoughtcrime.securesms.backup.FullBackupImporter.processAttachment(FullBackupImporter.java:140)
12-19 23:45:39.169  4632  7965 W RegistrationActivity:  at org.thoughtcrime.securesms.backup.FullBackupImporter.importFile(FullBackupImporter.java:87)
12-19 23:45:39.169  4632  7965 W RegistrationActivity:  at org.thoughtcrime.securesms.RegistrationActivity$2.doInBackground(RegistrationActivity.java:396)
12-19 23:45:39.169  4632  7965 W RegistrationActivity:  at org.thoughtcrime.securesms.RegistrationActivity$2.doInBackground(RegistrationActivity.java:388)
12-19 23:45:39.169  4632  7965 W RegistrationActivity:  at android.os.AsyncTask$2.call(AsyncTask.java:333)
12-19 23:45:39.169  4632  7965 W RegistrationActivity:  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
12-19 23:45:39.169  4632  7965 W RegistrationActivity:  at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:245)
12-19 23:45:39.169  4632  7965 W RegistrationActivity:  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
12-19 23:45:39.169  4632  7965 W RegistrationActivity:  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
12-19 23:45:39.169  4632  7965 W RegistrationActivity:  at java.lang.Thread.run(Thread.java:764)

[elkhadiy]

Oh my bad, that's because it's written in python3 so you can go ahead and pip uninstall signal-backup-manager then pip3 install .

Though signal-back's error message is weird, what's the first 4 bytes of your backup file?

~$ echo $(( 16#$(xxd -p -l 4 signal-yyyy-mm-dd-hh-mm-ss.backup) ))

[edaubert]

Well... My bads...

The file I get to run the command was corrupted.

I redownload it and now there is no error with signal-back and your tool. However there still the Bad Mac exception on android.

echo $(( 16#$(xxd -p -l 4 signal-yyyy-mm-dd-hh-mm-ss.backup) ))

gives me 54 with the non corrupted file.

[elkhadiy]

Are you sure Signal tries to restore from the file you checked?

For example you only have that one in /sdcard/Signal/Backups/?

[edaubert]

Yep I try with 2 different backups but each time, only one backup was available in this folder.

[edaubert]

Hi,

Well today, I choose to reset my phone because I mess with it to do some tests with multiple apps. And I choose to retry the installatio of Signal and the import of my backup and everything worked fine.

I don't know what was the cause of this issue but it seems resolved for me after the reset.

Anyway thanks for the help. Signal is really a cool app and I missed it !

[sarevok-anchev]

@elkhadiy I tried your tool and there's an error:

Bad frame @ file offset 645847
Raw Frame data:
b'\xd1\x0e\xd7\xc8\xcc\t\xd0/\x06/\xe5#\xb6\xcbo2:\xab\xfc\xa4]\x1e\xc3\x8e\xf4!C\xdd"\xd1\x15\xd3<\xa6\xea\xca\xeb\x9c$\xc5\n\x01\x12\x83v\x0eC\x0c\x14\xccq\xba\x96#\x123d\x98\xe4\x82\xb4m\xd3/\xa1\x15\xa2\xef1\xa4\r\xdd\x9d\xf3\x17$\x87\x0c\x9d\x11\n\xf3A\x16_\xf7i\x81\xf2NKQ\xd5Zd<\x05\t.k\x03\xa5X\xb5gN\x04\x0cF\xf8\xb2\xbd\x92\xed(\xe5.\xb7@\xad\xdcM\xbe\xd5*m5\xbfeM}\x07\x10t>\xada\x1b\x8b\xe2\x15\x91oE\xbb\xf8\xa9\xa8E8z\xc4\xca\x87{\x82X_di\xcf\xb9\xad\xec\xb9=\xf13\x11\xfc<\xf9i;\xd8sx\x83\xe3\x8b:r6\xadX\x1d\xac2\xed@\xba\xbc\xc0`\xb0\x9d\x07\xc7\xab\xe6j\xf0#`\x9e\x10\x18\x12\xcc\xf8)\xb5\xdeC\xba'
Attempting to build frame obj:
Traceback (most recent call last):
  File "/home/user/.local/bin/signal-bkp-decrypt", line 11, in <module>
    load_entry_point('signal-backup-manager==0.1.dev1', 'console_scripts', 'signal-bkp-decrypt')()
  File "/home/user/.local/lib/python3.6/site-packages/signal_backup_manager/cli.py", line 23, in run
    bkp = SignalBackup(args.backup_file, args.passphrase)
  File "/home/user/.local/lib/python3.6/site-packages/signal_backup_manager/signal_backup.py", line 50, in __init__
    for frame in self.__get_backup_frames():
  File "/home/user/.local/lib/python3.6/site-packages/signal_backup_manager/signal_backup.py", line 130, in __get_backup_frames
    frame.ParseFromString(plaintext)
google.protobuf.message.DecodeError: Error parsing message

Note that:

1. This is the only backup file in /sdcard/Signal/Backups, so I'm 100% sure it's the one Signal is attempting to use.

2. Signal-back can decrypt the backup: signal-back check $filename result is Backup looks okay from here.

3. Earlier you said: What I can say for now is that before 4.30.2, when restoring a backup, Signal was happily decrypting all messages without checking their message authentication code (MAC). #8327 fixes this check. .. however I've attempted to restore with 4.29.7 it's not possible either. I do remember having restored this backup with previous versions, but a) I don't remember which version and b) of course the backup file has changed since then, and therein might lie the problem.

I must say it's a little bit frustrating that the option to set a very long passphrase and encrypt the app data was removed in previous versions, I never lost a single backup that way over the years, now I'm forced to relay on the signal backup functionality, which apparently in certain unknown circumstances creates corrupted backups, and it's already the second time I'm losing data.

Personally I brought more than 100 people into Signal over the years, and now I can't reach any of them for more than a month because of this bug.

A bit offtopic, but I needed to vent - I had a good grip on backup flow, now need to rely on a dumbed down version, and it's the second time that it screws my data up. Not good.

Anyway..

Let's look towards a possible solution.

Can you offer some guidance on what the best next steps are from this position?

I don't want to lose the conversation history yet again, and since there's roughly a month and a half of undelivered messages now, I would like to keep the same encryption keys so that they get delivered (eventually).

The error must be in some corrupted attachment. signal-back has (on quick glance) all of the messages. It would be way less catastrophic to lose some attachments rather than lose the whole conversation history.

Nevertheless, signal-back can recover a lot of images and audio from this supposedly corrupted backup. Maybe even all of them - I wouldn't know.

Is it possible to make signal ignore a corrupted attachment rather than abort the backup in an all-or-nothing fashion?

Perhaps even have an option to ignore such errors altogether (at the users' peril) ?

How about extending your tool to ignore the errors and discard the file/message in question, and outputting a new backup file minus the "bad frames" ? That could be a fast solution to get signal going quickly again.

What do you think @elkhadiy?

[elkhadiy]

Oh yeah sorry I derped in my tool, forgot to catch the potential frame construction exception.

Signal-back can decrypt the backup: signal-back check $filename result is Backup looks okay from here.

Hmmm weird... I'll try and see what he does (but I'm not that good with golang).

however I've attempted to restore with 4.29.7 it's not possible either.

Yeah pretty sure you won't restore from this backup with the current state of affairs since at least this frame seems to be pretty borked, if protobuf can't parse it...

Is it possible to make signal ignore a corrupted attachment rather than abort the backup in an all-or-nothing fashion? How about extending your tool to ignore the errors and discard the file/message in question, and outputting a new backup file minus the "bad frames" ? That could be a fast solution to get signal going quickly again.

Yep, was thinking about proposing this in the app's backup restore process. Something flexible that would report bad messages or attachments and restores anyway... I'll put together something that'll analyse the database entries and reports what is most likely missing (sms/mms entry or attachment and maybe even its filename I think)

I was also planning on writing a module that rebuilds the backup using the same passphrase anyway for a friend of mine that wants to import his Facebook messenger history. Pretty sure we can fix your backup file!

I'll get back to you hopefully very soon.

[sarevok-anchev]

Beautiful, thank you.

[mike-edel]

@elkhadiy Thanks for the tool - was struggling a bit to get it working on Windows. Been a while since I ran stuff in a shell. Getting some odd errors: Bad frame @ file offset 76 Raw Frame data: b'\xf3EaU' Attempting to build frame obj:

Full console output: signal-bkp-decrypt.exe -b c:/signal-2018-11-08-22-04-02.backup -p '040871752236135417905193908720' Bad frame @ file offset 76 Raw Frame data: b'\xf3EaU' Attempting to build frame obj: Traceback (most recent call last): File "C:\Users\mikee\AppData\Local\Programs\Python\Python37-32\Scripts\signal-bkp-decrypt-script.py", line 11, in <module> load_entry_point('signal-backup-manager==0.1.dev1', 'console_scripts', 'signal-bkp-decrypt')() File "c:\users\mikee\appdata\local\programs\python\python37-32\lib\site-packages\signal_backup_manager\cli.py", line 23, in run bkp = SignalBackup(args.backup_file, args.passphrase) File "c:\users\mikee\appdata\local\programs\python\python37-32\lib\site-packages\signal_backup_manager\signal_backup.py", line 50, in __init__ for frame in self.__get_backup_frames(): File "c:\users\mikee\appdata\local\programs\python\python37-32\lib\site-packages\signal_backup_manager\signal_backup.py", line 130, in __get_backup_frames frame.ParseFromString(plaintext) File "c:\users\mikee\appdata\local\programs\python\python37-32\lib\site-packages\google\protobuf\message.py", line 185, in ParseFromString self.MergeFromString(serialized) File "c:\users\mikee\appdata\local\programs\python\python37-32\lib\site-packages\google\protobuf\internal\python_message.py", line 1083, in MergeFromString if self._InternalParse(serialized, 0, length) != length: File "c:\users\mikee\appdata\local\programs\python\python37-32\lib\site-packages\google\protobuf\internal\python_message.py", line 1109, in InternalParse new_pos = local_SkipField(buffer, new_pos, end, tag_bytes) File "c:\users\mikee\appdata\local\programs\python\python37-32\lib\site-packages\google\protobuf\internal\decoder.py", line 850, in SkipField return WIRETYPE_TO_SKIPPER[wire_type](buffer, pos, end) File "c:\users\mikee\appdata\local\programs\python\python37-32\lib\site-packages\google\protobuf\internal\decoder.py", line 799, in _SkipGroup new_pos = SkipField(buffer, pos, end, tag_bytes) File "c:\users\mikee\appdata\local\programs\python\python37-32\lib\site-packages\google\protobuf\internal\decoder.py", line 850, in SkipField return WIRETYPE_TO_SKIPPER[wire_type](buffer, pos, end) File "c:\users\mikee\appdata\local\programs\python\python37-32\lib\site-packages\google\protobuf\internal\decoder.py", line 782, in _SkipFixed64 raise _DecodeError('Truncated message.') google.protobuf.message.DecodeError: Truncated message.

Don't know what to make of it to be honest. And no idea whether signal-back does actually decrypt the whole thing. I ended up getting a huge xml file though.

[sarevok-anchev]

Any updates @elkhadiy ?

[unDocUMeantIt]

i'm also seeing this on a friend's phone, trying to restore a fresh backup from an old phone on a new one.

@elkhadiy your tool is great, and indeed it complains about a corrupted frame. could it be extended to re-encrypt the backup to a new file with all corrupted frames dropped?

[JamesHarrison]

I've just seen this on my phone; I was unable to send (#8602) and figured a reinstall would probably do the trick, but I'm unable to restore my messages. Fairly chunky backup at 60,000 messages. Can't get debruitage to run on my desktop as Ubuntu doesn't have the fts5 extension for sqlite packaged, but will try again on Arch later. Having Signal ignore (with some kind of message) corrupt MAC frames would be the ideal outcome, I think - losing 60,000 messages for one glitch isn't fabulous.

[bepaald]

Just wanted to let you guys know, I'm working on this. I've been able to decrypt (and interpret) the backup data for a long time and just started working on reencryption last week and making good progress.

I think a lot of the past messages shouldn't be in this github issue as they do not relate to any bug in Signal (supporting corrupted backups would be a feature request). The only possible bug here is that signal pops up a wrong password error when a bad Mac is encountered, while I think a 'file corrupted' popup would be way more likely if it happens any time after the first frame is decoded. So, if I have something to test for you, I will create a thread on the forum and post a link here. Note it might take a long time (think months) , as I sometimes don't have spare time for an extended period.

Also, I do not know how Signal handles backups with frames missing, so there are no guarantees this will work. Frames are not necessarily independent things (attachment data belongs to an entry in the 'part' database, which belongs to an entry in the 'mms' database. The restoration might for all I know still fail if one of those frames is missing. Also, random corruption in a random location might result in anything, that is, if you're unlucky, a single bit flipped might make not just one frame, but all following frames unreadable.

Anyway, I didn't intend for this message to get this long. Just wanted to let you know, a tool is coming though it might be a while.

[unDocUMeantIt]

The only possible bug here is that signal pops up a wrong password error when a bad Mac is encountered, while I think a 'file corrupted' popup would be way more likely if it happens any time after the first frame is decoded.

we were able to reliably reproduce 'corrupted' files. no matter how often we made new backups on the one phone, they always turn out to be corrupted on the other one. i wouldn't rule out the possibility that signal's backup routine has a hidden issue leading to corrupted frames in backups which triggered it, and the restore problems are merely a symptom.

[JamesHarrison]

I'd +1 that suggestion, @unDocUMeantIt - my phone has dozens of GB free, backups were allowed to complete fully onto internal storage with no sleep/screenlock, and restore attempts likewise. I can't think of any potential source for corruption of the backup other than in creation by Signal.

If the error is caused because the contents themselves are also being checked during restore, but in normal operation such corrupted contents are tolerated, then the behaviour should be modified to catch what would cause corruption at "point of entry" to storage and abort then, and to permit restore behaviour to ignore those corrupt frames on restoration.

[bepaald]

Well, first of all, I should indeed not have been so self assured stating there is no bug here. I am not a Signal developer and I am not 100% sure about anything.

However, having said that, being very familiar with the backup file format, and having read the code, I still find it very unlikely that a bug is present. You should realize the MAC is just a hash of the already encrypted bytes of data. During the backup creation process, data is serialized into an array of bytes, these bytes are then encrypted to produce another, different array of bytes. Then a hash is calculated from these bytes (and, indirectly, your passphrase), this hash is the MAC. Then, the encrypted bytes and the hash are written to the file.

Notice that serializing the data could go wrong, but it does not matter (I don't know if Signal even checks for this). Encrypting the data could go wrong as well, creating an impossibble-to-decrypt array of random bytes. This also does not matter. The MAC is calculated on the (bad, uninterpretable, random) array of bytes without regards to what these bytes represent.

During the backup importing process. First the array of encrypted bytes is read into memory, the same hash is calculated from them, and compared to the stored MAC. No decrypting is taking place, the bytes are in no way interpreted at this point and it is not known wether the bytes can be decrypted or would turn into a valid signal backup frame if they were. If the hash does not match the stored hash, you get the "Bad MAC" exception in the log and an 'incorrect passphrase' message pops up (erroneously). But really the MAC can not show if the bytes are 'correct' on the receiving phone, only that they are the same. Having a bad mac shows the bytes have not arrived unaltered on the new phone. This means one of three things:

1. The MAC is calculated wrong (and differently on the two phones). This would mean a bug in Java's crypto package. Unlikely...
2. The passphrase is wrong. Very unlikely if it has managed to successfully calculate the MAC of even one frame before.
3. The bytes have been changed. ... likely? I think this would mean one of the phones or inter-transfer storage has bad blocks of memory or some connection issues during transferring the backup file.

Again, I could be wrong about this.

If anyone still has a phone which reliably reproduces bad backups, try this:

1. Create a backup, let it finish. Rename the backup, do NOT delete it.
2. Immediately create another backup. Make sure the backup contents do not change in between (do not receive any messages). Both backups exist simultaneously on the same device, so they cannot occupy the same (possibly bad) flash memory.
3. Calculate checksums (for example md5sum) of both files on the phone, so that after transferring it can be checked that the copies arrive identical to the original.
4. Verify that the exact same frame gives the 'Bad MAC' exception from both backup files.

I think this would make it much more likely that Signal is the problem. Step 4 might be a little tricky at this moment, by my (forthcoming) program would make it easy.

I have by now, by the way, been able to read a backup, actually edit the contents of one of the messages and reencrypt with a new passphrase. I then imported this new backup on an actual phone with a normal signal installation successfully. I expect to be able to post a first version dropping bad frames later today (or at least this week). To compile the program you will need a recent c++ compiler (I use gcc version 8.2.1 20181127), and libcryptopp.

[bepaald]

I have posted a thread with the tool in its current shape here: https://community.signalusers.org/t/tool-to-re-encrypt-signal-backup-optionally-changing-password-or-dropping-bad-frames/6497

I hope it can help some of you.

[robertauer]

I can confirm that this "Bad MAC" bug still exists. I tried to migrate my Signal data from LineageOS 14.1 (Android 7.1, Signal version 4.35.3) to LineageOS 16.0 (Android 9.0, Signal version 4.35.3). I went through the first three steps of bepaald's suggested test (https://github.com/signalapp/Signal-Android/issues/8355#issuecomment-467786797). Both identical backups I created were transferred correctly to the new phone (sha256sums were the same) and both failed with the "Bad MAC" error in the log when I tried to import them. I will now try bepaald's script to remove bad frames...

[deleted]

I'm seeing this same problem while migrating data to a new phone.

Exporting from: Signal v4.39.4 on Android 8.1.0 (Pixel 1) Importing to: Signal v4.39.4 on Android 9 (Pixel 3a)

The backup file is about 1GB. It gets about 1900 records into the import, then give me the "Unable to restore encrypted backup" error message. I hooked up adb logcat and got this:

05-22 19:41:25.948   564   609 W SurfaceFlinger: Attempting to set client state on removed layer: Dim Layer for - Task=100#0
05-22 19:41:25.948   564   609 W SurfaceFlinger: Attempting to destroy on removed layer: Dim Layer for - Task=100#0
05-22 19:41:26.071   766  1033 I CHRE    : @ 24149.145: [AR_CHRE] still: 92
05-22 19:41:26.599 24220 24375 V Cursor  : Filling cursor window with start position:0 required position:0
05-22 19:41:26.634 24220 24375 I FullBackupImporter: Ignoring import for statement: CREATE TABLE sqlite_sequence(name,seq)
05-22 19:41:29.272   766  1033 I CHRE    : @ 24152.344: [AR_CHRE] still: 87
05-22 19:41:32.468   766  1033 I CHRE    : @ 24155.543: [AR_CHRE] still: 100
05-22 19:41:34.190 24220 24375 W RegistrationActivity: null
05-22 19:41:34.190 24220 24375 W RegistrationActivity: java.io.IOException: Bad MAC
05-22 19:41:34.190 24220 24375 W RegistrationActivity:  at org.thoughtcrime.securesms.backup.FullBackupImporter$BackupRecordInputStream.readAttachmentTo(FullBackupImporter.java:299)
05-22 19:41:34.190 24220 24375 W RegistrationActivity:  at org.thoughtcrime.securesms.backup.FullBackupImporter.processAttachment(FullBackupImporter.java:141)
05-22 19:41:34.190 24220 24375 W RegistrationActivity:  at org.thoughtcrime.securesms.backup.FullBackupImporter.importFile(FullBackupImporter.java:87)
05-22 19:41:34.190 24220 24375 W RegistrationActivity:  at org.thoughtcrime.securesms.RegistrationActivity$2.doInBackground(RegistrationActivity.java:376)
05-22 19:41:34.190 24220 24375 W RegistrationActivity:  at org.thoughtcrime.securesms.RegistrationActivity$2.doInBackground(RegistrationActivity.java:369)
05-22 19:41:34.190 24220 24375 W RegistrationActivity:  at android.os.AsyncTask$2.call(AsyncTask.java:333)
05-22 19:41:34.190 24220 24375 W RegistrationActivity:  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
05-22 19:41:34.190 24220 24375 W RegistrationActivity:  at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:245)
05-22 19:41:34.190 24220 24375 W RegistrationActivity:  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
05-22 19:41:34.190 24220 24375 W RegistrationActivity:  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
05-22 19:41:34.190 24220 24375 W RegistrationActivity:  at java.lang.Thread.run(Thread.java:764)
05-22 19:41:35.668   766  1033 I CHRE    : @ 24158.742: [AR_CHRE] still: 100
05-22 19:41:37.729  1167  2703 W NotificationService: Toast already killed. pkg=org.thoughtcrime.securesms callback=android.app.ITransientNotification$Stub$Proxy@4170b12

I've tried re-doing the export with the same results. It seems to be choking on that one attachment for whatever reason.

I have to disagree with @bepaald about this not being a bug. If Signal can generate backups with invalid attachment MACs, seems to me there's a bug in the way the MAC is computed or the formatted output is written.

I wish I could provide a reliable way to reproduce this without supplying my entire 1GB backup file and all my message & attachments. I think I'm going to see if I can get Android Studio set up and reproduce this against my data in a debugger.

[deleted]

Did a little hacking. Breakpointed it and caught the error in a debugger. It is indeed a failed MAC match:

It seems mildly interesting that the plaintext of the attachment (the output of cipher.doFinal()) is a zero-byte array (lines 282 and 285) despite 13693646 bytes having been fed to the cipher. Any crypto experts on this thread? What does it mean if you try to decrypt AES CTR mode and get zero-byte plaintext output?

Makes me think something has gone wrong either with the encryption of the attachment, or with frame alignment while streaming in the backup file.

Just for fun, I tried disabling the MAC check by commenting out the line that throws the exception and running that import, and it ran out of RAM:

05-22 23:57:15.721  5531  5587 E UncaughtExceptionLogger: java.lang.RuntimeException: An error occurred while executing doInBackground()
05-22 23:57:15.721  5531  5587 E UncaughtExceptionLogger:   at android.os.AsyncTask$3.done(AsyncTask.java:354)
05-22 23:57:15.721  5531  5587 E UncaughtExceptionLogger:   at java.util.concurrent.FutureTask.finishCompletion(FutureTask.java:383)
05-22 23:57:15.721  5531  5587 E UncaughtExceptionLogger:   at java.util.concurrent.FutureTask.setException(FutureTask.java:252)
05-22 23:57:15.721  5531  5587 E UncaughtExceptionLogger:   at java.util.concurrent.FutureTask.run(FutureTask.java:271)
05-22 23:57:15.721  5531  5587 E UncaughtExceptionLogger:   at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:245)
05-22 23:57:15.721  5531  5587 E UncaughtExceptionLogger:   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
05-22 23:57:15.721  5531  5587 E UncaughtExceptionLogger:   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
05-22 23:57:15.721  5531  5587 E UncaughtExceptionLogger:   at java.lang.Thread.run(Thread.java:764)
05-22 23:57:15.721  5531  5587 E UncaughtExceptionLogger: Caused by: java.lang.OutOfMemoryError: Failed to allocate a 1802744496 byte allocation with 4572351 free bytes and 507MB until OOM, max allowed footprint 9144703, growth limit 536870912
05-22 23:57:15.721  5531  5587 E UncaughtExceptionLogger:   at org.thoughtcrime.securesms.backup.FullBackupImporter$BackupRecordInputStream.readFrame(FullBackupImporter.java:314)
05-22 23:57:15.721  5531  5587 E UncaughtExceptionLogger:   at org.thoughtcrime.securesms.backup.FullBackupImporter$BackupRecordInputStream.readFrame(FullBackupImporter.java:256)
05-22 23:57:15.721  5531  5587 E UncaughtExceptionLogger:   at org.thoughtcrime.securesms.backup.FullBackupImporter.importFile(FullBackupImporter.java:81)
05-22 23:57:15.721  5531  5587 E UncaughtExceptionLogger:   at org.thoughtcrime.securesms.RegistrationActivity$2.doInBackground(RegistrationActivity.java:376)
05-22 23:57:15.721  5531  5587 E UncaughtExceptionLogger:   at org.thoughtcrime.securesms.RegistrationActivity$2.doInBackground(RegistrationActivity.java:369)
05-22 23:57:15.721  5531  5587 E UncaughtExceptionLogger:   at android.os.AsyncTask$2.call(AsyncTask.java:333)
05-22 23:57:15.721  5531  5587 E UncaughtExceptionLogger:   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
05-22 23:57:15.721  5531  5587 E UncaughtExceptionLogger:   ... 4 more

Also, I've verified that the MD5 sums of the backup files on the source and target phones match, so I'm pretty sure it's not corruption due to failed copying, although it's possible that the source phone could have failed to write the data correctly in the first place.

[bepaald]

I'm still intrigued by this issue. Do you happen to have enough free space on your device to get two backup files on there at the same time? I would still like to see the problem occur at the exact same place in two different backups (as described here).

I have been rewriting my program, which scans for bad frames, to actually build a database so it will be easier to see exactly what attachment (and what message) causes the problem. Unfortunately, I can't get it to compile under Windows (that is, I do not have Windows and cross-compiling isn't working). But, if you feel comfortable using Linux, I will happily give you the sources. The program might be able to tell you exactly what message the attachment belongs to and will try to dump the attachment data (corrupted or not) so you can inspect the attachment both in Signal as well as what is in the backup. Let me know if you want to try. Also, as a suboptimal workaround, if you know what message causes the problem, you could try to get a working backup by deleting it.

It seems mildly interesting that the plaintext of the attachment (the output of cipher.doFinal()) is a zero-byte array (lines 282 and 285) despite 13693646 bytes having been fed to the cipher. Any crypto experts on this thread? What does it mean if you try to decrypt AES CTR mode and get zero-byte plaintext output?

I am definitely NOT a crypto-expert, but I don't think this is an issue. The doFinal() message is really mainly to reset the Cipher, and yes, it might return some data if there was still some left in the buffer, but I think that would actually be the unusual case.

Note: the following is even more speculation then everything else I'm saying:

Just for fun, I tried disabling the MAC check by commenting out the line that throws the exception and running that import, and it ran out of RAM

I think this is interesting as well. The backup importer doesn't just read data until it gets some end marker, instead every frame is preceded by 4 bytes giving the length of the next frame, which is then read. So, even if a frame was bad - the encryption gives bad output and the MAC is calculated or written incorrectly - the next frame should be all good again. The exception above shows it's trying to allocate almost 1.7 Gb (that is more than the entire backup file!) which might mean it is reading a corrupted frame size. Note these four bytes framesize are not part of the frame itself and are not encrypted, just a plain 32bit integer in between each frame. So this might mean more data is corrupted in the backup file (and not because of encryption). If this is the case I would have expected the exception to be thrown at line 311, did the code move down a few lines in your edits?

[...] although it's possible that the source phone could have failed to write the data correctly in the first place.

This is still the first thing I would like to see ruled out. The input to the encryption functions varies greatly, just toggling the backup preference changes the keys used for encryption, and thus the data that is hashed to get the MAC (the MAC is calculated from the encrypted data). In fact, toggling the preference is probably not even necessary as there is random key material generated when a new backup is created. The chance that a second backup has the same bad MAC seems really low, unless of course the bug is very common and easy to run into, but then many more people should have this issue. Anyway, I'm not saying there is definitely no bug, I just still think the more likely explanation should be ruled out first.

[deleted]

@bepaald Oh, yeah. You're correct about doFinal(). I misread that the first time. Plaintext is getting iteratively written to the output stream after successive calls to cipher.update().

I'd love to get ahold of your verification code. I'm running Linux as well. I do have multiple exports I can compare, taken at slightly different points in time.

I also my backup through xeals/signal-back extract, and the attachment with the same ID that fails MAC check in the debugger "decrypts" to 14MB of garbage, which supports the corrupted-on-export theory. All the prior attachments extract just fine. signal-back check doesn't find any problems, but I don't think it's checking attachment frames at all.

I would have expected the exception to be thrown at line 311, did the code move down a few lines in your edits?

Yeah, I added a few lines for debugging & then commented them out.

[bepaald]

@deleted Here you go: https://send.firefox.com/download/9150b1f452b16fc8/#DHCNlAGrBkVpcUqtWONO9A. See newer code in a later post.

There is a little buildscript that works for me, but might need some editing depending on your library paths and such. It should at least give you an idea how to compile it. Make sure you have a recent version of g++, and have (development packages of) crypto++ and sqlite3.

Also, this program is slightly edited, custom made for you, it does not actually have all the options it says when run without arguments. Just run ./badmacinfo name-of-backup-file.backup [30digitspassphrase] and it will try to do its thing. I hope it will be useful somehow.

Let me know if you need help with this. If you have any bright ideas that can be tested with the some adjustments to the program let me know (or code them yourself of course).

If you have a second phone, it might be interesting to forward the attachment (possibly the only unchanging bytes in your multiple broken backups) to that phone and see if its backups also turn bad...

[deleted]

@bepaald Thanks for the code. Unfortunately I haven't been able to build it. Furthest I've been able to get is in a debian:sid container, but it still doesn't seem to distribute an up-to-date version of libcrypto++:

BUILDING (2/22): gcc -std=c++2a -c -Wall -Wextra -Wshadow -Wold-style-cast -Woverloaded-virtual -pedantic -fomit-frame-pointer -O3 -march=native -o "cryptbase/o/getbackupkey.o" "cryptbase/getbackupkey.cc"
cryptbase/getbackupkey.cc: In member function 'bool CryptBase::getBackupKey(const string&)':
cryptbase/getbackupkey.cc:27:13: error: 'byte' is not a member of 'CryptoPP'
   CryptoPP::byte digest[CryptoPP::SHA512::DIGESTSIZE];

This page seems to suggest that the CryptoPP:byte type was added in libcrypto++ 6.0.

What distro & version are you using?

[bepaald]

@deleted O wow, I wouldn't have thought the 'unstable' version of debian would come with an almost three year old version of cryptopp. I'm running arch with cryptopp 8.2.0, I see that version is also available in debian 'experimental', is it possible to install that one? Alternatively, you could try to just remove the CryptoPP:: from the all the occurrences of CryptoPP::byte, as you probably already learned from the page you linked, byte was in the global namespace in the earlier versions of cryptopp. But it might lead to a clash with the byte added in c++ (though I do not have a using directive for std::) and you might run into other problems with the old version of cryptopp, I don't think I have ever written programs with version prior to 6, I don't know if everything will be backwards compatible.

[deleted]

Got it to work in an Arch container. For anyone following along from home, here's the Dockerfile I ended up with:

FROM archlinux/base

RUN pacman --noconfirm -Sy gcc crypto++ sqlite3 libstdc++5
ADD . /src
WORKDIR /src
RUN bash BUILDSCRIPT.sh

I also had to add -lstdc++ to the end of the last line of the build script to get it to link properly.

Thanks for this. The info it dumps on MAC failure is really helpful. Two different exports indeed fail on the same attachment file. I'm going to try deleting that and redoing the export.

[deleted]

@bepaald Update: I deleted the entire conversation with the bad attachment in it. That allowed my to export an uncorrupted backup, so I'm likely going to walk away from this problem for now.

If you're still intrigued, I've posted to output of your tool: https://github.com/deleted/badmacinfo-output

That has the stdout from the tool (with phone numbers scrubbed) and the binary of the decrypted attachment. It's an animated .gif that seems to display fine, but has some artifacts near the end that might be corrupted data.

Thanks for your help! The badmacinfo tool was really useful for me. I'd encourage you to publish the source here on github, if you're so inclined. It may help others.

[bepaald]

@deleted Thanks for following up! In my testing, I just changed 10 bytes in one of my backups and happened to catch an animated gif as well. I also just saw a few artifacts in one of the last frames, so that seems about right. I can't seem to open the file you uploaded at all by the way, no program I tried recognizes it as a valid image (it doesn't even start with a gif signature).

I don't think there is very much I can do without actually having a broken backup myself, and even then it's probably a difficult problem to definitively solve. Some questions that could possibly have been interesting to get answers to (since you deleted the conversation, you probably can't answer any of these anymore, and that's ok, but I'm going to ask them just in case, or for the next person that comes to this thread with this problem): Did you also try deleting just the one single message, and did that not work? And also, did you view the .gif from within your signal conversation before deleting it, did it look good from there? Did you happen to save the image from your conversation, and did that work and was it uncorrupted?

Anyway, I'm glad you have been able to get a working backup and use your new phone with (most of) your messages transferred.

EDIT

Thanks for your help! The badmacinfo tool was really useful for me. I'd encourage you to publish the source here on github, if you're so inclined. It may help others.

Yeah, I plan to release some code sometime in the future, it's just not there yet (as far as the main features I had planned, mainly editing the backup). Also, I have an exporter that is able to reencrypt a parsed backupfile, and a (mostly) working GUI for viewing the backup files on PC (should look almost exactly like on your phone's screen. But - and don't ask me how this happened - they're three separate programs and I feel like I should probably merge them into one giant signal-backup-tool. Now if only I had all the free time in the world...

In the meantime I hope people just find this thread or the one in the community forums where I also managed to fix someones backup with an older version of this tool.

[spospartan104]

@bepaald I had the same problem so I will actually test the viewing gif, deleting single message, etc in this. Will report back within the week I hope.

[solarimix]

typical Signal App bad vibes from programmers :(

[spospartan104]

@bepaald So I grabbed the tool you posted above with the firefox send link but am not receiving the same output style as 'deleted' Not sure if it's because of the custom style or not

spospartan104@:~/bin/signal-debugging/badmacinfo $./badmacinfo Signal/Backups/signal-2019-04-14-04-09-14.backup $(cat backupcode) newfile Reading backup file...

WARNING: Bad MAC in frame: theirMac: (hex:) ce d5 9e a3 9f 39 53 e9 31 87 ourMac: (hex:) f6 49 63 4e f8 74 e6 22 5b de 35 b0 23 9e 25 93 53 f8 25 43 0b 44 49 12 00 71 15 d5 b7 4d bf d0 terminate called after throwing an instance of 'CryptoPP::InvalidKeyLength' what(): AES/CTR: 0 is not a valid key length Aborted (core dumped)

[bepaald]

@spospartan104 I'm sorry, I don't know exactly how that can happen. Does it count the frames (and the percentage) up before this happens, or is this the very first thing it prints? Is it possible your backup is corrupted from the very first frame? What happens if you try to restore this backup in Signal, does it count up the frames and then suddenly stop, or does it fail immediately?

Of course it is also possible I have a bug in my program. In fact, there certainly is, it should not try to create an AES/CTR object with a 0 key size, it should fail earlier if the key has length zero (but I don't understand how it gets in that situation). I have a new version, which will not fix your problem, but might tell us something about why it fails. Maybe you could try this one https://send.firefox.com/download/a004f35d6b9b77a6/#lGciKr0mQpXxdNSZTmEodQ?

[spospartan104]

@bepaald no apologies necessary, you're being a big help. Thank you. I shall try that latter this evening. If you have an updated source I'll gladly poke around there too. Need to rebuild my Java environment anyway

On Mon, May 27, 2019, 07:59 bepaald notifications@github.com wrote:

@spospartan104 https://github.com/spospartan104 I'm sorry, I don't know exactly how that can happen. Does it count the frames (and the percentage) up before this happens, or is this the very first thing it prints? Is it possible your backup is corrupted from the very first frame? What happens if you try to restore this backup in Signal, does it count up the frames and then suddenly stop, or does it fail immediately?

Of course it is also possible I have a bug in my program. In fact, there certainly is, it should not try to create an AES/CTR object with a 0 key size, it should fail earlier if the key has length zero (but I don't understand how it gets in that situation). I have a new version, which will not fix your problem, but might tell us something about why it fails. Maybe you could try this one https://send.firefox.com/download/a004f35d6b9b77a6/#lGciKr0mQpXxdNSZTmEodQ ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/signalapp/Signal-Android/issues/8355?email_source=notifications&email_token=AC7QU7ACES6XAVG7Z4UQFOLPXPEI7A5CNFSM4GC73FNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWJTUIY#issuecomment-496187939, or mute the thread https://github.com/notifications/unsubscribe-auth/AC7QU7GYU7HZUAUTU2YWXELPXPEI7ANCNFSM4GC73FNA .

[bepaald]

@spospartan104 Any luck with the code I posted earlier?

[spospartan104]

@bepaald sorry ran out of time the other day. Trying to see if I can build this now. I don't see a Make file so gotta remember my ol' gcc

EDIT: If you can provide your build steps I'd apprecaite it