bepaald
commented
2 years ago Thank you so much for you report, it is always good to hear when things are still working.
I've made pr #75 to the readme describing the default password behavior when none is specified -- I presumed that it would generate a new one but it actually uses the password of the second (last?) backup specified.
Yes, I noticed, I will respond to that one right after this.
non-unique storage key [...] entries in usually empty tables [...]
These are actually both an indication of the same problem. I have never seen a backup file with entries in the 'storage_key' table, I'm not sure how to get entries in that table, and so I have never tested it properly. I had the program set up to try and deal with that table in a somewhat default way (but issue a warning), but I have now been reading Signal's source code and made adjustments. Hopefully (though I am unable to test), the errors and warnings concerning this table should now be gone. The solution, by the way, is to delete the double key entries before the merge. In your case, this wasn't done and thus the program failed to insert the double entries during merge. This causes an ugly error, but results in the exact same database as the program with the fix would create. So, thanks for reporting!
attachment data not found
This is a recurring problem. Really there is no error here, the attachment data is simply not there. Not too long ago I thought I had finally figured out under what circumstances this happened and suppressed the warning in those cases (fc1ddba0ede08e3f81288a5204179fe81496b0f5), but apparently not. You might be able to help me by running signalbackup-tools [input] [password] --scanmissingattachments on the merged backup. Though given the number of warnings, it will probably generate a lot of output. It might help me further understand when attachment data is expected to be missing.
Anyway, thanks again. Your report helped fix a small error in the program. Even so, I see no problems in you rmerge so I hope everything is working for you. If you encounter any problems later on, please let me know!
mivanit
ChiIIBiII
jirislav
I used the merge backup feature to merge backups of sizes ~14gb and ~110mb, and everything seems to work fine!
I've made pr #75 to the readme describing the default password behavior when none is specified -- I presumed that it would generate a new one but it actually uses the password of the second (last?) backup specified.
I haven't checked everything in detail (obviously) but the total number of messages and attachments both seem correct. This was done with backups version > 140, if that's useful.
Below are the errors/warnings I noticed from the logs. I didn't save the full logs, but searching for
errororwarningonly returned items of the types below.non-unique storage key
It might be worth noting that the passwords to the different backups were different, and the backups were non-overlapping with a gap of several weeks
section of log
``` Importing statements from source table 'storage_key'...8 entries... Error : After sqlite3_step(): UNIQUE constraint failed: storage_key.key Query: "INSERT INTO storage_key (_id,type,key) VALUES (?,?,?)" Error : After sqlite3_step(): UNIQUE constraint failed: storage_key.key Query: "INSERT INTO storage_key (_id,type,key) VALUES (?,?,?)" Error : After sqlite3_step(): UNIQUE constraint failed: storage_key.key Query: "INSERT INTO storage_key (_id,type,key) VALUES (?,?,?)" Error : After sqlite3_step(): UNIQUE constraint failed: storage_key.key Query: "INSERT INTO storage_key (_id,type,key) VALUES (?,?,?)" Error : After sqlite3_step(): UNIQUE constraint failed: storage_key.key Query: "INSERT INTO storage_key (_id,type,key) VALUES (?,?,?)" Error : After sqlite3_step(): UNIQUE constraint failed: storage_key.key Query: "INSERT INTO storage_key (_id,type,key) VALUES (?,?,?)" Error : After sqlite3_step(): UNIQUE constraint failed: storage_key.key Query: "INSERT INTO storage_key (_id,type,key) VALUES (?,?,?)" Error : After sqlite3_step(): UNIQUE constraint failed: storage_key.key Query: "INSERT INTO storage_key (_id,type,key) VALUES (?,?,?)" Importing statements from source table 'mention'...0 entries... Importing statements from source table 'payments'...0 entries... ```entries in usually empty tables
this one comes up many times, always right after
Compacting table: payments (_id)I've never made any payments with signal though, as far as I can recall
section of log
``` setMinimumId mention compactIds Compacting table: mention (_id) setMinimumId reaction compactIds Compacting table: reaction (_id) setMinimumId notification_profile Adjusting 'notification_profile_allowed_members.notification_profile_id' to match changes in 'notification_profile' Adjusting 'notification_profile_schedule.notification_profile_id' to match changes in 'notification_profile' compactIds Compacting table: notification_profile (_id) setMinimumId notification_profile_allowed_members compactIds Compacting table: notification_profile_allowed_members (_id) setMinimumId notification_profile_schedule compactIds Compacting table: notification_profile_schedule (_id) setMinimumId payments compactIds Compacting table: payments (_id) WARNING : Found entries in a usually empty table. Trying to deal with it, but problems may occur. setMinimumId storage_key compactIds Compacting table: storage_key (_id) setMinimumId sender_key_shared compactIds Compacting table: sender_key_shared (_id) setMinimumId sender_keys compactIds Compacting table: sender_keys (_id) setMinimumId pending_retry_receipts compactIds Compacting table: pending_retry_receipts (_id) setMinimumId avatar_picker compactIds Compacting table: avatar_picker (_id) setMinimumId distribution_list Adjusting 'recipient.distribution_list_id' to match changes in 'distribution_list' Adjusting 'distribution_list_member.list_id' to match changes in 'distribution_list' compactIds ```attachment data not found
there are a bunch of messages (~4k) like the following after
Writing SqlStatementFrame(s)...