Minor: If using parameter "--no-showprogress" then "Dealing with table ..." in log output is doubled

[hugogithubs]

Hi, I found a minor issue with log output if using "--no-showprogress". Some output is doubled.

I hope you don't think it's too picky if I report small or cosmetic issues to you. But who knows how other users process the log output in their own scripts ... :-)

Best regards

Description: One of the "Dealing with table ..." statements is too much. All lines should have a "done" at end of line.

Used Version:
signalbackup-tools (signalbackup-tools) source version 20231231.113206 (Win)

Actual result:

Writing SqlStatementFrame(s)...
  Dealing with table 'avatar_picker'...   Dealing with table 'avatar_picker'... 0/0 entries...
  Dealing with table 'recipient'... done
  Dealing with table 'thread'... done
  Dealing with table 'message'... done
  Dealing with table 'call'...   Dealing with table 'call'... 0/0 entries...
  Dealing with table 'call_link'...   Dealing with table 'call_link'... 0/0 entries...
  Dealing with table 'cds'... done
  Dealing with table 'chat_colors'...   Dealing with table 'chat_colors'... 0/0 entries...
  ...
Writing SharedPrefFrame(s)...

Expected result:

Writing SqlStatementFrame(s)...
  Dealing with table 'avatar_picker'... 0/0 entries... done
  Dealing with table 'recipient'... done
  Dealing with table 'thread'... done
  Dealing with table 'message'... done
  Dealing with table 'call'... 0/0 entries... done
  Dealing with table 'call_link'... 0/0 entries... done
  Dealing with table 'cds'... done
  Dealing with table 'chat_colors'... 0/0 entries... done
  ...
Writing SharedPrefFrame(s)...

[bepaald]

Thanks for reporting, definitely not too picky. Besides I'm working my way through all the output statements anyway to deal with #164, so I thought I'd do that now and hopefully fix this along the way.

Note that with --no-showprogress, I now skip the X/Y entries... part even if it's 0/0, just for consistency. I thought I'd mention it, since you might depend on it when parsing the output in a script. Let me know if it works correctly for you now.

Thanks again!

[hugogithubs]

Hi, log output looks good now!

But while checking output I found following log output lines (which were also there before with older versions of signal-backup-tools.exe):

When merging next cropped backup file to previous merged long term backup file I get:

Importing statements from source table 'kyber_prekey'...119 entries...
  NOTE: Dropping column 'kyber_prekey.stale_timestamp' from source : Column not present in target database
[Error]: After sqlite3_step(): UNIQUE constraint failed: kyber_prekey.key_id
         -> Query: "INSERT INTO kyber_prekey (_id,account_id,key_id,timestamp,last_resort,serialized) VALUES (?,?,?,?,?,?)"

Nevertheless it looks like it is working anyway but I am not sure. Should I be worried about this? Or is it related to the fact that I am using a merge file for months?

Due to lack of storage space, I delete regularly the oldest data on my cell phone, but would like to have a "complete backup file" with all data as a backup. This is the reason for months old merge file.

[bepaald]

log output looks good now!

Good, that's one down.

When merging next cropped backup file to previous merged long term backup file I get:

Ok, the NOTE is expected when importing into older databases that do not have the same tables/columns. the error was a bug. I think I fixed it now, but the fix is basically just to delete the doubled kyber_prekeys before importing, so the end result should be exactly what you were getting anyway. Maybe you cold let me know if the error is gone, and if no new errors show up?

I must admit though, I have not fully figured out what the kyber_prekey table actually is, so I'm not 100% sure how to deal with it.

Nevertheless it looks like it is working anyway but I am not sure. Should I be worried about this? Or is it related to the fact that I am using a merge file for months?

Due to lack of storage space, I delete regularly the oldest data on my cell phone, but would like to have a "complete backup file" with all data as a backup. This is the reason for months old merge file.

Well, you certainly have admirable faith in the merge functionality :)

I think if you keep adding new messages, from new databases, into the same old target database, and you do this for years, something is sure to go wrong at some point. The NOTE is a first indication, but over time, more and more notes will probably appear, and at some point there will be an actual incompatibility. Probably before that, you will have a large database, containing all your messages, at some database version that is incompatible with Signal itself, so even though its valid (and this program can deal with it, to export HTML and such) you could never import it into Signal anyway. Signal only imports a certain number of old databases (currently version 149 and up).

The way around this would be to — at least every once in a while — use the newest database as the target and the older as the source, that should bring the 'complete' backup file up to the current database version. But that may cause other problems in the long run, at the very least it will be a lot slower.

[hugogithubs]

Thanks for your reply.

Well, you certainly have admirable faith in the merge functionality :) For the Moment I can say that it does its job very well :-)

The way around this would be to — at least every once in a while — use the newest database as the target and the older as the source, that should bring the 'complete' backup file up to the current database version. But that may cause other problems in the long run, at the very least it will be a lot slower.

Yes, that was also my first thought when I realized that I am running into a problem in the future and swapped the files for the first time today when merging. Now I'm on database version 210 with my long term merged file.

When now merging next cropped daily files the error messages are gone with version version 20240101.173827 (Win) after "updating" database version to 210.

I will now test your latest release and report back to you regarding kyber_prekey.

[hugogithubs]

Hi,

I am sorry, but anything is broken now with last Version: 20240101.210703 (Win) when merging. (With previous version it worked but throw errors as mentioned above).

 .....
cleanDatabaseByMessages
  Deleting attachment entries from 'part' not belonging to remaining message entries
  Deleting other threads from 'thread'...
updateThreadsEntries
  Dealing with thread id: 54
  Deleting entries from 'mention' not belonging to remaining mms entries
  Deleting removed groups...
  Deleting unneeded MessageSendLog entries...
  Deleting reactions to non-existing messages
  Deleting call details from non-existing messages
  Deleting unreferenced recipient entries...
  Deleted 6 unreferenced recipients
  Deleting unneeded notification profiles entries...
  Deleting unused avatars...
  Deleting unused attachments...
  Delete others from 'identities'
  Deleting group_receipts entries from deleted messages...
  Deleting group_receipts from non-existing recipients
  Deleting drafts from deleted threads...
  Deleting remapped recipients for non existing recipients
  Vacuuming database

terminate called after throwing an instance of 'std::bad_any_cast'
  what():  bad any_cast

==============================

[bepaald]

Oops, so sorry, I really should have tested my additions better before heading to bed. Should be fixed now. Apologies again.

[bepaald]

Are both problems mentioned in this issue solved? If so, I'd like to close this. Thanks!

[hugogithubs]

Sorry. Yes, thank you. You can close!