After merging different db versions Some avatar pictures are disappeared and replaced by initials

[hugogithubs]

Used SW: signalbackup-tools (signalbackup-tools) source version 20240208.201758 (Win)

If I merge and export to html some of avatar sticker / pictures are replaced by its initials. I noticed this issue also some time ago. If I check the two files before with exporthtml then they are ok.

Unfortunately, I can't find a rule for this problem yet. It doesn't seem to matter how I merge (small to big / older to newer and vice versa)

I don't know if it has anything to do with following, but I often have log entrys like:

[Warning]: Source database contains 'remapped_recipients'. This case may not yet be handled correctly by this program!
2 : 339 -> 321

Old id:
(no results)

New id:
_id|type|e164|aci|pni|username|email|group_id|distribution_list_id|call_link_room_id|registered|unregistered_timestamp|blocked|hidden|profile_key|profile_key_credential|profile_sharing|profile_given_name|profile_family_name|profile_joined_name|profile_avatar|last_profile_fetch|system_given_name|system_family_name|system_joined_name|system_nickname|system_photo_uri|system_phone_label|system_phone_type|system_contact_uri|system_info_pending|notification_channel|message_ringtone|message_vibrate|call_ringtone|call_vibrate|mute_until|message_expiration_time|sealed_sender_mode|storage_service_id|storage_service_proto|mention_setting|capabilities|last_session_reset|wallpaper|wallpaper_uri|about|about_emoji|extras|groups_in_common|avatar_color|chat_colors|custom_chat_colors_id|badges|needs_pni_signature|reporting_token|phone_number_sharing
321|0|(NULL)|XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX|(NULL)|(NULL)|(NULL)|(NULL)|(NULL)|(NULL)|2|XXXXXXXXXXXXX|0|0|XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX|XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX|0|XXXXX||XXXXX|(NULL)|XXXXXXXXXXXXX|(NULL)|(NULL)|(NULL)|(NULL)|(NULL)|(NULL)|2|(NULL)|0|(NULL)|(NULL)|0|(NULL)|0|0|0|2|(NULL)|(NULL)|0|XXXXXX|(NULL)|(NULL)|(NULL)|(NULL)|(NULL)|(NULL)|1|A130|(NULL)|(NULL)|(hex:) |(NULL)|(NULL)|0

 REMAP RECIPIENTS! 
  Mapping 264 -> 263 (up)
  Mapping 339 -> 321 (up)
  Mapping 135 -> 344 (up)

[bepaald]

Hi, thanks for your report. I will try to reproduce this tomorrow.

I think I already know what's happening, I hope I have time to fix it tomorrow, otherwise monday for sure.

Thanks again!

[bepaald]

Two things can be happening here. One is intended, the other is a bug (that is hopefully now fixed).

1. When merging, the input file is considered the 'true' database state. If a recipient is imported from the source, that already exists in the input, it is skipped together with any avatars that may be present for this recipient in the source. The recipient (and its datastate) in the input are considered valid and complete. This prevents old or removed avatars from being restored unintentionally. However, it could also cause the reverse: if you are importing a newer backup into an older one, and in the newer backup an existing recipient has a new avatar not present in the old backup, it is not imported.

   Originally, this was not a big problem as the function was only created to keep people from losing messages, and when restoring the backup to phone most avatars would likely be restored at some point during normal operations. Obviously this doesn't happen in the HTML export. I do not know of a good way around this that can't cause problems when restoring the backup.

2. There is (was) also a bug happening. I actually consider this a pretty big problem. I almost can't believe I didn't think of this earlier, and this has been like this basically since the merge functionality was created.

   During the merge, the id's of the recipients in the database are changed (multiple times actually). This is one of the most complex operations in the process and care must be taken to change the id's the same way in every place they occur (mentions, reactions, group members, etc...). But up until now, I was not updating the ids in the avatars (I think mostly because the avatars actually live outside the table where the rest of these ids are). But as soon as I read your message it hit me, without even looking at the code.

   This bug would mostly only affect the source backup, and only happen when recipients are not present in both input and source databases, so not likely when doing incremental merges unless new contacts are added, and the newer backup is used as source.

The bug should now be fixed (I will need to run further tests tomorrow though), though already deleted avatars are obviously not brought back. Let me know if you notice any improvement.

Thanks!

[hugogithubs]

Hi, sorry for late response and feedback, but I had no time to check earlier.

I started "from scratch" for my longterm backup file again to be sure nothing is lost. All avatar stickers are now included. For my type of use the Issue seems therefore to be solved.

Thanks a lot.

[bepaald]

sorry for late response and feedback, but I had no time to check earlier.

No problem of course.

All avatar stickers are now included. For my type of use the Issue seems therefore to be solved.

That's great to hear. I hope it stays that way.

Thank you for constantly finding these issues in the merge and htmlexport options!