i keep getting consistency error

[Rainy-Window]

Hello, i've 4 databases that i was able to convert their schemas to the same schema of WhatsApp version (with very minor difference in tables & rows)

but when i try to run this program in binary, it keeps failing.

May you please provide a way or method for us to update the binary file to keep it updated to the latest schema provided in the app? i've very basic knowledge of SQLite databases but i've managed to export database schema and notice difference in schema and different table it will be of great help as the script and binary file needs to be updates and bring more issues every time WhatsApp updates. so it will be good for us to update it our needs. thank you a lot @natario1 @MrBarbie please help if you could.

this is the error i get when i try to merge all of the 4 databases at once. (i've filtered out lines that has no consistency error):

Details

[*] Input databases: - input/msgstore1.db - input/msgstore2.db - input/msgstore3.db - input/msgstore4.db [*] Checking consistency of input/msgstore1.db Warning: Database input/msgstore1.db is not consistent. Column chat.last_important_message_row_id references table message, where we expected to find a list of 1 entries, but only 0 were found. Total count (source): 221617, total count (target): 675 Missing=[1] Warning: Database input/msgstore1.db is not consistent. Column chat.change_number_notified_message_row_id references table message, where we expected to find a list of 4 entries, but only 3 were found. Total count (source): 221617, total count (target): 675 Missing=[1] Warning: Database input/msgstore1.db is not consistent. Column message.chat_row_id references table chat, where we expected to find a list of 594 entries, but only 593 were found. Total count (source): 675, total count (target): 221617 Missing=[435] Warning: Database input/msgstore1.db is not consistent. Column message_media.message_row_id references table message, where we expected to find a list of 13092 entries, but only 13043 were found. Total count (source): 221617, total count (target): 13092 Missing=[158422, 162576, 163874, 165349, 165351, 167591, 168793, 169064, 171436, 172427, 173128, 173432, 173435, 173775, 175538, 178296, 178304, 178528, 187036, 189588, 189854, 190203, 190350, 191425, 191839, 191840, 191844, 192005, 192007, 195417, 195418, 198920, 198922, 198923, 200273, 200274, 200338, 200339, 203245, 204262, 211003, 212967, 213947, 215571, 223007, 223310, 233684, 257996, 259504] ------------------------------------------------------------------------------ [*] Checking consistency of input/msgstore2.db Warning: Database input/msgstore2.db is not consistent. Column message.chat_row_id references table chat, where we expected to find a list of 51 entries, but only 50 were found. Total count (source): 53, total count (target): 10317 Missing=[-1] Warning: Database input/msgstore2.db is not consistent. Column message_media.message_row_id references table message, where we expected to find a list of 472 entries, but only 465 were found. Total count (source): 10317, total count (target): 472 Missing=[402, 1097, 2187, 8031, 8032, 8033, 10653] ------------------------------------------------------------------------------- [*] Checking consistency of input/msgstore3.db Warning: Database input/msgstore3.db is not consistent. Column chat.last_read_message_row_id references table message, where we expected to find a list of 70 entries, but only 69 were found. Total count (source): 57612, total count (target): 235 Missing=[50141] Warning: Database input/msgstore3.db is not consistent. Column chat.last_read_receipt_sent_message_row_id references table message, where we expected to find a list of 70 entries, but only 69 were found. Total count (source): 57612, total count (target): 235 Missing=[14] Warning: Database input/msgstore3.db is not consistent. Column chat.last_read_message_sort_id references table message, where we expected to find a list of 53 entries, but only 52 were found. Total count (source): 57612, total count (target): 235 Missing=[50141] Warning: Database input/msgstore3.db is not consistent. Column message.chat_row_id references table chat, where we expected to find a list of 123 entries, but only 122 were found. Total count (source): 235, total count (target): 57612 Missing=[-1] Warning: Database input/msgstore3.db is not consistent. Column message_vcard_jid.vcard_row_id references table message_vcard, where we expected to find a list of 3 entries, but only 2 were found. Total count (source): 2, total count (target): 4 Missing=[-1] Warning: Database input/msgstore3.db is not consistent. Column message_add_on.sender_jid_row_id references table jid, where we expected to find a list of 11 entries, but only 10 were found. Total count (source): 1267, total count (target): 226 Missing=[-1] Warning: Database input/msgstore3.db is not consistent. Column message_media.message_row_id references table message, where we expected to find a list of 3683 entries, but only 3639 were found. Total count (source): 57612, total count (target): 3683 Missing=[1002, 1792, 1889, 10542, 10693, 12784, 12787, 27335, 27588, 29974, 33087, 33088, 33332, 33348, 33416, 33419, 33420, 33421, 34719, 36360, 38277, 39298, 39534, 39948, 40149, 40379, 42412, 44654, 45568, 48736, 49250, 49626, 49658, 50002, 50315, 51649, 54862, 56191, 58540, 59450, 59604, 59611, 63079, 64171] ------------------------------------------------------------ [*] Checking consistency of input/msgstore4.db Warning: Database input/msgstore4.db is not consistent. Column chat.last_read_message_row_id references table message, where we expected to find a list of 81 entries, but only 80 were found. Total count (source): 61576, total count (target): 383 Missing=[63844] Warning: Database input/msgstore4.db is not consistent. Column chat.last_read_receipt_sent_message_row_id references table message, where we expected to find a list of 82 entries, but only 77 were found. Total count (source): 61576, total count (target): 383 Missing=[57046, 51870, 35056, 68089, 63046] Warning: Database input/msgstore4.db is not consistent. Column chat.last_message_reaction_row_id references table message, where we expected to find a list of 59 entries, but only 44 were found. Total count (source): 61576, total count (target): 383 Missing=[2669, 3749, 566, 3163, 80, 3499, 2227, 3488, 3067, 3431, 2532, 3024, 1563, 3262, 3741] Warning: Database input/msgstore4.db is not consistent. Column chat.last_seen_message_reaction_row_id references table message, where we expected to find a list of 59 entries, but only 44 were found. Total count (source): 61576, total count (target): 383 Missing=[2669, 3749, 566, 3163, 80, 3499, 2227, 3488, 3067, 3431, 2532, 3024, 1563, 3262, 3741] Warning: Database input/msgstore4.db is not consistent. Column chat.last_read_message_sort_id references table message, where we expected to find a list of 81 entries, but only 79 were found. Total count (source): 61576, total count (target): 383 Missing=[63844, -9223372036854775808] Warning: Database input/msgstore4.db is not consistent. Column chat.last_read_receipt_sent_message_sort_id references table message, where we expected to find a list of 82 entries, but only 77 were found. Total count (source): 61576, total count (target): 383 Missing=[57046, 51870, 35056, 68089, 63046] Warning: Database input/msgstore4.db is not consistent. Column message.chat_row_id references table chat, where we expected to find a list of 363 entries, but only 362 were found. Total count (source): 383, total count (target): 61576 Missing=[-1] Exception in thread "main" java.lang.IllegalStateException: Database input/msgstore4.db is not consistent. Column message_thumbnail.message_row_id references table message, where we expected to find a list of 1893 entries, but only 1888 were found. Total count (source): 61576, total count (target): 1893 Missing=[25366, 30891, 33486, 40834, 56080] at dev.natario.ConsistencyKt.verifyConsistency(Consistency.kt:21) at dev.natario.ConsistencyKt.verifyConsistency$default(Consistency.kt:3) at dev.natario.MainKt.main(Main.kt:50)

[Rainy-Window]

@natario1 @MrBarbie May you please help me solve my issues with merging? My latest WhatsApp complains about that i MUST update the app in 15 days, and i'm afraid that i'll have to go with every database file again and use it with the newest WhatsApp to make them prepared for the newest schema for merging.

if you don't have much time, may you please give me a way or instruct me how to update this tool to meet newer schema requirements? thank you

[MrBarbie]

For how to compile your custom version and also how to adjust the schema please refer to #5 . I had some discussions with the original author there to get it more explained how various parts would work and what needs to be adjusted.

Just to be sure, you are merging the oldest ones together first? Cause what those errors are telling you is that there are some ids that it cant find. Those ids should be there cause something is refering to it but it is not in your tables. Basically like I explained in #9 if you think all those missing things are fine you can turn off the consistency checks by adding ignoreConsistencyChecks = true to each of those that are failing https://github.com/natario1/whatsapp-database-merger/issues/9#issuecomment-1432719716

Also for any future person visiting here seems like the schema has changed quite a bit since #5 which maybe causing more things to fail and be incomplete so it will take a lot more work to recheck the new schema. TL:DR dont hold your breath for an update

[Rainy-Window]

thank you for your reply @MrBarbie

i've read the issue thoroughly multiple times. and i honestly didn't understand most of it. i've no knowledge of this. but i may start to learn about Kotlin and basic editing for these scripts..

Just to be sure, you are merging the oldest ones together first?

No i was trying to combine all of them at once, do you recommend merging the oldest 3 first then with the newest with the result?

if nothing worked and everything is overwhelmed, i will do ignore consistency errors. and see what happens.

[Rainy-Window]

Hello @MrBarbie, how are you? i hope you're doing well. i was able to merge the 4 databases successfully after removing any consistency error I face for each of the database, but there's something not right and i hope you can help me with. or guide me what to do about it. the chats in the oldest database are all present in the final database but newer chats (eg a new number chat) in the 2nd, 3rd, or the last databases don't travel across to the final result database. for example: 1st database: john1 - mark1 - Julia1 2nd database: Jeremy - Malik - Julia2 3rd database: mark2 - Jane 4th database: John2 - Jack - Julia3 so after removing consistency error entries and merge is succesful: RESULT databse will have the following chats: john1+2 mark1+2 julia 1+2+3

only new chats that has already been spoken to in the Base Database (1) will be merged.. but others don't appear.

may you give me instructions or hints how to solve this issue please

[Marvolo1590]

Also for any future person visiting here seems like the schema has changed quite a bit since #5 which maybe causing more things to fail and be incomplete so it will take a lot more work to recheck the new schema. TL:DR dont hold your breath for an update

Hello @MrBarbie and @natario1

Right now it's May 2024 and like many others, I'm desperately trying to get different WA databased merged. I've been researching for weeks. And I did find information on how to do it right within DB Browser for SQlite by just adding the missing chat lines to the old database which doesn't contain the new ones.

However, my gut feeling tells me this would be far too simple, especially since all those many tables seem to be interacting and connected with each other. And I don't want to risk anything. I'd like to do it right.

Sadly, I hardly know anything about databases and coding, so I've been looking for detailed help. I'd even be willing to pay someone to do it (as some sort of freelance job).

Would someone of you be able to help me with this in May 2024 (assuming the databases and schemas have changed by now)? Thank you very much!