search

ElDavoo / wa-crypt-tools

Manage WhatsApp .crypt12, .crypt14 and .crypt15 files.
GNU General Public License v3.0
631 stars 80 forks source link

"Invalid key" error while trying to restore re-encrypted database in WhatsApp #109

Closed kingbtcvl closed 7 months ago

kingbtcvl commented 9 months ago

Hey guys, I have the following problem concerning the encryption function:

I have and old database msgstore_old.db.crypt14 and the corresponding key file. Using the decryption function wadecrypt I can successfully obtain the database msgstore_old.db. Moreover I have my recent database msgstore_new.db.crypt15 which is end-to-end encrypted with a 64 characters key a69f... that I know. Again using wadecrypt with this key I can successfully obtain the database msgstore_new.db. Now I want to merge this two database. Using Julia and its SQLite package, I can execute a bunch of SQL commands which yields a merged database msgstore_merged.db. After achieving that I want to restore this merged database in WhatsApp, of course. So I use waencrypt with my key a69f... to get msgstore_merged.db.crypt15. I checked that everything is fine by decrypting this database with wadecrypt again as a little test and it worked, so I felt I was ready to restore this merged and encrypted database in WhatsApp.

Consequently I deinstalled WhatsApp completely, put my msgstore_merged.db.crypt15 as msgstore.db.crypt15 in newly created WhatsApp\Database folder and also copied the old WhatsApp\Media folder where I put the new media, too. Now I installed WhatsApp again, opened it, regestired wiht my phone number until I took me to the backup dialog. Since I am on Android WhatsApp wanted me to backup from my Google Account, but no problem. I closed WhatsApp, switched to airplane mode, opened WhatsApp again, waited a bit, skipped the error connection message and there we go, WhatsApp trying to find a local backup. My backup was recoginized by WhatsApp, too. So far, so good.

Now WhatsApp asks me to provide a password for encryption. I tell it that I have used a 64 character key instead in it leads me to the dialogue where I am finally supposed to put in my key. I typed that key a69f... in and it failed telling me it would be invalid :-( I tried that a second time, double and triple checking that there was no typo and after being absoltely sure I again failed and WhatsApp told me that the key I put in was wrong again. (I even made a third attempt decrypting msgstore_merged.dbwith a key 5555.... that consits of 64 times a 5, but this did not work either while encrypting it in WhatsApp).

So, it seems like altough I can successfully redecrypt my encrypted file msgstore_merged.db.crypt15 using the wadecryptfunction from this repo, WhatsApp still hates the encrypted file. I read through the issues in this repo and reading issue #9 I found out that WhatsApp obiously requires the correct header in the encrypted file to be able to read it successfully. This means that I see some light at the end of the tunnel. However I haven't figured out yet, how to get the correct header into my encrypted file. Due to the comments in #9 I gained some hope that the command waencrypt a69f... msgstore_merged.db msgstore_merged.db.crypt13 --reference msgstore_new.db.crypt13 --type 13 would do the job, but looking at the current source code I feel that the--reference option is just an empty dummy at the moment?! Am I correct in this assumption or did I simply miss the part of the code where this option effectively does something?

I mean I could simply try it and see what happens, of course, but I don't want to waste a try here because WhatsApp already started to complain about me regestering the same phone number to often in a row. Therefore I had to swicht from SMS verification to get call verfication already and I am a little afraid of getting banned permanently :D

So any clues on how to make my merged database get accepted by WhatsApp would be highly appreciated!

ElDavoo commented 9 months ago

Hi, As written here (I don't know if it printed), waencrypt is still a work in progress that needs to be completed. That's why: 1) #9 is open 2) Documentation has not been written (and what I say is difficult to understand) 3) The pip release (the one that people were supposed to get?) is not updated

Probably the feature list is incorrect? As you correctly guessed, you should copy it from the backup file you want to modify and re-encrypt it using the same feature list with the --reference that unfortunately as you guessed it's still not implemented.
Your case is also particular because you want to merge two DB files.
So to give a rough idea of what should be done:

1) Hack wadecrypt to let it print the feature table (this too will be done in the future), decrypt your old DB files and get their feature list.
2) Merge the lists (how? no idea? A logical OR should work?)
3) Use waencrypt with --feature-list (also --max-feature if necessary) to insert the table you got previously.

This is assuming the feature list is incorrect. It might also be: 1) The encryption process is wrong (bug) 2) The header format is wrong (bug) 3) You didn't set --wa-version and --jid 4) Your merged DB is wrong in some way (especially if the schema changed)

Because unfortunately the application won't tell you what exactly is wrong.

Unfortunately, waencrypt requires lots more parameters to make a working backup, so explaining and letting everything work becomes difficult both technically and from a user experience view. At this state it's a little more than a proof of concept as it's difficult to test (you need to open the app and re-register to observe the restore process...)

kingbtcvl commented 9 months ago

Thanks a lot for you detailed explanation, @ElDavoo

So to give a rough idea of what should be done:

  1. Hack wadecrypt to let it print the feature table (this too will be done in the future), decrypt your old DB files and get their feature list.
  2. Merge the lists (how? no idea? A logical OR should work?)
  3. Use waencrypt with --feature-list (also --max-feature if necessary) to insert the table you got previously.

I think I am able to try that and see what happens. The first point does not sound too difficult to achieve and the third one is easy when you already have the feature list. However, the second point worries me a little already. Altough using a logical OR does seem to be the natural choice at first glance, it might cause the backup process to fail again. From what I have figured out about the features so far, the older the database the less features exist because new features are added in newer versions of WhatsApp. This might suggest that old features might have already become obsolete and pernanently deactivated for newer backups. If that's the case then a logical OR might not work for those features. You could argue to simply take all new features, but that might also be wrong because the new database is very small compared to the old one. There are especially quite a few tables in the old database that contain data, which are completely empty in the new database. If any of the features refers to such a table, the feature is active in the old backup and not in the new one, but must be active in the merged backup. 🤔

Could you maybe elaborate a little more on what is known about the features? Where can I find more information about them to understand what they really represent as far as the databse is concerned. I mean, I see from backup_expiry.proto that there are 35 features (f_5 to f_39), which are all boolean values except for the unused backup size f_38. However, what I don't get is how we know that for example feautre 5 corresponds to the call log, whereas feature 18 corresponds to the message system. What I also don't understand exactly either is what it means when a feature is active compared to what it means when a featue is not active in a database.

This is assuming the feature list is incorrect. It might also be:

  1. The encryption process is wrong (bug)
  2. The header format is wrong (bug)
  3. You didn't set --wa-version and --jid
  4. Your merged DB is wrong in some way (especially if the schema changed)

Well, hopefully point one and two are not the case for my database (or anybody's database). But concerning point one, I am confident that the encryption process itself is fine, since I could decrypt the re-encrypted database successfully. Concerning point two, I believe that it is not too problematic because I saw that you and some other guy have already managed to restore a re-encrypted backup. Yes, concerning point three I haven't used that options so far, but I will for my next test since these information is obviously crucial for the header. And finally, concerning the last point four, I do think that even if the database was wrong in some way, WhatsApp will at least start to restore the database instead of telling me that the encryption key is invalid right from the beginning. So that is something I expect to figure out only after I made WhatsApp decrypt my re-encrypted database. However, I was quite carefully here and especially ensured that the shema of the database remained unchanged.

ElDavoo commented 9 months ago

Could you maybe elaborate a little more on what is known about the features?

Nothing except their names unfortunately.

However, what I don't get is how we know that for example feature 5 corresponds to the call log, whereas feature 18 corresponds to the message system.

They're taken from the official app, but I didn't study their effects on the DB files as it goes out of the scope of this project.

I believe that it is not too problematic because I saw that you and some other guy have already managed to restore a re-encrypted backup.

I only tested re-encrypting the exact same DB without modification, so going further is untested.

Yes, concerning point three I haven't used that options so far, but I will for my next test since these information is obviously crucial for the header.

Well, the jid option is mandatory, as it identifies the phone number linked to your account. If it doesn't match, WA should recognize it's not your backup and not restore it. The WA version could change without the restore failing (unless the version is more recent than your current app version (?).

For the other things you said, I know as much as you.

kingbtcvl commented 9 months ago

Could you maybe elaborate a little more on what is known about the features?

Nothing except their names unfortunately.

However, what I don't get is how we know that for example feature 5 corresponds to the call log, whereas feature 18 corresponds to the message system.

They're taken from the official app, but I didn't study their effects on the DB files as it goes out of the scope of this project.

Where exactly can one find those names? I would like to have a closer look at it because I am quite unsure whether or not I have all features now.

I mean I now added these lines of code to get the feature vector by inspecting all implemented features. The result is that my new database supports all of them and my old database supports all except for f_34, which is supposed to correspond to message_main_verification. But now, I wonder how to figure out if there aren't any more new features that have not been implemented here yet. I thought that I could kind of calculate that, finding out that in my case protobuf_size = 132. So the header consists of 132 bytes in total. But since I don't know the number of bytes of the unsued device_model string, I canot calculate how many of those 132 bytes must be left for the features, can I?

kingbtcvl commented 9 months ago

I only tested re-encrypting the exact same DB without modification, so going further is untested.

So, I am happy to help testing 😆

Well, the jid option is mandatory, as it identifies the phone number linked to your account. If it doesn't match, WA should recognize it's not your backup and not restore it. The WA version could change without the restore failing (unless the version is more recent than your current app version (?).

Actually I think, it's pretty strange that the jid option is mandatory. I mean, imagine you have a backup from on old phone and you bought a new one with a new number. Why should it be forbidden to load a backup. That does not make too much sense, since you would need the key anyway, assuming it is end to end encrypted.

ElDavoo commented 9 months ago

Where exactly can one find those names?

By decompiling the app with jadx and looking at the decompiled source code.

I thought that I could kind of calculate that, finding out that in my case protobuf_size = 132.

I don't think it's the right way to do that, I would change the proto message

kingbtcvl commented 9 months ago

By decompiling the app with jadx and looking at the decompiled source code.

Interesting. If I am unsuccessful with the current approach, I might have a look at that, too.

I thought that I could kind of calculate that, finding out that in my case protobuf_size = 132.

I don't think it's the right way to do that, I would change the proto message

Yeah, unfortuantely it seems that it is not that easy to sum up all the bytes that the header contains up to feature f_39 and then see if there are any bytes left for potentially new features. However, changing the proto message works great. I simply added one new optional test feature and then checked if this test feature had explicitlly given a new value. It turns out not, so I guess I can conclude that there are no new features right now.

So, I think I am close to start a new try for making WhatsApp restore my merged database. Just two little more things. First, I considered to create the head for msgstore_merged.db.crypt15 using the same IV that my msgstore_new.db.crypt15 contains, ie. I also want to apply the --iv option since I assume that this can only make things more likely to work, but not less likely. Would you agree here? Second, I figured out that with current version of the code the backup_version in the header will always have the default value of C.DEFAULT_BACKUP_VERSION = 0. I think there should be a new option --backup-verson for that because in my case for example the backup version turn out to be 1.

kingbtcvl commented 9 months ago

Ladies and gentlemen,

I can happily report that my test was successful and I have just made WhatsApp recoginize my re-encrypted merged (i.e. modified) database. 🥳 Hence I can confirm that setting the correct header for a crypt15 file (and probably also for a crypt14 file, but I did not test that) while encrypting with waencrypt leads to an encrypted file, which WhatsApp accepts. Therefore I would personally consider this issue as solved.

For those of you who want to follow my steps, here is what I did:

←[38;20mINFO - wadecrypt.py:222 : Header information in your crypt15 file:←[0m ←[38;20mINFO - wadecrypt.py:223 : IV: a902fb72341e7c7201f3dd2866a6a01e←[0m ←[38;20mINFO - wadecrypt.py:231 : Key type: 1←[0m ←[38;20mINFO - wadecrypt.py:232 : WhatsApp version: 2.24.3.81←[0m ←[38;20mINFO - wadecrypt.py:234 : The last two numbers of the user's Jid: 73←[0m ←[38;20mINFO - wadecrypt.py:235 : Backup version: 1←[0m ←[38;20mINFO - wadecrypt.py:238 : Features: [5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 39]←[0m ←[38;20mINFO - wadecrypt.py:239 : Max feature number: 39←[0m

@ElDavoo So as already said, the problem is solved now from my point of view and you can merge my PRs and/or close this issue if you like. That's up to you, of course. Anyway, thanks a lot for your support and this repo again!

ElDavoo commented 9 months ago

Well done! I will take a look at that after the 22nd

haare123 commented 8 months ago

Ladies and gentlemen,

I can happily report that my test was successful and I have just made WhatsApp recoginize my re-encrypted merged (i.e. modified) database. 🥳 Hence I can confirm that setting the correct header for a crypt15 file (and probably also for a crypt14 file, but I did not test that) while encrypting with waencrypt leads to an encrypted file, which WhatsApp accepts. Therefore I would personally consider this issue as solved.

For those of you who want to follow my steps, here is what I did:

  • I started with my merged database msgstore_merged.db, my latest WhatsApp Backup msgstore_new.db.crypt15 and the corresponding 64-byte-key 8a359fb873c488f521aee87e3003f27ab376b9799cc39a70142d23e67a90b15c
  • I used wadecrypt from the PR Added: header-info option #112 with the command wadecrypt 8a359fb873c488f521aee87e3003f27ab376b9799cc39a70142d23e67a90b15c msgstore_new.db.crypt15 msgstore_new.db --header-info to get the header information of my latest WhatsApp Backup. The output looked like this:

←[38;20mINFO - wadecrypt.py:222 : Header information in your crypt15 file:←[0m ←[38;20mINFO - wadecrypt.py:223 : IV: a902fb72341e7c7201f3dd2866a6a01e←[0m ←[38;20mINFO - wadecrypt.py:231 : Key type: 1←[0m ←[38;20mINFO - wadecrypt.py:232 : WhatsApp version: 2.24.3.81←[0m ←[38;20mINFO - wadecrypt.py:234 : The last two numbers of the user's Jid: 73←[0m ←[38;20mINFO - wadecrypt.py:235 : Backup version: 1←[0m ←[38;20mINFO - wadecrypt.py:238 : Features: [5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 39]←[0m ←[38;20mINFO - wadecrypt.py:239 : Max feature number: 39←[0m

  • Next I used waencrypt from the PR Added: backup-version option #111 with the command waencrypt 8a359fb873c488f521aee87e3003f27ab376b9799cc39a70142d23e67a90b15c msgstore_merged.db msgstore.db.crypt15 --iv a902fb72341e7c7201f3dd2866a6a01e --wa-version 2.24.3.81 --jid 73 --backup-version 1 --enable-features 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 39 --max-feature 39 to get my correctly encrypted database msgstore.db.crypt15.
  • Then after uninstalling WhatsApp completely and reinstalling it, I simply used this output msgstore.db.crypt15 to let WhatsApp restore this database as a local backup. (I only provided the the Database folder com.whatsapp/WhatsApp/DataBases containing only the encrypted msgstore.db.crypt15 file and the Media folder com.whatsapp/WhatsApp/Media with all my merged media, but nothing else)
  • And that's all. It worked like a charm and now my WhatsApp looks as if my merged database has always been there for ages. 😎

@ElDavoo So as already said, the problem is solved now from my point of view and you can merge my PRs and/or close this issue if you like. That's up to you, of course. Anyway, thanks a lot for your support and this repo again!

Hey there, this is my first post and im very new to "programming" anything at all. Yesterday i installed python+sql... My Problem is as follows: I have saved an entire "com.whatsapp" folder from my phone (1st March 2024) with a key file and a corresponding decrypted msgstore.db,without the added crypt12,14,15... (no idea if the key file was from a crypt12,14,15...). Now i had to redo my phone and i tried to sell whatsapp the msgstore.db file but without success. so i spend the entire yesterday evening trying to find a program to re-ecrypt my msgstore.db file with the key until i stumbled upon this project here (i tried the "whapa"-project aswell and whatsapp can "read" the backup file but gets stuck on "preparing" -.-)

Then after all the installing and figuring out how to actually type stuff in cmd i managed to test my abilites with a newly created backupfile from whatsapp. (got backup from com.whatsapp\Whatsapp\Databases and noted the 64-digit keyfile in a textfile.) It successfully decrypted and i could open the file via SQlite (no idea if you can see the header-info there :/ ) Then i saw your post here and tried the stuff with the --header-info, after downloading all of your changed files and replacing the originals in the corresponding wa-crypt-tools folders with yours. Sadly this happened: wadecrypt: error: unrecognized arguments: --header-info

do i have to somehow reinstall the wa-crypt-tools with your changed stuff and if so, how? :D I have the same problem with your changed "waencrypt" command and all of the headerinfo you put in there, doesnt work here aswell :/

Also is the step with the merging msgstore.db files necessary to "convert" my old msgstore.db into a newer version of ".db" so it can correctly apply the crypt15 thing?

I am very sorry if you read this and have to facepalm at the usage of my "progamming terms" and also at my english (i'm german)

I look forward to a response, TY

kingbtcvl commented 8 months ago

I am very sorry if you read this and have to facepalm at the usage of my "progamming terms" and also at my english (i'm german)

No worries, we all have different skill levels and I guess I can understand you. Moreover your english is pretty decent.

I have saved an entire "com.whatsapp" folder from my phone (1st March 2024) with a key file and a corresponding decrypted msgstore.db,without the added crypt12,14,15... (no idea if the key file was from a crypt12,14,15...).

Well, if you have a copy of the entire "com.whatsapp" folder, then you also have your encrypted database msgstore.db.crypt14, don't you? I mean you should find your encrtypted database in the subfolder "com.whatsapp/WhatsApp/DataBases". Because if that's the case, why would you bother with re-encrypting your decrypted msgstore.db at all instead of simply letting WhatsApp restore from your msgstore.db.crypt14 file? So I would reccomend to check this out first.

It successfully decrypted and i could open the file via SQlite (no idea if you can see the header-info there :/ )

I would assume that you can't see the header infos here because the header is technically no part of the actual database.

Then i saw your post here and tried the stuff with the --header-info, after downloading all of your changed files and replacing the originals in the corresponding wa-crypt-tools folders with yours. Sadly this happened: wadecrypt: error: unrecognized arguments: --header-info

do i have to somehow reinstall the wa-crypt-tools with your changed stuff and if so, how? :D

Yes, just replacing the files won't help, You need to rebuild the executable files wadecrypt and waencrypt. I would recommend you to uninstall wacrypt-tools completely via python -m pip uninstall wa-crypt-tools and then reinstall the version from the PR #112 via python -m pip install . (after navigating to the folder where you saved the PR repository). No need to manually download the files separately and replacing the originals ones by the way. Just go to the repository of the PR (https://github.com/kingbtcvl/wa-crypt-tools/tree/headerinfo) and download the whole thing there. Good luck!

haare123 commented 8 months ago

Hey, thank you first of all for your fast reply! <3

Well, if you have a copy of the entire "com.whatsapp" folder, then you also have your encrypted database msgstore.db.crypt14, don't you? I mean you should find your encrtypted database in the subfolder "com.whatsapp/WhatsApp/DataBases". Because if that's the case, why would you bother with re-encrypting your decrypted msgstore.db at all instead of simply letting WhatsApp restore from your msgstore.db.crypt14 file? So I would reccomend to check this out first.

I only have the com.whatsapp from i guess was the install directory. there is no .crypt file anywhere to be seen, since i did not make a "backup" via whatsapp apparently :/ the Media Folder by itself is empty now ...

It successfully decrypted and i could open the file via SQlite (no idea if you can see the header-info there :/ )

I would assume that you can't see the header infos here because the header is technically no part of the actual database.

Then i saw your post here and tried the stuff with the --header-info, after downloading all of your changed files and replacing the originals in the corresponding wa-crypt-tools folders with yours. Sadly this happened: wadecrypt: error: unrecognized arguments: --header-info do i have to somehow reinstall the wa-crypt-tools with your changed stuff and if so, how?

Yes, just replacing the files won't help, You need to rebuild the executable files wadecrypt and waencrypt. I would recommend you to uninstall wacrypt-tools completely via python -m pip uninstall wa-crypt-tools and then reinstall the version from the PR #112 via python -m pip install . (after navigating to the folder where you saved the PR repository). No need to manually download the files separately and replacing the originals ones by the way. Just go to the repository of the PR (https://github.com/kingbtcvl/wa-crypt-tools/tree/headerinfo) and download the whole thing there. Good luck!

so i successfully re-installed this package with the PR#112 update and the decrypted --header-info worked out well. now i got my old msgstore.db with the key file and the header from the "new" backup from whatsapp .crypt15 and the correspondeing 64-key.

now all i have to do is this "merging" part you were talking about, so i convert my msgstore.db into the new .db from to encrypt it with the 64-key or does the old msgstore.db work with the 64 key by itself?

Thanks in Advance <3

haare123 commented 8 months ago

Edit: i tried to recrypt my old msgstore.db with the 64 key and the header info i received from the new backup and i received this error message:

waencrypt: error: unrecognized arguments: --backup-version 1

what to do? :/

kingbtcvl commented 8 months ago

so i successfully re-installed this package with the PR#112 update and the decrypted --header-info worked out well. now i got my old msgstore.db with the key file and the header from the "new" backup from whatsapp .crypt15 and the correspondeing 64-key.

Great, the sounds good!

now all i have to do is this "merging" part you were talking about, so i convert my msgstore.db into the new .db from to encrypt it with the 64-key or does the old msgstore.db work with the 64 key by itself?

Well no, actually you don't really want to do the merging process as far as I can see. I did that because I wanted to combine two databases to get a merged database that holds all the data of the two single ones that I had. But you only have one database msgstore.db that already holds all data you want to have and you want to re-encrypt it and then restore that one in WhatsApp. Right? So essentially what you want to do is basically something like waencrypt 8a359fb873c488f521aee87e3003f27ab376b9799cc39a70142d23e67a90b15c msgstore.db msgstore.db.crypt15 --iv a902fb72341e7c7201f3dd2866a6a01e --wa-version 2.24.3.81 --jid 73 --backup-version 1 --enable-features 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 39 --max-feature 39 of course, using not these parameters, but those ones you just got from reading your header infos.

i tried to recrypt my old msgstore.db with the 64 key and the header info i received from the new backup and i received this error message: `` waencrypt: error: unrecognized arguments: --backup-version 1

what to do? :/

Well, you are very close, bro. For using the waencrypt with the --backup-version option you must uninstall your current wacrypt-tools again and use the one from PR #111 (i.e. the one from the repo https://github.com/kingbtcvl/wa-crypt-tools/tree/backupversion). That's because I created two different pull requests.

haare123 commented 8 months ago

Hey, thanks for your kind words and im really proud of myself already hehe

Well, you are very close, bro. For using the waencrypt with the --backup-version option you must uninstall your current wacrypt-tools again and use the one from PR #111 (i.e. the one from the repo https://github.com/kingbtcvl/wa-crypt-tools/tree/backupversion). That's because I created two different pull requests.

ok, can u please show me step by step how to properly uninstall + install this pull request, because i cant seem to get it working :(

C:>git clone https://github.com/kingbtcvl/wa-crypt-tools/tree/headerinfo Cloning into 'headerinfo'... fatal: repository 'https://github.com/kingbtcvl/wa-crypt-tools/tree/headerinfo/' not found

C:>git clone https://github.com/kingbtcvl/wa-crypt-tools/tree/headerinfo.git Cloning into 'headerinfo'... fatal: repository 'https://github.com/kingbtcvl/wa-crypt-tools/tree/headerinfo.git/' not found

this is the stuff i get from cd...

kingbtcvl commented 8 months ago

ok, can u please show me step by step how to properly uninstall + install this pull request, because i cant seem to get it working :(

Should be exactly the same as before when you successfully uninstalled the original version and installed the one which allowed you to get the header-info

haare123 commented 8 months ago

ok, i figured it out somehow, sorry if i ask weird stuff because i dont have the knowledge and i dont write everything down what i do. It is alot to know/do which for a non-programmer just looks really funky haha

when i enter the waencrypt command with all the proper header info, i get this:

...... 32 33 34 35 36 37 39 --max-feature 39 ←[33;20mWARNING - waencrypt.py:57 : This is still a work in progress, that will be completed in the future.←[0m ←[38;20mINFO - waencrypt.py:86 : Done!←[0m

is there something wrong here or should i try to get this backup working on my phone now? and do i need to remover end-to-end decryption first on my whatsapp application or do i have to keep it enabled with the 64-key i have?

kingbtcvl commented 8 months ago

...... 32 33 34 35 36 37 39 --max-feature 39 ←[33;20mWARNING - waencrypt.py:57 : This is still a work in progress, that will be completed in the future.←[0m ←[38;20mINFO - waencrypt.py:86 : Done!←[0m

That's exactlly the output you want to have, so you can go with this re-encrypted backup and try to restore it in WhatsApp.

and do i need to remover end-to-end decryption first on my whatsapp application or do i have to keep it enabled with the 64-key i have?

Well, actually you need to completely uninstall WhatsApp I guess. At least that is how it worked for me. I did the following steps on Android to make WhatsApp read the re-encrypted database:

haare123 commented 8 months ago

thanks alot!

i will have to try that this evening unfortunately, because i have reinstalled whatsapp too much apparently, so i cant receive SMS verification code and the calling thing does not work (the computervoice telling me to press 4 and even if i press it in the call-numberpad it does not do anything and the call ends after 20 seconds -.-) all in all pretty poor company haha

haare123 commented 8 months ago

Capture You are my Life-Saver!!! thank you very much!

Well, actually you need to completely uninstall WhatsApp I guess. At least that is how it worked for me. I did the following steps on Android to make WhatsApp read the re-encrypted database:

  • I went to settings, then apps, then WhatsApp, then storrage. I cleared the cache there and delted the data.
  • Next I uninstalled WhatsApp
  • After uninstalling I made sure that there was no WhatsApp folder on my phone (this is found under Android/media/com.whatsapp, but was gone after uninstalling WhatsApp, exactly as I wanted it to.
  • Then I created this folder again and also the subfolders Android/media/com.whatsapp/WhatsApp/Databases and Android/media/com.whatsapp/WhatsApp/Media
  • Next I put the re-encrypted databse into the Databases folder and all my media in the Meida folder
  • Only after that I installed WhatsApp from the App Store again and started it

Right here i also want to add something, which worked for me, but not sure if needed. described in this reddit post: https://www.reddit.com/r/whatsapp/comments/11oiwse/working_as_of_march_2023_method_for_restoring_a/?sort=new

2) Clear the Google Play Cache and Google Cache:

Settings >Apps>see all apps>Navigate to "google Play Services">open>Force Stop>clear Cache>Clear Data.

Settings >Apps>see all apps>Navigate to "google">open>Force Stop>clear Cache>Clear Data. (Note, this step will also clear stuff like Wallet data)

This step is important as far as i can tell, because it will disconnect your phone from google drive backup system. BUT just to be sure i also went in my google drive and then navigated to the "wheel in top right corner"(maybe called Settings aswell)>settings>then on the left side Manage Apps>then find whatsapp backup in the list and click on options and remove backup services.

and i also found several reddit users mentioning thisfollowing step:

4) Allow WhatsApp to access storage before opening WhatsApp for the first time.

Settings >Apps>see all apps>Navigate to and tap "WhatsApp">Permissions>Allow access to ALL PERMISSIONS (you can deactivate permissions when everything works out well ;) )

  • I registered my number and WhatsApp found my local backup which I could restore by typing in my 64-byte key (I guess WhatsApp asked me whether I want to restore the local backup from a password or key before it took me to the screen where I could provide my key)

same for me, it showed a backup was found, then the option to restore, which i clicked obviously (fingers shaking...) right after it asked me for my password, but on the very bottom side of the screen it said something like "use 64-digit key instead". typed it in, then it worked :)

I am so glad that i finally managed to do this and really hope in the future to not have to do it again. But atleast in that case i know where to find helpful people like @kingbtcvl

In Case any other Person like myself (total noob in programm or any related stuff) finds this post, i believe in you and if you put effort in it, i am sure you can make it work!

Good Luck to all of you!

kingbtcvl commented 8 months ago

Nice to see, that you made it! Congratulations, @haare123

Sarmingsteiner commented 8 months ago

We are trying go get our existing WA backup to a new phone number, since the old phone number is not accessible by WA anymore. So we chose a new phone number ending with the same 2 digits (JID) as the original number. The last 2 digits of the JID were verified as described above using --header-info. Both the original encrypted (version 14) backup as well as the decrypted and re-encrypted backup were found by WA. However, restoration failed with both backups without any specific error message or progress bar. Any idea or help is highly appreciated!

kingbtcvl commented 8 months ago

🤔 That comes unexpected, but restoring a re-ecrypted crypt14 database is untested terretority. It might be that this does not work at all, does not work on a new phones or new number or does not work in your specific case. So first, could you check if the header info of your re-encrypted crypt14 file is eaxactly the same (since your two phone numbers have the same last 2 didigits) as the header info of your original crypt14 by simply re-decrypting the re-encrypted file with --header-info. If that's not the case, then something went wrong during the re-encryption process.

We are trying go get our existing WA backup to a new phone number, since the old phone number is not accessible by WA anymore.

Could you elaborate on that? What exactly do you mean when you say the old phone number is not accessible by WA anymore? Was it officially blocked by WA? Did you officially switch your phone number in WA before the old number got unacessible? Probably not, right? My intuition tells me that this is the root of your problem...

Sarmingsteiner commented 8 months ago

Many thanks @kingbtcvl, 1) Yes, the --header-info output shows exactly the same metadata, including "Server salt" and "Google ID". 2) Maybe it makes a difference that it is a WA Business Account (and file) we are trying to restore. The WA Business account got locked for a strange reason and was "downgraded" to a "normal" WA account when it was unlocked a few minutes after making a request. So you are right, the "core" problem is, that our "old" account cannot be accessed neither by the WA Business app (it always says "You are using an unofficial WA app", nor by the "normal" WA app (it is a fixed landline number which is only supported by WA Business app) anymore. 3) We do have an older database backup still encrypted with crypt15 (on an old phone). We were also able do decrypt it using wadecrypt and the key file called encrypted_backup.key (instead of key which worked for crypt14). Both key files cannot be read as plain text in an editor such as notepad++. 4) The encrypted.15 database is also recognized as restorable by WA Business app. However, when entering the (definitely correct) password prompted in WA Business UI, the app states a message similar to "No account information found, backup cannot be decrypted by password, use restore encryption key instead". However, we never created (consciously) a restore encryption key (64 bit) and it seems its creation cannot be triggered on the old phone anymore (when trying to disable end-to-end-encryption on the old phone, the only option offered is to enter the (correct) password, but not to enter a restore encryption key).

kingbtcvl commented 8 months ago

Ok, that sounds all quite complicated and indeed it could be that there is some kind of difference concerning a normal WA backup and a business account WA backup. But then again, the difference does not necessarily need to be written in the header, but might be written into the acutal database itself. I don't know. However, what I would essentially try here is decrypting the old crypt14 file from the WA business account and re-encrypting it as a crypt15 file to the new normal WA account:

Sarmingsteiner commented 8 months ago

Many thanks @kingbtcvl for you fast reply!

I have been trying all day to download the missing media files from Google Drive Backup - finally successfully: https://github.com/YuriCosta/WhatsApp-GD-Extractor-Multithread/issues/47#issuecomment-1986946778

So with all the latest data, I will try your suggestion to "convert" the crypt14 database in a crypt15 database (hopefully next week).

haare123 commented 8 months ago

Ok, that sounds all quite complicated and indeed it could be that there is some kind of difference concerning a normal WA backup and a business account WA backup. But then again, the difference does not necessarily need to be written in the header, but might be written into the acutal database itself. I don't know. However, what I would essentially try here is decrypting the old crypt14 file from the WA business account and re-encrypting it as a crypt15 file to the new normal WA account:

  • on your new phone, completely deinstall WA. Make sure to delete all app data and the app cache first
  • reinstall WA normally without trying to restore any backup, just a clean, fresh and normal WA on your phone
  • activate end to end encryption in WA, write a few messages and then create a backup in WA.
  • You should have a crypt15 file now as well as a 64-byte-key (write it down, because it's gonna be essential if everything works out)
  • read the header info of that crypt15 file by decrypting it using the 64-byte-key you wrote down
  • decrypt your old crypt14 file with the old key file you have to get an encrypted database containing the messages and data you want to restore
  • now re-encrypt this database as a crypt15 file using the 64-byte-key and the header info you got from your clean WA installation
  • then deinstall WA completely again and try to restore this re-encrypted crypt15 file

This is in essence what i did some posts above hehe, just with 1 phone (and obviously same number). I wonder if this works out in the end, because of the similarity of the phone numbers ( last digits same). Because if you just get unlucky with that, you are essentially screwed :(

kingbtcvl commented 8 months ago

Edit: I thought the last answer was from @Sarmingsteiner, too, so parts of my answer did not make sense and I therefore delted them and edited the rest.

I wonder if this works out in the end, because of the similarity of the phone numbers ( last digits same).

My guess is that the smilarity of the phone numbers i neither advantageous nor disadvantageous. From WA point of view @Sarmingsteiner's new phone number was respectively is not related to any WA Business account and especially not to the old WA business account hist old number was related to. Hence WA sees that as the normal number of a normal person. Therefore my guess is that if he gets the header info of a crypt15 file of clean fresh WA installation with that new number he kind of has a clean header. If he copies that clean header to his old decrypted database by "converting" it from a crypt14 to a crypt15 then he might be able to pretend to always have had a clean backup. At least that would be my hope here.

Because if you just get unlucky with that, you are essentially screwed :(

What do you mean that? I don't see how he could get screwed irreversiblely here. I mean I have no idea if he gets lucky and this will work out for a database that orginally comes from a businees acount, but the state he has right now can't get worse, can it?

Sarmingsteiner commented 8 months ago

Many thanks again @kingbtcvl , your solution worked and we were able to restore our chats on the phone! Here are the steps for anyone facing similar issues:

kingbtcvl commented 8 months ago

Many thanks again @kingbtcvl , your solution worked and we were able to restore our chats on the phone!

Nice to see yet another person successfully restoring their old WA chats! Congratulations!

ElDavoo commented 7 months ago

Many thanks again @kingbtcvl , your solution worked and we were able to restore our chats on the phone!

That's strange, I think the user's phone number is stored multiple times / in many ways in the actual database file, so even if changed in the header it would cause all sorts of problem if not treated correctly.
Anyway, I didn't do any test with WA business, so I'm glad everything is working fine.

ElDavoo commented 7 months ago

As #9 is the reference issue for encryption and we only discussed on something not really technical i'm converting this issue to a discussion.
If OP is happy they can then close it

ElDavoo commented 7 months ago

anyway now the header is printed in verbose