Closed Takepy closed 1 year ago
What is the WhatsApp version and which platform (Android/iOS) are you using?
I'm using Android. WhatsApp versions that have been used are 2.22.3.77 and 2.22.5.7 (Beta) 2.22.3.77 was running on an emulator so that I could pull the /com.whatsapp/ directory. On the 2.22.5.7 version I used the previously pulled key to decrypt the db.
Are both versions and environments result in same database schema?
Yes, they both resulted in the same database schema
Interesting. I will try to reproduce it with 2.22.3.77 on emulator.
I tried to login WhatsApp with 2.22.3.77 on emulator. However, the database includes the "messages" table. Do you remember when you installed WhatsApp on the emulator for the first time (i.e. creation date of the database)?
I installed WhatsApp on the emulator on the 9th of this month and then imported my chat history using the chat backup in WhatsApp. On my actual phone, WhatsApp has been installed like 6 months ago.
I checked some similar projects and suspect that "message" table is a legacy table for storing messages. On your phone, is the WhatsApp conversation restored from old backup after installation?
I restored the WhatsApp conversations from Google Drive (doing weekly backups). Never deleted my WhatsApp chat history though so the messages go all the way back to 2013
As far as I observed, it is possible that "message" was used in the early years of WhatsApp. Could you check if new installation without backup will result the same situation?
I'll give it a try later and report back.
Sorry, it took a little bit longer until I had time to try it out.
Setup a new emulator, installed WhatsApp and didn't import my backup.
The table messages
exists now.
So I'm guessing my backup is just too old for it to work then? :(
Sorry, it took a little bit longer until I had time to try it out.
No problem!
So I'm guessing my backup is just too old for it to work then? :(
I think so. I would like to add support for older version. However, I don't have such database, hence, I am not able to do it. If you have time and you want the script to work with your database, you can take a look inside the database and fine tune the script. Anyway, I will keep my eyes on the opportunity for supporting old databases. Thanks for your report!
Hey everyone! I just stumbled upon the same issue with WhatsApp version 2.22.7.73 on Android on two different devices using two different phone numbers without restoring any backup.
I'll try to reinstall WhatsApp and try it again, but it seems like the database scheme is changing. The new messages table has the following schema:
CREATE TABLE message (
_id INTEGER PRIMARY KEY AUTOINCREMENT,
chat_row_id INTEGER NOT NULL, from_me INTEGER NOT NULL,
key_id TEXT NOT NULL, sender_jid_row_id INTEGER,
status INTEGER, broadcast INTEGER,
recipient_count INTEGER, participant_hash TEXT,
origination_flags INTEGER, origin INTEGER,
timestamp INTEGER, received_timestamp INTEGER,
receipt_server_timestamp INTEGER,
message_type INTEGER, text_data TEXT,
starred INTEGER, lookup_tables INTEGER,
message_add_on_flags INTEGER, sort_id INTEGER NOT NULL DEFAULT 0
);
Edit: Tried a second time, database scheme is still the same.
Edit2: In WhatsApp version 2.21.21.17 both tables messages
and message
do seem to exist.
I'll try to reinstall WhatsApp and try it again, but it seems like the database scheme is changing. The new messages table has the following schema:
I can produce the same result on 2.22.7.73 and "messages" table no longer exist.
@Takepy As I can reproduce your situation now, I will work on supporting "message" table.
I am curious about how WhatsApp decides which table to store messages π€¨. Is "message" a legacy table? Or "messages" is the legacy one?
I am curious about how WhatsApp decides which table to store messages π€¨. Is "message" a legacy table? Or "messages" is the legacy one?
I'm not certain, using the outdated WhatsApp version both tables exist, in the more recent version (downloaded from whatsapp.com yesterday) only the message
table existed.
Edit: Just checked, in my current setup (outdated version) messages are only stored in the messages
table (which works awesome with your script!) and the message
table appears to be empty.
WhatsApp using "message" table spread different types of message to many other tables. Good chance for me to rewrite the project :)
@Takepy were you able to find a way around this issue? Looks like I'm in the same boat, unfortunately and my whatsapp database is using the "legacy" format as well. I've got messages dating back all the way to 2014.
@karanrajpal14 I came across the same issues as well, got a new phone and couldnt export the messages. Did you have any luck getting it to work?
I've started on a c# project to extract the messages which is working to extract and dump to text, just working on the html part now
Has anyone made some progress here - it now seems as if it has to do with new versions of whatsapp? Or does it only affect people with a very long message history? I am very ready to back everything up and try moving to a full puppeteered whatsapp with mautrix-whatsapp, but things seem to still be shifting way too much.
@KnugiHK any way we can help?
Has anyone made some progress here - it now seems as if it has to do with new versions of whatsapp? Or does it only affect people with a very long message history? I am very ready to back everything up and try moving to a full puppeteered whatsapp with mautrix-whatsapp, but things seem to still be shifting way too much.
@KnugiHK any way we can help?
The hardest part of this project is to understand the structure of the WhatsApp database. Any clues on the structure and pull requests are welcome.
I played with it a bit yesterday and replaced the main query in extract.py with this:
SELECT message.sender_jid_row_id as key_remote_jid,
message._id,
message.from_me as key_from_me,
message.timestamp,
message.text_data as data,
message.status,
message_future.version as edit_version,
message_thumbnail.thumbnail as thumb_image,
message_media.file_path as remote_resource,
message_media.mime_type as media_wa_type,
message_location.latitude,
message_location.longitude,
message_quoted.key_id as quoted,
message.key_id,
message_quoted.text_data as quoted_data,
message_media.media_caption
FROM message
LEFT JOIN message_quoted
ON message_quoted.message_row_id = message._id
LEFT JOIN message_location
ON message_location.message_row_id = message._id
LEFT JOIN message_media
ON message_media.message_row_id = message._id
LEFT JOIN message_thumbnail
ON message_thumbnail.message_row_id = message._id
LEFT JOIN message_future
ON message_future.message_row_id = message._id
WHERE key_remote_jid <> '-1';
That brings me past the missing messages table, but will clash in extract.py:240 trying to match the remote id to something in the big data dictionary. So, I assume me trying to replace key_remote_jid with sender_jid_row_id might go into the right direction, but might need some more joints to actually turn things into a phone number.
There are also a messages_view and a chats_view now that seem to match some more tables with each other. Did anybody else figure some dependencies based on their connections out?
I played with it a bit yesterday and replaced the main query in extract.py with this:
SELECT message.sender_jid_row_id as key_remote_jid, message._id, message.from_me as key_from_me, message.timestamp, message.text_data as data, message.status, message_future.version as edit_version, message_thumbnail.thumbnail as thumb_image, message_media.file_path as remote_resource, message_media.mime_type as media_wa_type, message_location.latitude, message_location.longitude, message_quoted.key_id as quoted, message.key_id, message_quoted.text_data as quoted_data, message_media.media_caption FROM message LEFT JOIN message_quoted ON message_quoted.message_row_id = message._id LEFT JOIN message_location ON message_location.message_row_id = message._id LEFT JOIN message_media ON message_media.message_row_id = message._id LEFT JOIN message_thumbnail ON message_thumbnail.message_row_id = message._id LEFT JOIN message_future ON message_future.message_row_id = message._id WHERE key_remote_jid <> '-1';
That brings me past the missing messages table, but will clash in extract.py:240 trying to match the remote id to something in the big data dictionary. So, I assume me trying to replace key_remote_jid with sender_jid_row_id might go into the right direction, but might need some more joints to actually turn things into a phone number.
There are also a messages_view and a chats_view now that seem to match some more tables with each other. Did anybody else figure some dependencies based on their connections out?
I will have a look at the clash later.
I am curious about how WhatsApp decides which table to store messages π€¨. Is "message" a legacy table? Or "messages" is the legacy one?
"messages" is the legacy one, please read https://thebinaryhick.blog/2022/06/09/new-msgstore-who-dis-a-look-at-an-updated-whatsapp-on-android/
I played with it a bit yesterday and replaced the main query in extract.py with this:
SELECT message.sender_jid_row_id as key_remote_jid, message._id, message.from_me as key_from_me, message.timestamp, message.text_data as data, message.status, message_future.version as edit_version, message_thumbnail.thumbnail as thumb_image, message_media.file_path as remote_resource, message_media.mime_type as media_wa_type, message_location.latitude, message_location.longitude, message_quoted.key_id as quoted, message.key_id, message_quoted.text_data as quoted_data, message_media.media_caption FROM message LEFT JOIN message_quoted ON message_quoted.message_row_id = message._id LEFT JOIN message_location ON message_location.message_row_id = message._id LEFT JOIN message_media ON message_media.message_row_id = message._id LEFT JOIN message_thumbnail ON message_thumbnail.message_row_id = message._id LEFT JOIN message_future ON message_future.message_row_id = message._id WHERE key_remote_jid <> '-1';
That brings me past the missing messages table, but will clash in extract.py:240 trying to match the remote id to something in the big data dictionary. So, I assume me trying to replace key_remote_jid with sender_jid_row_id might go into the right direction, but might need some more joints to actually turn things into a phone number.
There are also a messages_view and a chats_view now that seem to match some more tables with each other. Did anybody else figure some dependencies based on their connections out?
key_remote_jid same with raw_string from table jid
Are there any updates on this?
Are there any updates on this?
Not yet. Hopefully there will be updates on Dec.
Are there any updates on ?
Found a solution but i am not sure every aspect of project is working properly.
First https://github.com/andreas-mausch/whatsapp-viewer/issues/151#issuecomment-1229244368 I applied this solution this solition modified old msgstore.db.
After that i still recieved errorS. So with SQLlite i add new tables to overcome error: no such table: ... After adding couple tables, code is working now. And able to output a result.
But as i said i cannot confirm that every aspect of project is working properly.
Hi everyone, I am working on porting the code to the WhatsApp database with new schema. The main part is completed and there are some bugs need to be fixed.
I created a new branch called "message_table" which support message
table directly from wtsexporter command. Feel free to give it a try but please expect bugs from the branch. Bug reports are welcome and should be reported in this issue.
Also, big thank for the SQL statement written by @ulno. For the replacement of key_remote_jid
, since sender_jid_row_id
could be 0, instead of treating it as the key_remote_jid
, I joined the chat
and jid
table together to obtain the accurate key_remote_jid
.
First of all, thank you for working on this. I just downloaded the branch and gave it a try. Unfortunately it didn't work for me.
β°βο wtsexporter -a
Gathering contacts...(97)
Traceback (most recent call last):
File "C:\Users\******\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\******\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "C:\Users\******\AppData\Local\Programs\Python\Python310\Scripts\wtsexporter.exe\__main__.py", line 7, in <module>
File "C:\Users\******\AppData\Local\Programs\Python\Python310\lib\site-packages\Whatsapp_Chat_Exporter\__main__.py", line 174, in main
messages(db, data)
File "C:\Users\******\AppData\Local\Programs\Python\Python310\lib\site-packages\Whatsapp_Chat_Exporter\extract.py", line 195, in messages
c.execute("""SELECT count() FROM messages""")
sqlite3.OperationalError: no such table: messages
I made sure to uninstall the old version I had with pip uninstall whatsapp-chat-exporter
and then installed the new one.
Is there anything I'm missing?
First of all, thank you for working on this. I just downloaded the branch and gave it a try. Unfortunately it didn't work for me.
β°βο wtsexporter -a Gathering contacts...(97) Traceback (most recent call last): File "C:\Users\******\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Users\******\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "C:\Users\******\AppData\Local\Programs\Python\Python310\Scripts\wtsexporter.exe\__main__.py", line 7, in <module> File "C:\Users\******\AppData\Local\Programs\Python\Python310\lib\site-packages\Whatsapp_Chat_Exporter\__main__.py", line 174, in main messages(db, data) File "C:\Users\******\AppData\Local\Programs\Python\Python310\lib\site-packages\Whatsapp_Chat_Exporter\extract.py", line 195, in messages c.execute("""SELECT count() FROM messages""") sqlite3.OperationalError: no such table: messages
I made sure to uninstall the old version I had with
pip uninstall whatsapp-chat-exporter
and then installed the new one. Is there anything I'm missing?
Can you verify that the __main__.py
file import extract_new from Whatsapp_Chat_Exporter import extract_new as extract
instead of extract from Whatsapp_Chat_Exporter import extract, extract_iphone
__main__.py
did not include from Whatsapp_Chat_Exporter import extract_new as extract
.
Replaced the files in \site-packages\Whatsapp_Chat_Exporter
manually with the ones from the branch now and ran the script again.
Now being faced with this
β°βο wtsexporter -a
Gathering contacts...(97)
Traceback (most recent call last):
File "C:\Users\******\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\******\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "C:\Users\******\AppData\Local\Programs\Python\Python310\Scripts\wtsexporter.exe\__main__.py", line 7, in <module>
File "C:\Users\******\AppData\Local\Programs\Python\Python310\lib\site-packages\Whatsapp_Chat_Exporter\__main__.py", line 184, in main
messages(db, data)
File "C:\Users\******\AppData\Local\Programs\Python\Python310\lib\site-packages\Whatsapp_Chat_Exporter\extract_new.py", line 206, in messages
c.execute("""SELECT jid.raw_string as key_remote_jid,
sqlite3.OperationalError: no such column: message_media.media_caption
__main__.py
did not includefrom Whatsapp_Chat_Exporter import extract_new as extract
. Replaced the files in\site-packages\Whatsapp_Chat_Exporter
manually with the ones from the branch now and ran the script again. Now being faced with thisβ°βο wtsexporter -a Gathering contacts...(97) Traceback (most recent call last): File "C:\Users\******\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Users\******\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "C:\Users\******\AppData\Local\Programs\Python\Python310\Scripts\wtsexporter.exe\__main__.py", line 7, in <module> File "C:\Users\******\AppData\Local\Programs\Python\Python310\lib\site-packages\Whatsapp_Chat_Exporter\__main__.py", line 184, in main messages(db, data) File "C:\Users\******\AppData\Local\Programs\Python\Python310\lib\site-packages\Whatsapp_Chat_Exporter\extract_new.py", line 206, in messages c.execute("""SELECT jid.raw_string as key_remote_jid, sqlite3.OperationalError: no such column: message_media.media_caption
It should be fixed in the latest commit: https://github.com/KnugiHK/Whatsapp-Chat-Exporter/commit/d3892a4e4f4f0b533107f871a23de22153cc879a.
The new branch worked on my side -- thanks a lot ! Although all medias are missing from the htmls (i tried both with and without the -e flag). Thanks a lot for your work, your tool is great.
The new branch worked on my side -- thanks a lot ! Although all medias are missing from the htmls (i tried both with and without the -e flag). Thanks a lot for your work, your tool is great.
Hello, @cbzittoun , I had the same problem with the media. Try to pass the parent directory of "Media" instead of the address to "Media" itself.
I still have three issues, though:
I think I was pointing to no Media folder actually, as I was running the cmd for the 2nd time. I was expecting the script to copy the files, not move them (although slower, it would be safer).
Maybe it is worth an explicit warning that input files are being moved around?
@cbzittoun
The new branch worked on my side -- thanks a lot ! Although all medias are missing from the htmls (i tried both with and without the -e flag). Thanks a lot for your work, your tool is great.
Actually, embedding media to the HTML file (-e option) is removed and was presented in the very early version. It is planned to have embedding again. Media should be located under WhatsApp
folder (i.e., WhatsApp/Media
). If your current working directory has the folder named WhatsApp
, it will automatically move that folder. If you want to specify another directory, use -m
.
I think I was pointing to no Media folder actually, as I was running the cmd for the 2nd time. I was expecting the script to copy the files, not move them (although slower, it would be safer).
Maybe it is worth an explicit warning that input files are being moved around?
We move the discussion of this to https://github.com/KnugiHK/Whatsapp-Chat-Exporter/issues/25.
@velecto1
I still have three issues, though:
* All exported conversations start with "This message is not supported". * The flag "-s" seems not to be working. * I've realized that it would also be helpful for me to split the result into multiple HTML files see_no_evil.
Can you confirm if the first two issues happen only in the
message_table
branch? If they happen in the main branch, please open another issue for that.
Hi, sorry for the delay; I didn't have the old backup available immediately.
1) Nope, the "This message is not supported" thing does not display in the resulting HTML when converting an old database. The chat is actually complete, so maybe it is the "message" saying: "Messages and calls are end-to-end encrypted..."
2) The 'main' branch doesn't have the '-s' flag available at all.
Hi, sorry for the delay; I didn't have the old backup available immediately.
1. Nope, the "This message is not supported" thing does not display in the resulting HTML when converting an old database. The chat is actually complete, so maybe it is the "message" saying: "Messages and calls are end-to-end encrypted..." 2. The 'main' branch doesn't have the '-s' flag available at all.
@velecto1
I still have three issues, though:
* All exported conversations start with "This message is not supported". * The flag "-s" seems not to be working. * I've realized that it would also be helpful for me to split the result into multiple HTML files see_no_evil.
Can you confirm if the first two issues happen only in the
message_table
branch? If they happen in the main branch, please open another issue for that.
Hello there u can contact me I can help u out but in private !
@velecto1
I still have three issues, though:
* All exported conversations start with "This message is not supported". * The flag "-s" seems not to be working. * I've realized that it would also be helpful for me to split the result into multiple HTML files see_no_evil.
Can you confirm if the first two issues happen only in the
message_table
branch? If they happen in the main branch, please open another issue for that.Hello there u can contact me I can help u out but in private !
Appreciated! Feel free to poke me around in the Matrix network: https://matrix.to/#/!KtGjfSytJvzxewsXVd:matrix.org
I've had the same problem. I was able to extract a Whatsapp DB from Android using the extractor tool and noticed it only has the 'message' table. Will give the test branch a try and see how it goes.
I've had the same problem. I was able to extract a Whatsapp DB from Android using the extractor tool and noticed it only has the 'message' table. Will give the test branch a try and see how it goes.
This issue will likely happen to more and more people in the future as WhatsApp updates more people's database schema.
I just used the code from the message_table branch and was able to successfully export my Whatsapp database. I really appreciate your effort for this useful tool. The only thing I noticed in the resulting files is, that in group chats the name of the group is shown instead of the name of the person writing the message.
I just used the code from the message_table branch and was able to successfully export my Whatsapp database. I really appreciate your effort for this useful tool. The only thing I noticed in the resulting files is, that in group chats the name of the group is shown instead of the name of the person writing the message.
Thanks for reporting that, I can reproduce the problem.
Edit: @flansch The problem should be fixed with https://github.com/KnugiHK/Whatsapp-Chat-Exporter/commit/4cb4ac3e7b96595248a71d0232284b86be0cde96. The commit is presented in message_table branch, see if the problem solved with the commit.
I updated to the latest version of the message_table branch and can confirm that the error is gone, thank you for that. In the meantime I had another issue: When opening the files in Safari on Mac the encoding is not detected properly and emojis and german umlauts are not displayed correctly.
By adding <meta charset="UTF-8">
to the head section of one of the files, I was able to fix that. Could you please automatically add this tag.
When running the script I get following error message
My database doesn't contain the table
messages
, I only have one calledmessage
which contains the data but the schema of the table looks different from what it's supposed to look like. What could be the issue here?