Media files support

[jbollacke]

Hi, this is a pretty interesting project. Especially for me as I am currently making the switch ;-)

I understand that media files are not supported yet. I am interested in contributing but before I set up my xcode I would like to know what exactly is the problem for media files? Do you think its possible (w/ ease)?

[residentsummer]

Hi, @jbollacke! I don't think it's a huge problem, just quite involved process :)

There is a thing called WAMediaItem, that represents audios, videos, pictures, vcards and shared coordinates. Most of them also have an associated file in storage and a thumbnail. To import a media from android backup you'll need to do at least 2 things:

• Fill the needed props of WAMediaItem. This includes some figuring out which fields from android db maps to iOS ones.
• Inject a file into iOS backup. I haven't done this, but it shouldn't be a problem. The backup has rather simple format: bunch of files + sqlite metadata storage.

These are fields of WAMediaItem:

To figure out what goes where, you may try sending/receiving a few images or videos on iOS and analyze the resulting ChatStorage.sqlite.

[dosenflieger]

I just successfully transfered my wife's WA chats (without media files) from an Android device to her new iPhone (iOS 10.3.2; WA 2.17.31) and wanted to thank you for providing us with this nifty open source tool!

I've also looked into injecting media files into the WA domain of a local itunes backup but I'm not sure how to fill in the required plist meta data (BLOB 'file') for new files into Manifest.db, especially 'InodeNumber'. Tools like iMazing allow adding files to a copy(!) of an iOS backup but also warn about possible (file system?) corruption when restoring it to an iOS device.

[residentsummer]

Hi, @dosenflieger! I'm glad that this tool helped you!

Looks like that the plist is a dump of this object: MBFile.

The bad news is that Inodes are low level filesystem structures (IIRC entries in FS index of all files) and messing with with them may indeed lead to corruption. To be honest, I don't know if it's even possible to create a correct inode number without an access to the device filesystem...

There is a slight hope that editing the backup may not be necessary - it's a mediaURL field of the WAMediaItem entity. Maybe if you fill it with the corresponding value from an Android backup, the Whatsapp will download the file by itself?

I must say that there is a couple of insane ways to transfer files to the device, if the above method will work partially or not at all (e.g. if only media for a last few days/months will be available from whatsapp servers). But i'd rather not try them ;)

[b4stien]

Just my 2ct on the "Whatsapp will download the file by itself": nothing is stored on Whatsapp server, so it's highly unlikely that the file will be downloaded from there.

[jbollacke]

nothing is stored on Whatsapp server,

Well, Amazon S3 then...

[Mc-muffin]

I don't wanna be "that guy" but is this project abandoned?

[residentsummer]

@Mc-muffin it depends on your definition of "abandoned". Do I have time (or intention) to implement new features - no. May some help or troubleshooting be provided - yes. Pull requests are always welcome, though :)

By the "activity" of this repo I can conclude that it helped some people to migrate textual history, so I guess this part still works. There are some outstanding help requests though.

As of media import, I have no plans to implement it myself. When I started the project, this feature was on my list. However, it became apparent that it requires too much work and doesn't worth the effort. I'm not aware if other community members are actively working on it.

[Mc-muffin]

Oh I see, the media files aren't that much of an issue right now for me (I mean, I can view them on the PC if I really need to) I'm going to use this tool on the next few days when I get a hold of a macbook, I hope I won't face any issues :)

Another thing I want to add is a big thank you, for creating this handy open source tool, is the only alternative that I know of that offers this transfer process (and even then it's free!) if I was skilled enough with databases or iOS coding I would be happy to give you a hand, sadly I don't know much :(

Thanks again for providing the tool and support ^-^

[dheerajramchandani]

Hi,

First of all, thanks a lot for creating this cool tool and making it free for public use. I'd say I was fortunate enough to stumble upon it because any search on Google for such migration tool returns results of only dubious paid softwares. One could pay for such software if its available on App Store and I am surprised why Apple doesn't provide such utility.

Coming to the main point, I am very much interested in working on this to fix the caveats. I am a novice in Objective C but was able to understand the code from high level. I have started looking at some tutorials to understand the syntax and basics of Objective C. As such I am a python programmer (currently pursuing my master's in Computer Science) and the motivation for me to take up this work is that I might make the shift in future and that we (my wife) has already suffered the loss once back in May 2019. Its just that I do not want to lose media files and that is why I want this to work before I buy an Apple device or not buy it at all. As I said that I am new to Objective C and trying to learn it only for this project, I want to ask would you be able to guide me develop this enhancement? I know from other threads that you already made it clear that you do not want to work anymore on this project but still asking.

To start working, could you please briefly explain what do you mean by placeholders? Is that en empty rectangular object in the chat conversation but no media? Or blurred thumbnail of the media? What are shared locations? Do they mean the Google locations which we sometimes share with others? Similarly, could you please explain what do you mean when you say messages for contacts with changed phone numbers are not linked? Usually, there is a new conversation for any new phone number but in case of a contact with new phone number, there is always a link in the old conversation that can be tapped to reach the new conversation. Those links are useful when the new phone number of a contact is not saved.

Regarding the media files problem, I see that you have already mentioned the solution but could you please explain it in detail? Specially the injecting part?

Last but not the least, regarding resources, do you think its a hinderance to work on this project if I do not have an iPhone at the moment? I am not willing to buy it before anything concrete and in fact, being a student I can't afford to buy it just for experimentation (I wouldn't want to keep it if the migration doesn't work). Another thing is that I understand I should be working with Xcode for this project - is there a minimum system requirement or a 2011 MacBook (IIRC with macOS Sierra) is fine to work with?

Looking forward to a positive response, thanks in anticipation !!!

[residentsummer]

Hi, @dheerajramchandani . Thank you for kind words and for your interest in improving the tool!

I'll try to answer your questions to my best.

To start working, could you please briefly explain what do you mean by placeholders? Is that en empty rectangular object in the chat conversation but no media? Or blurred thumbnail of the media?

I guess you're referring to the first item in the list of caveats in readme. It's just a text that tool inserts into conversation where unsupported media item should be. See this code block.

What are shared locations? Do they mean the Google locations which we sometimes share with others?

Yes, those are messages, that contain GPS coordinates or address of something. They are identified by message type 5 and not handled by watoi, just like pictures. I guess it's easy to support them, but not sure if someone needs this.

Similarly, could you please explain what do you mean when you say messages for contacts with changed phone numbers are not linked?

There are quite a few DB entities involved when restoring a conversation history. Conversations (both private and group) correspond to WAChatSession, participants are WAGroupMember, individual messages are WAMessage. Each message is linked to its author and to chat session. One of the steps in conversation history recovery process is to launch a whatsapp on target device - during this stage app populates some of the required entities in the DB, including WAChatSessions and WAGroupMembers. Participants got linked to the contacts in the iOS phone book - this way you got contact names in your conversations instead of the phone numbers. The issue is that it only syncs current phone numbers - if person changed it's number throughout the conversation, only messages with the latest number will be linked to the contact and get human-readable name. Messages before the change will display with a phone number, guessed from the author identifier in the whatsapp network. The same applies for the participants that left the conversation, as WAGroupMember entities ain't created for them too.

Usually, there is a new conversation for any new phone number but in case of a contact with new phone number, there is always a link in the old conversation that can be tapped to reach the new conversation. Those links are useful when the new phone number of a contact is not saved.

If I remember it right, Whatsapp did not have this feature back then, so I doubt this links are imported by watoi.

Last but not the least, regarding resources, do you think its a hinderance to work on this project if I do not have an iPhone at the moment? I am not willing to buy it before anything concrete and in fact, being a student I can't afford to buy it just for experimentation (I wouldn't want to keep it if the migration doesn't work). Another thing is that I understand I should be working with Xcode for this project - is there a minimum system requirement or a 2011 MacBook (IIRC with macOS Sierra) is fine to work with?

Any mac can run Xcode AFAIK, so I guess it will do :) Having an iOS device is 99% required - you'll need to be able to restore backups a lot to check if your solutions work. Try to borrow an old iPhone, that's not being used anymore (that can run iOS 10 or later) - it should do fine.

Regarding the media files problem, I see that you have already mentioned the solution but could you please explain it in detail? Specially the injecting part?

So, to the main part. I haven't been able to find information on how to add files to iOS backup. Some approaches that come to mind:

• When initially receiving a message with attached media, Whatsapp have to download it somehow. My guess that they're being fetched through mediaUrl field and then stored on the device. Restoring via this method will require uploading media items to some location, putting the URLs for each file into the backup, restoring it to the device and triggering download (not sure how it's done).
• Certain proprietary tools can insert files into the backup, but they mention a risk of backup corruption due to this. I assume they're somehow faking MBFile structure.
• This method come to me while writing this message :) You can evade creating fake MBFiles by sending files to the device while it is running. Open iTunes, choose your iPhone and select "File Sharing" tab. There will be a list of apps that support uploading documents via iTunes. Choose any (e.g. Pages) and press "Add" button at the bottom. Seems that it's possible to upload files of any type, even if they're not supported by the app. That's exactly what we need to import media! The plan is to put all media files (pics, videos, voice records) onto the device via iTunes, make a backup with these added files and than "move" and associate them with WAMediaItems by editing a backup's Manifest.db.

[dheerajramchandani]

Thanks for all the info and apologies for delayed response. My semester is going to start from Monday and I am not sure how much time will I be able to dedicate to this project, so the speed may be very slow. But as we know, slow and steady wins the race 😄.

Regarding iOS device, I think I will be able to arrange one device. But before that could you please confirm if the project in its current state can "merge" WhatsApp Android database with iOS database or it can only replace it.

[residentsummer]

Hi, @dheerajramchandani !

if the project in its current state can "merge" WhatsApp Android database with iOS database or it can only replace it.

Yes, Android messages are merged into existing iOS database with the assumption that they're older (all messages from android db are older that any message from ios db). I can think of one case when this assumption is wrong:

• Initially, whatsapp on iOS was used, with enabled backups (the default setting IIRC)
• User migrated to Android, all history is lost (it's even worse for our scenario if it was migrated using something similar to watoi)
• User migrates back to iOS. Whatsapp detects backup and restores it (don't know if it's automatic or no). Now we have three "chunks" of conversation history:

1. Oldest chunk from iOS backup, recovered by Whatsapp on new device
2. Messages from Android db, that we're trying to recover here
3. Newly arrived messages

The proper order is 1->2->3, but after watoi it will be 2->1->3. It's fixable, though.

[residentsummer]

@dheerajramchandani Actually, I don't have 100% certainty if this tool works now at all, because there is no tracking or some place when people can report if migration worked or not... But the project receives stars few times a month, bug reports are rare, so I guess it works :)

EDIT (Dec 13, 2020): Last confirmed success was on Nov 22, 2020.

[dheerajramchandani]

@residentsummer Thanks for your inputs. I think the first thing I will try to do before proceeding further is to see if it works. I will keep you updated.

[eyalk]

Hi I tried to use the tool to see if works. However in Android the msgstore.db is encrypted and the tool accept only decrypt db. How do you extract the decrypt file?

Thanks Eyal

On Sat, 22 Aug 2020 at 6:43 Dheeraj Ramchandani notifications@github.com wrote:

@residentsummer https://github.com/residentsummer Thanks for your inputs. I think the first thing I will try to do before proceeding further is to see if it works. I will keep you updated.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/residentsummer/watoi/issues/1#issuecomment-678588448, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABNQ2TMB6O7JG47EZFOQMTSB45ELANCNFSM4DKLCBHQ .

[residentsummer]

@eyalk IIRC I've used something called WhatsApp-Key-DB-Extractor for decryption.

[ludufre]

Hi. I was finally able to inject files into the iOS backup and restore with success.

The big issue was the Binary PList in file column of the Files table on the Manifest.db.

For now the script has nothing related to the WhatsApp files themselves. I did it in NodeJS because it's a language I'm more comfortable with.

I plan to implement it here in Object-C, maybe I'll be able to submit a PR in the next few days.

In my tests, I injected 1000 files in a iPhone 7.

WIP scripts (Get latest InodeNumber and InjectFile): https://github.com/ludufre/wa-explorer/tree/main/script

[dheerajramchandani]

Hey @ludufre , thanks a lot for all the effort. I just could not take up the project due to one or the other thing. But what I am reading online since last 4-5 months, work is in progress at WhatsApp to bring this feature. Nonetheless, great work. Its no less than community service 🙂.

[residentsummer]

@ludufre I wonder if iPhone continues to work normally after just taking next available inode number?

For example, after using the device for a while and doing something that causes creation of new files in a natural way (e.g. receiving and downloading an image in Whatsapp, or downloading something like pdf into another app), will the new file pick up inode number after injected files, or will it took the number from one of them and overwrite it, causing corruption?

[ludufre]

@residentsummer after restored about 15k files (jpg, opus, mp4), I enabled iCloud Photos and all my 30k photos has been downloaded with success. No corruption so far. Look promising

[residentsummer]

@ludufre that's nice, thanks for the follow up.

[j0h]

@residentsummer after restored about 15k files (jpg, opus, mp4), I enabled iCloud Photos and all my 30k photos has been downloaded with success. No corruption so far. Look promising

@ludufre wow, appreciate your work on this! Would you be able to write up some instructions on how you got it working? Lots of instructions are scattered in issues and I'm trying to figure out the full process.

[fjrisco117]

@ludufre did you eventually find any issues or got corrupted files? Can you provide some instructions please? Thank you

[fjrisco117]

@j0h did you make any progress? I'm just starting myself.

[martinodonnell]

@ludufre Is this still a WIP or is this a working version. I am going to work on this this weekend if I can get my android database decrypted

[timppa23]

@martinodonnell Hi, have you made any progress? I spent like 10 hours exploring this issue. In the end I was able to transfer my chats to iPhone without media. https://github.com/YuvrajRaghuvanshiS/WhatsApp-Key-Database-Extractor this worked for me for decrypting. Made a backup with apple configurator and after restoring the backup I was a bit suprised that it had lost all my apps. Previous restore from iCloud restored everything.

[martinodonnell]

@timppa23 I was able to add pictures into the backup and then put that on the phone. I was never able to get them to show up on the Whatsapp or even on the photos. All my apps did come back. Maybe you added data wrong.

I used watoi to transfer the media.

I spent a couple nights over a 4 week period on it but decided to leave it a couple months and wait for Whatsapp to release the next beta which should have this feature

[monkora]

Did anyone manage to do this after @ludufre?