Investigate whatsapp model to backup and restore older chat

[sojharo]

As discussed, we need to investigate how whatsapp is doing backup of user chat on google drive or iCloud. Later, we would implement same for kibochat.

From initial thought, whatsapp gives options to set time to backup chat. i.e. daily, weekly and monthly.

[jekram]

@sumairasaeed This is FYI for you. No action needed.

[sumairasaeed]

ok. Acknowledged.

On Thu, Dec 15, 2016 at 9:07 AM, Cloudkibo notifications@github.com wrote:

@sumairasaeed https://github.com/sumairasaeed This is FYI for you. No action needed.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Cloudkibo/Android/issues/355#issuecomment-267233756, or mute the thread https://github.com/notifications/unsubscribe-auth/AKbhp4h6jsN94Z8oHgJ17nQsLe554U4oks5rIL0XgaJpZM4LNsRv .

[sojharo]

I have investigated on this. Whatsapp uses Google Drive for Android API to backup the user data on Drive. It gives the options to update daily, weekly, monthly or only when user taps backup button. It also gives the option to backup only when user is on wifi. We have option to backup videos as well or we can opt out from backing up videos.

Now, for Android, all the application data is stored in special folder on Google Drive called Backups. Here, we can see which of our apps have stored data on Google Drive. This doesn't show what the data actually is, but just shows the name of application and amount of data. Here, we can see whatsapp data on my google drive:

We also have option to turn off or delete the backup stored by the application.

Now, I further looked into how to do this. I found several articles for backup. I read following few of them. Also, there is difference between application data backup and Android 6.0 auto backup. Android auto backup is the way android automatically backup all the applications user has installed. This doesn't backup data but only settings and configurations of each application. Android 6.0 does it automatically for all apps.

https://developers.google.com/drive/android/appfolder

http://www.androidpolice.com/2016/12/08/google-drive-backups-tab-starts-showing-gives-glimpse-device-app-backups/

https://developer.android.com/google/backup/index.html

https://developer.android.com/guide/topics/data/backup.html

https://developers.google.com/drive/v3/web/appdata

https://github.com/googledrive/android-demos

In this, I still need to understand how to actually write the data from sqlite to google drive. How to put timings in it. So, we should keep this task open for now. This is under construction. I would also learn about adding calendar API to do sync on periodic basis.

[jekram]

@sojharo Thanks Good information.

@sumairasaeed This is FYI for you only. No action required.

[sojharo]

I went into sample codes of this to understand how we can use it to sync our own data to Google Drive App folder. From code, I understood that we can write files to this folder or create new folders within. Now, we need to store our sqlite database content to this drive first. For this, we can create one file for each table and store entries line by line. End of line will mean that entry has ended here. Just like whatsapp, we would give the options to backup daily, weekly, monthly or when user wants. However, to restore the data from this backup, we would only do this during installation of app. Whatsapp also does it in this way. Other thing we would be uploading to Google Drive App folder is to upload images which are in chat. Also, we would be uploading documents and videos in chat to drive. However, we would first start slow and then start adding these things. Once it is done, then we won't store data to server.

Here are the files that I read to understand how to implement this functionality.

https://github.com/googledrive/android-demos/tree/master/app/src/main/java/com/google/android/gms/drive/sample/demo

Specially following files:

https://github.com/googledrive/android-demos/blob/master/app/src/main/java/com/google/android/gms/drive/sample/demo/CreateFileActivity.java

https://github.com/googledrive/android-demos/blob/master/app/src/main/java/com/google/android/gms/drive/sample/demo/EditContentsActivity.java

https://github.com/googledrive/android-demos/blob/master/app/src/main/java/com/google/android/gms/drive/sample/demo/CreateFileInAppFolderActivity.java

As a next step, I would create the design document for this feature and then open tasks.

[jekram]

OK. Sounds good. An next step let's create a design document. One thing we need our design is that the file save and restore should be abstracted so we can support multiple back end storages. Like Amazon S3, or MSFT storage. In V1 implementation we would only implement google drive. However, our design should not be limited.

[sojharo]

After looking into overall feature, I have created a document which discusses overall sub-features of this enhancement and also discuss the logic to backup and restore.

https://github.com/Cloudkibo/Android/blob/master/Documentation/Backup.md

[jekram]

@sojharo Thanks for the documentation.

For the Data Structure. Can you comment on how Whatsapp is using? Is it similar to what you are proposing?

Data Structure

Now, in order to store the sqlite table entries in files, we would create file for each table that we have in our schema. Each line in the file will represented data entry in that table. The file format would be called csv so that we easily separate the columns and make them in tabular format. We would also store display pictures of groups and for this we would maintain a separate file which would indicate which image belongs to which group.

2. Are there any open source libraries that we can use to save these files.

[sojharo]

I will look for libraries. However, even after trying I could not know how whatsapp is storing the information which data structure they are using. This is because the Google Drive has a special folder for App Data. This folder is completely encrypted and only shows the names of app which have stored data there. It doesn't show what type of data is stored.

I will try to find libraries today. Also there is a way called serialising, I would read that too.

[sojharo]

I had looked into it yesterday and found only one library which was for nodejs.

https://www.npmjs.com/package/gdrive-appdata

However, from reading library code, I learnt that they are storing the data in format of JSON in files instead of csv.

I have opened tasks also and have started to work

[sojharo]

We can close this task as we have other tasks to track in this feature.

[jekram]

Great job