Messages Import Export issue

Abcd1234-dot commented 4 months ago

Checklist

[X] I can reproduce the bug with the latest version given here.
[X] I made sure that there are no existing issues - open or closed - to which I could contribute my information.
[X] I made sure that there are no existing discussions - open or closed - to which I could contribute my information.
[X] I have read the FAQs inside the app (Menu -> About -> FAQs) and my problem isn't listed.
[X] I have taken the time to fill in all the required details. I understand that the bug report will be dismissed otherwise.
[X] This issue contains only one bug.
[X] I have read and understood the contribution guidelines.

Affected app version

1.0.1

Affected Android/Custom ROM version

Android 8.1

Affected device model

Moto G5 Plus

How did you install the app?

GitHub releases

Steps to reproduce the bug

Open app, go to settings and click on export messages.
It asks for file name without .json
Android system may ask you to complete task with native android files or any other file manager installed on device (in my case fossify file manager).
Choose location for exporting.
Now try to import same that just exported.

Expected behavior

It should be imported without any issues and restore all the SMS and MMS.

Actual behavior

While trying to import same file, it always prompt native android files and the exported file stays greyed out and not able to select it. It also doesn't contain any extensions by default (especially if exported via fossify file manager). But adding extension .json manually doesn't help much either. In my observations, if file is exported via native android files, added .json when android files asks for name and exporting it to "Downloads" directory only works. Even if I export it to any other location other than downloads, it still stays greyed out.

Screenshots/Screen recordings

No response

Additional information

It could be fossify file manager bug as I tried to export back-up from Notally, extension was always missing. Also similar bug is reported as (https://github.com/FossifyOrg/Notes/issues/13). But it doesn't explain why this works in only Download folder and not to any other location.

biopsin commented 4 months ago

Duplicate of https://github.com/FossifyOrg/Messages/issues/6

Aga-C commented 4 months ago

@biopsin From the descriptions, these two are completely different. Here the file is exported, but without extension, so it can't be imported.

Abcd1234-dot commented 4 months ago

Duplicate of #6

There's no crash or any visible error in this case, export is always successful (with or without extension). And even with proper extension, the same file can not be imported unless it is exported to "Downloads" folder in home directory, and if its unmoved, untouched. If I move same file to another location and try to import it, it is greyed out. If i again move it to Downloads and try to import it, it still stays greyed out. So, both issues are very different ones in my opinion.

biopsin commented 4 months ago

But what sizes is the exported file?

Abcd1234-dot commented 4 months ago

But what sizes is the exported file?

The file which gets exported is usually of few MBs (around 5 MB), I had 2-3 years of SMS which i was backing up.

tom93 commented 3 months ago

There are two separate issues:

Missing .json extension on export. As you wrote, very similar to FossifyOrg/Notes#13. I'll open a PR. (Update: #136)
Android reports the MIME type as "application/octet-stream" for .json files outside the Download folder. I can reproduce this on Android 9. In particular, if I export to the Download folder (even without the .json extension) then the MIME type is reported as "application/json" (using "Get info" in the native Files app), and if I move this file outside the Download folder (e.g. into Pictures) then the MIME type is reported as "application/octet-stream". I'm not sure why that is (I suspect the Download folder stores additional metadata, but I can't find any reference). Update: Yes. The Content URI is of the form content://com.android.providers.downloads.documents/... rather than content://com.android.externalstorage.documents/... which shows that the provider is DownloadProvider. As a workaround, how about adding "application/octet-stream" to the list of allowed file types? (Update: Implemented in #137)

tom93 commented 3 months ago

@Abcd1234-dot Do you want to check if #136 and #137 fix this issue for you? They do for me.

(I'm not a project member so you shouldn't trust me, but if you don't want to build from source / wait for a project member to build, then I've attached an APK file that I built myself: unofficial-messages-2-foss-debug-for-issue-88.apk.zip.)

Abcd1234-dot commented 3 months ago

I don't know how to compile from source code. the apk file you attached is compiled one from the source code of fossify messages and includes your fix? if yes then i am willing to give it a try. And yeah probably you identified actual the problem.

tom93 commented 3 months ago

the apk file you attached is compiled one from the source code of fossify messages and includes your fix?

Yes. (I had to put it in a .zip file because GitHub limits attachment file types.)

@naveensingh Are you okay with me building and uploading APKs like that for testing?

naveensingh commented 3 months ago

@tom93 Yeah, it's okay. As these things can be risky in general, there will be some guidelines later. I'll be setting up workflows to build APKs automatically so PR authors don't have to do it themselves but it'll require a PR to have testers needed label (see https://github.com/FossifyOrg/Launcher/pull/12).

Abcd1234-dot commented 3 months ago

@tom93 I tried the attached apk. It fixes one issue. Now I am able to select .json file stored anywhere in internal storage. But when I select it, it prompts the "importing messages" window with tick/untick sms and mms messages and ok and cancel buttons and when i click on ok, "importing" appears for few seconds and then disappear and nothing happens!! the "importing messages" window stay there as before. I guess import is not successful or some other kind of error (no visible error tho)?? I tried my old backup and new backup but same results.

Abcd1234-dot commented 3 months ago

I think this issue exists in all apps which allows import export backups, so it might need to port to all such apps if the PR is successfully merged.

tom93 commented 3 months ago

I'm glad it fixed the file selection issue. I don't know why the import isn't working, I tried reproducing in an Android 8 emulator but it worked fine for me.

I'm not sure how to debug the new issue. All I can think of is to open the .json file in a text editor (might need to rename to .txt) and check that it actually contains the messages you expect. And the adb logs might be useful if you are able to get them. Update from the future: The issue was that the import is really slow.

I hope the devs can help more...

For reference, here are my steps:

Receive a fake SMS (emulator controls: "..." > Phone > Send Message)
Open Fossify Messages and reply (it will be stuck in "Sending...", that's fine)
Export messages
Delete the conversation
Import messages

The messages reappear correctly. The only difference compared to what you described is that the "Import messages" popup disappears as soon as I click "OK". I also tested with MMS.

I think this issue exists in all apps which allows import export backups

Do you mean the missing .json extension and file selection issue? If so, I've mentioned porting those fixes (https://github.com/FossifyOrg/Messages/pull/136#issuecomment-2001994711, https://github.com/FossifyOrg/Messages/pull/137#issue-2189992661) and I plan to follow up on that once approved.

Abcd1234-dot commented 3 months ago

I cross verified the exported file by opening it on my desktop, it contains messages as expected. my old backup contains too much content that made my desktop with low ram frozen (last 2 years of SMS). But my new backup contains very less messages. there's no visible error and exporting is always successfull in my case. i guess logs might be useful if someone tells me how to produce logs. I would provide them to @tom93 personally or devs and won't post them publicly for security reasons.

tom93 commented 3 months ago

Getting the Android logs is a bit of a hassle*.

But since you're okay with running debug builds, and since the import code is fairly short, I can add some custom debugging code that will log what it's doing to a text file in the Download folder. That might be enough to identify the issue.

*To get the Android logs, you need to download the platform tools, enabled developer mode and USB debugging on your phone, and then run adb logcat from the command line.

P.S. For future reference, if you're having trouble opening large text files on a PC, I recommend trying Notepad, WordPad (included with Windows until Windows 11), or installing Notepad++. But since the issue also happens with the new small backup, we should focus on that.

Abcd1234-dot commented 3 months ago

I have logs now. I am not sure how to share them with you privately as there seems no way to send personal message on github. you may ping me on rot6eqsj@duck.com and i can share it in email.

tom93 commented 3 months ago

Great. You can view my email by appending ".patch" to any of my commits (example, from #137).

If you want to limit the logs, you can try to identify the relevant parts by searching for words such as "fossify", "fatal", "exception", and "error". Stack traces from crashes are typically the jackpot, they look like this:

--------- beginning of crash
12-17 11:23:20.841 30867 31904 E AndroidRuntime: FATAL EXCEPTION: Thread-18
12-17 11:23:20.841 30867 31904 E AndroidRuntime: Process: org.fossify.messages, PID: 30867
12-17 11:23:20.841 30867 31904 E AndroidRuntime: java.lang.OutOfMemoryError: Failed to allocate ... [or some other error/exception]
12-17 11:23:20.841 30867 31904 E AndroidRuntime:    at java.util.Arrays.copyOf(Arrays.java:3257)
12-17 11:23:20.841 30867 31904 E AndroidRuntime:    at kb.m.a(SourceFile:12)
12-17 11:23:20.841 30867 31904 E AndroidRuntime:    at kb.f.i(SourceFile:19)
12-17 11:23:20.841 30867 31904 E AndroidRuntime:    at kb.v.t(SourceFile:8)
12-17 11:23:20.841 30867 31904 E AndroidRuntime:    ...
12-17 11:23:20.841 30867 31904 E AndroidRuntime:    at java.lang.Thread.run(Thread.java:923)
12-17 11:23:20.842 1507 7831 W ActivityTaskManager: Force finishing activity org.fossify.messages/.activities.SettingsActivity

The short names in lines such as "at kb.m.a(SourceFile:12)" are not sensitive, they refer to locations in the source code and can be converted to the full names (see https://github.com/FossifyOrg/Messages/issues/6#issuecomment-2000602127).

I just noticed that you only get the "Importing..." message. There should also be an "Importing successful" message after that. I'm pretty certain that you are not seeing the second message (and that the dialog is not closing) due to an uncaught exception (as in the above example), which is very promising.

Abcd1234-dot commented 3 months ago

Yes exactly, I am not getting "import successful" message as it appears while exporting like "export successful", that's why i guess some underlying error? By the way I sent you logs via email and I literally don't understand a single word in the log file. You may find crashing errors for Play services, please ignore them.

tom93 commented 3 months ago

Hmm. I had a look at the log that you sent, there were no relevant crashes, but based on the debug messages of the form "D/MmsSmsProvider( 2504): getSingleAddressId: insert new canonical_address for xxxxxx, _id=58" it looks like it is importing, just doing it really slowly (the timestamps show it was importing for ~4 minutes and then it seem like you killed the app).

I suspect the calls to contentResolver.insert() are slow, which should be fairly easy to fix (use batching and bulkInsert). Edit: The code that checks if the SMS already exists also makes a query, might be a little trickier to also do that in bulk.

If you're up for an experiment, I'm curious what would happen if you try importing the smaller backup again and just leave it running for a couple of hours (e.g. overnight when you're not using your phone). I predict that the import will succeed eventually.

Alternatively, just wait until I write a PR.

Abcd1234-dot commented 3 months ago

I could try new back-up, but a silly question, what would happen if I try to restore backup while already having those messages in database? thats question for my new backup. (I am already messing with my important sms which is risky :') ) (I might try backing up messages again and deleted all existing messages then try to import again in mean time) and I have to admit when I opened .json file directly on my desktop (old backup) (not .txt file) it got frozen by saying too much data on json file (file is still about 5 MB but it contains last 2-3 years of sms which i expect to be very large number) but theoretically import should not be that slow as export was very quick when i backed up those sms! (took about a minute as I remember).

And yeah, I tried couple of times to import before getting logs and i fairly waited around 5 minutes to 10 minutes.

tom93 commented 3 months ago

The importer is supposed to detect whether the messages already exists, and skip importing ones that already exist.

Please don't do anything risky though (the ideal situation would be to import on a test device; I don't advise deleting the messages on your main device, even though in theory it should be possible to restore from the backup).

The import won't necessarily take the same amount of time as the export, because the code is different and might be slower. One idea is to try only importing SMS or only importing MMS (it's unclear which would be faster -- MMS are large but I assume there are fewer of them). Edit: And again, just leave the import running for as long as you can. You can keep the logcat open to get an idea of what's happening (the MmsSmsProvider messages are the ones to look for, I think you will get one per conversation; if you see new ones popping up in the log, that means the import is chugging along).

I've thought some more about making the import faster. It should be fairly easy to implement bulk insert & query, so feel free to just wait a couple of days until I put that together. (Implementing proper error handling that tries to import all the messages even if some of them cause errors is trickier, but that part isn't required for testing.)

Re opening the .json file, I'm not sure which program you are using to open it, but you can definitely open it with Notepad/WordPad (even though the extension is .json rather than .txt), you just need to use right click > "Open with", or temporarily rename the file. It just won't look pretty (and it might be too large for Notepad; but WordPad should be able to handle it).

Abcd1234-dot commented 3 months ago

I do not use MMS tho, only sms so i can test for that only. Alright, I will wait for your PR.

tom93 commented 3 months ago

I wrote a patch to do bulk insert (branch "import-bulk", commit) but unfortunately it only reduces the time by about 30% (when processing new messages; existing messages get skipped really quickly with the patch). Edit: That was on Android 8. On Android 14 the reduction is ~60%. Both were in an emulator.

I don't know if the import can be done faster, because most of the time is spent in Android's Telephony provider and that's not code that we can control. I guess the best approach is to provide better progress indication (example).

If you want to test it out, here is an APK: unofficial-messages-for-issue-88-fast-import-with-debug.apk.zip. In addition to the bulk insert patch, this APK also contains another patch (branch "import-bulk-debug", commit) that adds debugging messages to give some indication of progress. It also includes the patch from earlier to allow importing files with incorrect MIME type.

If you import using that APK, you should see a sequence of debug messages (they will appear on the screen and also be written to the log -- look for lines that contain "XXX"). Quick background: The patch divides the backup into "chunks", each with 999 messages, and imports one chunk at a time. The debug messages look like this:

Importing backup with 3000 messages This is the total number of SMS/MMS messages in the backup file. Bulk checking existing messages This means it's checking which messages from the first chunk have already been imported. Getting thread ID These debug messages are not interesting. Getting thread ID Writing a batch of 999 messages (skipping 0 existing messages) This means it's started to import the first chunk. It will take a while (~20 seconds for me) and it will seem like nothing is happening. Just wait. Bulk checking existing messages This means the first chunk was imported and it's now processing the second chunk. Writing a batch of 999 messages (skipping 0 existing messages) Bulk checking existing messages Writing a batch of 999 messages (skipping 0 existing messages) Bulk checking existing messages Writing a batch of 3 messages (skipping 0 existing messages) Finished import This means it finished. It's easy to miss the on-screen message, so I prefer looking at the logs.

If the messages in the chunk have already been imported, then the debug message will instead say Writing a batch of 0 messages (skipping 999 existing messages). Note that the on-screen debug messages get queued up, and might be displayed a while after the app produced them if it produced several in a row; on the other hand, the logs show the debug messages instantly. Edit: Apparently this behaviour depends on the Android version -- looks like newer versions display the messages all at once, piled on top of each other.

Abcd1234-dot commented 3 months ago

Your fix works. Import was successful this time and took around 5 minutes to import my old backup. all the previous sms are here since 2021.

I guess the best approach is to provide better progress indication (example).

there was no progress indication in notification tho.

but it introduced a new issue. all the imported messages are showing up randomly in the app! and not sorted by date i.e. latest ones on top and oldest ones on bottom. on home page of app, it just shows time under the sms thread (which should be the time when it actually received but instead it shows the time of successful import) and when i open that thread it shows correct date/time. see example below.

Below picture is top message on home page of app and you can see the time under the thread is the time when the backup was successfully imported. 8735db0f-fe5d-468a-9f03-74724d2388f7

the following picture is the same sms thread opened and you can see correct date on top which is from 2021! 9cd9eb1c-8c28-431b-8d17-05aea575685e

it is same for all the messages imported today! the already existing ones are still showing correct date on home page of app tho.

so i believe the home page acts like (newly imported) sms was received on the time/date of importing.

tom93 commented 3 months ago

Your fix works.

Great!

I guess the best approach is to provide better progress indication (example).

there was no progress indication in notification tho.

Yeah, the notification was just an idea. The APK I built only has the debug messages (which should appear on-screen as toasts like the original "Importing..." message).

all the imported messages are showing up randomly in the app! and not sorted by date i.e. latest ones on top and oldest ones on bottom.

Yeah, I noticed that when I was testing. I'll split that off into a new issue. Update: #146

Abcd1234-dot commented 3 months ago

So we can successfully close this issue when your PR gets merged?

Yeah, I noticed that when I was testing. I'll split that off into a new issue.

I hope you will start working on that as soon as possible as i already messed up with existing sms :') I have backed up before testing and i hope to restore them properly soon.

tom93 commented 3 months ago

So we can successfully close this issue when your PR gets merged?

Yes, it's better to create separate issues to track the new bugs, and allow this issue to be closed automatically when #137 is merged (that's the fix for the greyed-out files issue you originally reported).

I hope you will start working on that as soon as possible as i already messed up with existing sms :') I have backed up before testing and i hope to restore them properly soon.

Well, I'm not a project member (as I said before), so don't pin your hopes on me. I did spot something that might help and I was going to test that, but if it doesn't work out I'll probably leave it to the devs.

tom93 commented 3 months ago

I found a workaround for the timestamp issue (which turns out to be an Android bug) and built a new APK with all the relevant patches, including a patch for a serious deletion issue I discovered along the way. The code is in branch "all-import-fixes" of my fork (all commits, actual workaround). Here is the APK: unofficial-messages-all-import-fixes.apk.zip

It should display the same debugging messages as before, but with a new debug message of the form "Fixing dates for N conversations" at the very end.

Update: As @Abcd1234-dot pointed out below, my workaround doesn't work for threads that only contain MMS. It turns out that this is due to a bug in Android, see below. (Also, my hack to update existing conversations only applies to SMS, but even if I did the same for MMS it probably wouldn't help due to the Android bug.)

Abcd1234-dot commented 3 months ago

I will revert back with results after testing it.

Abcd1234-dot commented 3 months ago

I can confirm that your patches fix almost all the import export issues (including time stamp, selecting .json file stored anywhere on device) for SMS.

I think time stamp issue still exist for MMS but I am not 100% sure so someone needs to cross verify it.

Another thing thing I want to mention is, app takes time to reflect new changes especially if large number of changes i.e. large backup is restored. To cross verify it, following are steps I followed :

Install the debug apk provided by @tom93 while keeping Official fossify message app.
Set debug app as default app.
Restore large backup
Open Official fossify message app and confirm the pop up to set it as default app.
Observe if the changes are reflected or when they are reflected.

After restoring backup successfully and having all sms since 2021, whenever i launch official Fossify messages app, it takes around 5 seconds to get open and showing threads which was not happening before (not with previous versions of SMT app and fossify app).

I want to report something. If I export using the last apk you provided, it is not appending .json to export file (i tried export using native android files & fossify file manager), but still able to select and import the same file successfully without issue.

I also support the idea you earlier proposed that a notification showing progress of import/export.

tom93 commented 3 months ago

Great.

I think time stamp issue still exist for MMS but I am not 100% sure so someone needs to cross verify it.

You're right. My last APK included a temporary patch that updates the conversation timestamp even if the imported message already exists, but I only did that for SMS (not for MMS). If you are importing a new MMS though, then the conversation timestamp should be updated.

Another thing thing I want to mention is, app takes time to reflect new changes especially if large number of changes i.e. large backup is restored.

whenever i launch official Fossify messages app, it takes around 5 seconds to get open and showing threads which was not happening before

You should probably create a new issue for that. I tried to reproduce this; the first time I open the app it takes several seconds to load the new messages, but the next time I open the app it's near-instant (with 3000 messages across 1000 conversations, based on my example backup from #143). If the app is slow for you each time you open it then that's unacceptable, but if it's only slow during the first launch then it might be inevitable (would still be nice if it could be faster or showed progress, so still worth filing an issue).

If I export using the last apk you provided, it is not appending .json to export file

Yes, I didn't bother including the .json suffix patch in my last APK, sorry for the confusion.

Abcd1234-dot commented 3 months ago

If you are importing a new MMS though, then the conversation timestamp should be updated.

In my previous test, (there are some MMS from carrier provider in my old backup, you can check them in previously shared screenshots) When i restored my old backup (along with all other later backups i created), the time stamp was not fixed for MMS, results were same as i reported previously with those screenshot above.

You should probably create a new issue for that. I tried to reproduce this; the first time I open the app it takes several seconds to load the new messages, but the next time I open the app it's near-instant (with 3000 messages across 1000 conversations, based on my example backup from https://github.com/FossifyOrg/Messages/issues/143). If the app is slow for you each time you open it then that's unacceptable, but if it's only slow during the first launch then it might be inevitable (would still be nice if it could be faster or showed progress, so still worth filing an issue).

I think it was related with size of MMS, when cleared app data and also set MMS size to no limit in app settings, it got fixed.

I guess now we can finally say that whole import export issue along with time stamp can be closed when your PR gets merged successfully?

tom93 commented 3 months ago

the time stamp was not fixed for MMS

I tested this and I can reproduce this issue. It's due to a bug in Android -- the code here only updates the date and snippet for threads that contain at least one SMS, so it skips over threads that only contain MMS. Ideally this should be reported to Google but I'm not planning to do this myself.

There is not much that can be done from the app's side. The app could ignore the date & snippet reported by Android and instead recompute them on the app's side, but that seems like too much work to me.

Abcd1234-dot commented 3 months ago

If it is OS own issue, I guess we can do nothing to fix it. I thank you alot for your time and efforts to fix one of the fundamental functions of app.

BlastboomStrice commented 2 months ago

I found a workaround for the timestamp issue (which turns out to be an Android bug) and built a new APK with all the relevant patches, including a patch for a serious deletion issue I discovered along the way. The code is in branch "all-import-fixes" of my fork ([all commits](https://github.com/FossifyOrg/Messages #/compare/master...d480fdc019f5528edc75910b4091be2f0e5bd612), actual workaround). Here is the APK: unofficial-messages-all-import-fixes.apk.zip

I can confirm that this worked for me too! Device: Xiaomi Mi 10 (umi) Android 14 OS: HyperOS Thank you very much!!

Edit: Well... turns out it restores 1050 out of 1968 messages. Hmm. Its something, but still. It may have to do with the fact that I have two sim cards. Edit2: I fixed it! Turns out the .json file had subscriptionId's 0, 1, 2 and 3, while I only have 2 cards. I dont know exactly how they were split like that, it probably has to do with older improper imports. To fix it, I replaced subscriptionId":0 with subscriptionId":2 and subscriptionId":3 with subscriptionId":1 essentially merging the 4 IDs to 2 IDs.😄

FossifyOrg / Messages