Crash during export - Githubissues

I'm trying to export ~10000 SMS and ~5500 MMS from a phone that has been miraculously un-bricked (long, irrelevant story). The SMS portion of this task proceeds rapidly without issue. The MMS portion gets to message 780 or so and then the app closes and disappears abruptly without finishing.

I tried turning OFF the "Include (encoded) binary MMS data" option and the entire job runs to completion, however I would like to have the binary MMS data as well so this is not a solution.

I know very little about app development but it seems to me that the problem occurs inside ImportExportMessages.kt when an exception is thrown if there is a problem accessing the binary data.

It would be extremely helpful to me if this could fail more gracefully. I don't have the luxury of not having the corruption that seems to exist on that single message on this device.

I envision something like: 1) prompting the user if they would like to proceed exporting without the missing binary data, 2) printing a warning in the status line that the binary_data field is missing in some (or N) of the MMS messages, 3) a debug toggle option in the settings that would allow the app to quietly drop missing MMS data

Any help would be greatly appreciated!

Thanks for reporting this. Please provide a logcat, so I can try to figure out what's going wrong. Your suggestions are good, but I'd like to understand what's failing before proceeding.

Good idea. It seems to be memory related:

12-14 00:53:08.903 29245 29245 E AndroidRuntime: FATAL EXCEPTION: main 12-14 00:53:08.903 29245 29245 E AndroidRuntime: Process: com.github.tmo1.sms_ie, PID: 29245 12-14 00:53:08.903 29245 29245 E AndroidRuntime: java.lang.OutOfMemoryError: Failed to allocate a 46091152 byte allocation with 25165824 free bytes and 24MB until OOM, target footprint 200885184, growth limit 201326592 12-14 00:53:08.903 29245 29245 E AndroidRuntime: at java.lang.StringFactory.newStringFromChars(StringFactory.java:260) 12-14 00:53:08.903 29245 29245 E AndroidRuntime: at java.lang.StringFactory.newStringFromBytes(StringFactory.java:245) 12-14 00:53:08.903 29245 29245 E AndroidRuntime: at java.lang.StringFactory.newStringFromBytes(StringFactory.java:65) 12-14 00:53:08.903 29245 29245 E AndroidRuntime: at android.util.Base64.encodeToString(Base64.java:457) 12-14 00:53:08.903 29245 29245 E AndroidRuntime: at com.github.tmo1.sms_ie.ImportExportMessagesKt.mmsToJSON(ImportExportMessages.kt:221) 12-14 00:53:08.903 29245 29245 E AndroidRuntime: at com.github.tmo1.sms_ie.ImportExportMessagesKt.access$mmsToJSON(ImportExportMessages.kt:1) 12-14 00:53:08.903 29245 29245 E AndroidRuntime: at com.github.tmo1.sms_ie.ImportExportMessagesKt$mmsToJSON$1.invokeSuspend(Unknown Source:17) 12-14 00:53:08.903 29245 29245 E AndroidRuntime: at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) 12-14 00:53:08.903 29245 29245 E AndroidRuntime: at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106) 12-14 00:53:08.903 29245 29245 E AndroidRuntime: at kotlinx.coroutines.internal.LimitedDispatcher.run(LimitedDispatcher.kt:42) 12-14 00:53:08.903 29245 29245 E AndroidRuntime: at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:95) 12-14 00:53:08.903 29245 29245 E AndroidRuntime: at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:570) 12-14 00:53:08.903 29245 29245 E AndroidRuntime: at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:749) 12-14 00:53:08.903 29245 29245 E AndroidRuntime: at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:677) 12-14 00:53:08.903 29245 29245 E AndroidRuntime: at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:664) 12-14 00:53:08.903 29245 29245 E AndroidRuntime: Suppressed: kotlinx.coroutines.DiagnosticCoroutineContextException: [StandaloneCoroutine{Cancelling}@6a1a1a4, Dispatchers.Main]

I'm really unfamiliar with this type of development but it appears that the MMS content won't fit in memory a second time as a base64 string, and that's what is triggering the exception.

It seems to me that a possible solution could be to rewrite that section of code to use Base64OutputStream instead of Base64.encodeToString() as described in: https://stackoverflow.com/questions/22163825/outofmemory-when-creating-base64-string-in-java

Thank you - I had arrived at a similar conclusion. I've looked at the relevant SMS I/E code, and it seems like it should be pretty straightforward to switch to Base64OutputStream (although it may be necessary to add a ByteArrayOutputStream into the mix as well, since the Base64 encoded data needs to be supplied to a JsonWriter). I hope this can be done without duplicating the data in memory. When I get a chance, I'll take a crack at implementing it.

Okay, I implemented a version of SMS I/E using Base64OutputStream. Would you please test whether it solves the problem?

Thanks! I tested it out and unfortunately it seems to crash on the same message during the export step.

Here's the logcat:

12-26 09:54:50.549 1433 2321 I ActivityTaskManager: START u0 {cmp=com.github.tmo1.sms_ie/.SettingsActivity} from uid 10247 12-26 09:56:05.605 28026 28045 I hub.tmo1.sms_i: Waiting for a blocking GC ProfileSaver 12-26 09:56:05.619 28026 28045 I hub.tmo1.sms_i: WaitForGcToComplete blocked ProfileSaver on HeapTrim for 13.960ms 12-26 09:56:34.963 28026 28136 I hub.tmo1.sms_i: Waiting for a blocking GC Alloc 12-26 09:56:34.976 28026 28136 I hub.tmo1.sms_i: WaitForGcToComplete blocked Alloc on HeapTrim for 13.128ms 12-26 09:56:34.976 28026 28136 I hub.tmo1.sms_i: Starting a blocking GC Alloc 12-26 09:56:34.993 28026 28038 I hub.tmo1.sms_i: Clamp target GC heap from 199MB to 192MB 12-26 09:56:35.163 28026 28136 I hub.tmo1.sms_i: Starting a blocking GC Alloc 12-26 09:56:35.163 28026 28136 I chatty : uid=10247(com.github.tmo1.sms_ie) identical 2 lines 12-26 09:56:35.170 28026 28136 I hub.tmo1.sms_i: Starting a blocking GC Alloc 12-26 09:56:35.180 28026 28136 I hub.tmo1.sms_i: Clamp target GC heap from 199MB to 192MB 12-26 09:56:35.180 28026 28136 I hub.tmo1.sms_i: Alloc concurrent copying GC freed 6(16KB) AllocSpace objects, 0(0B) LOS objects, 8% free, 175MB/192MB, paused 20us total 10.313ms 12-26 09:56:35.180 28026 28136 I hub.tmo1.sms_i: Forcing collection of SoftReferences for 43MB allocation 12-26 09:56:35.181 28026 28136 I hub.tmo1.sms_i: Starting a blocking GC Alloc 12-26 09:56:35.191 28026 28136 I hub.tmo1.sms_i: Clamp target GC heap from 199MB to 192MB 12-26 09:56:35.191 28026 28136 I hub.tmo1.sms_i: Alloc concurrent copying GC freed 700(39KB) AllocSpace objects, 0(0B) LOS objects, 8% free, 175MB/192MB, paused 16us total 10.660ms 12-26 09:56:35.193 28026 28136 W hub.tmo1.sms_i: Throwing OutOfMemoryError "Failed to allocate a 46091152 byte allocation with 16874328 free bytes and 16MB until OOM, target footprint 201326592, growth limit 201326592" (VmSize 5238064 kB) 12-26 09:56:35.193 28026 28136 I hub.tmo1.sms_i: Starting a blocking GC Alloc 12-26 09:56:35.197 28026 28136 I hub.tmo1.sms_i: Starting a blocking GC Alloc 12-26 09:56:35.206 28026 28136 I hub.tmo1.sms_i: Clamp target GC heap from 199MB to 192MB 12-26 09:56:35.207 28026 28136 I hub.tmo1.sms_i: Alloc concurrent copying GC freed 4(16KB) AllocSpace objects, 0(0B) LOS objects, 8% free, 175MB/192MB, paused 19us total 9.782ms 12-26 09:56:35.207 28026 28136 I hub.tmo1.sms_i: Forcing collection of SoftReferences for 43MB allocation 12-26 09:56:35.207 28026 28136 I hub.tmo1.sms_i: Starting a blocking GC Alloc 12-26 09:56:35.217 28026 28136 I hub.tmo1.sms_i: Clamp target GC heap from 199MB to 192MB 12-26 09:56:35.217 28026 28136 I hub.tmo1.sms_i: Alloc concurrent copying GC freed 5(16KB) AllocSpace objects, 0(0B) LOS objects, 8% free, 175MB/192MB, paused 20us total 10.674ms 12-26 09:56:35.218 28026 28136 W hub.tmo1.sms_i: Throwing OutOfMemoryError "Failed to allocate a 46091152 byte allocation with 16874328 free bytes and 16MB until OOM, target footprint 201326592, growth limit 201326592" (VmSize 5238064 kB) 12-26 09:56:35.253 28026 28026 E AndroidRuntime: Process: com.github.tmo1.sms_ie, PID: 28026 12-26 09:56:35.253 28026 28026 E AndroidRuntime: at com.github.tmo1.sms_ie.ImportExportMessagesKt.mmsToJSON(ImportExportMessages.kt:223) 12-26 09:56:35.253 28026 28026 E AndroidRuntime: at com.github.tmo1.sms_ie.ImportExportMessagesKt.access$mmsToJSON(ImportExportMessages.kt:1) 12-26 09:56:35.253 28026 28026 E AndroidRuntime: at com.github.tmo1.sms_ie.ImportExportMessagesKt$mmsToJSON$1.invokeSuspend(Unknown Source:17) 12-26 09:56:35.255 1433 9748 W ActivityTaskManager: Force finishing activity com.github.tmo1.sms_ie/.MainActivity

Looks a little different this time. I'm not sure what your new code looks like but there must be some subtlety here where the entire stream is still being copied into a buffer temporarily and hitting OOM.

Poking around the web, there seem to be others having this same basic issue (encoding a large file to base64 then writing to json leading to OOM).

Oh, well - perhaps even a single copy of the data is too large for the available memory. We can't just stream straight from the on-disk data to the JSON output file, since we have to go through a JSONWriter, which seems to require an in-memory representation of the data.

Due to this problem as well as other considerations, I'm increasingly leaning toward the conclusion that embedding base64 encoded binay data in the main JSON output is a poor design choice, and it would be preferable to store such data in separate files. I'll probably eventually make this change, but it will be disruptive and break compatibility with earlier versions of the code, so I'm not ready to do it just yet. In the meantime, I'll look into implementing one or more of the more graceful failure modes that you suggested earlier.

Well, I'd sure appreciate that. I don't know for sure but I'm hoping there is only one message causing this issue on my device.

Just thinking about a change to storing binary data in separate files for a moment.. in some ways it'd be more convenient to have them stored in a subfolder alongside the json. Maybe tucked inside a parallel zip file, just to keep it neat. As an added benefit with that approach, one could quickly browse or view all the images or videos sent/received with regular software and no need to first parse and decode the json. In many cases, people are just looking to preserve images sent and received and don't need a blow-by-blow crime scene level of reconstruction- although that is certainly possible if needed using the metadata preserved in the json.

On my phone, the MMS attachments seem to be located in a folder like: "/data/user_de/0/com.android.providers.telephony/app_parts" with a filename like "PART_1632017884371_IMG_4980.jpg". As all the attachments are stored in that same folder (not subfolders), it appears that the filenames are already unique so there shouldn't be naming collisions to deal with.

So to elaborate on that.. in addition to writing messages_2022-12-26.json, it would write messages_2022-12-26.zip containing PART_1632017884371_IMG_4980.jpg and similar files, for example. The json writer could simply omit the "binary_data" key during an export operation. And during an import operation, if the json reader finds that key is missing it could enter a mode where it tries to open a file with the same name as the json input but the a .zip extension instead, and looks inside that zip file for the unique filename listed in the "_data" key instead (e.g. PART_1632017884371_IMG_4980.jpg).

With that approach you could maintain (forward) compatibility by checking for the presence or absence of the "binary_data" key. Older json files with the key present created by earlier versions of your software could be read in as before, while newly created json files would be read along with the sidecar zip file.

Just throwing out some ideas!

Oh, well - perhaps even a single copy of the data is too large for the available memory.

I realized that this is incorrect - it's evident from the first logcat that the unencoded data fits into memory, and it's only the attempt to create a second (base64 encoded) object on the heap that fails. Perhaps the reason the streaming version fails as well is that the conversion from the ByteArrayOutputStream to a string is also creating a second object in memory. I'm going to try whipping up a version that definitely avoids this by doing the conversion via streaming to a temporary file, and then reading the temporary file into memory.

Just thinking about a change to storing binary data in separate files for a moment.. in some ways it'd be more convenient to have them stored in a subfolder alongside the json.

Agreed.

With that approach you could maintain (forward) compatibility by checking for the presence or absence of the "binary_data" key. Older json files with the key present created by earlier versions of your software could be read in as before, while newly created json files would be read along with the sidecar zip file.

I appreciate the suggestion, but I'm not inclined to do this. I'd like to keep the app code as clean as possible, and only maintain one single file format. To the extent that I want to maintain compatibility between versions, I'll probably just include external conversion tools in Python to convert between old and new formats.

Just throwing out some ideas!

Thank you! They are good ones, and I'll probably wind up implementing at least some of them.

Okay, here's a version of the app that uses a temporary file for the base64 encoding, and so should only be making one copy of the binary data in memory. Does this solve the problem?

Well, a partial success?

This version got past the previous crash around MMS message 780 and made it to message 2400 or so before closing suddenly:

12-29 22:41:23.135 1555 4243 I ActivityTaskManager: START u0 {act=android.intent.action.MAIN cat=[android.intent.category.LAUNCHER] flg=0x10300000 cmp=com.github.tmo1.sms_ie/.MainActivity} from uid 10234 12-29 22:41:23.157 1555 1580 I ActivityManager: Start proc 8682:com.github.tmo1.sms_ie/u0a311 for pre-top-activity {com.github.tmo1.sms_ie/com.github.tmo1.sms_ie.MainActivity} 12-29 22:45:16.179 8682 8694 I hub.tmo1.sms_i: Background concurrent copying GC freed 4275(209KB) AllocSpace objects, 15(28MB) LOS objects, 75% free, 24MB/99MB, paused 8.495ms total 63.948ms 12-29 22:45:25.746 8682 8694 I hub.tmo1.sms_i: Background young concurrent copying GC freed 2568(172KB) AllocSpace objects, 13(30MB) LOS objects, 25% free, 81MB/109MB, paused 10.750ms total 39.624ms 12-29 22:45:26.335 8682 8694 I hub.tmo1.sms_i: Background concurrent copying GC freed 2841(162KB) AllocSpace objects, 19(105MB) LOS objects, 75% free, 24MB/99MB, paused 9.356ms total 83.040ms 12-29 22:45:37.848 8682 8694 I hub.tmo1.sms_i: Background concurrent copying GC freed 748(61KB) AllocSpace objects, 3(23MB) LOS objects, 61% free, 59MB/155MB, paused 12.621ms total 58.550ms 12-29 22:45:40.664 8682 8694 I hub.tmo1.sms_i: Background young concurrent copying GC freed 10406(439KB) AllocSpace objects, 28(99MB) LOS objects, 56% free, 67MB/155MB, paused 11.707ms total 76.654ms 12-29 22:45:42.907 8682 8694 I hub.tmo1.sms_i: Background young concurrent copying GC freed 5263(397KB) AllocSpace objects, 94(110MB) LOS objects, 66% free, 48MB/144MB, paused 8.323ms total 62.816ms 12-29 22:46:17.456 8682 8694 I hub.tmo1.sms_i: Background young concurrent copying GC freed 5951(307KB) AllocSpace objects, 28(79MB) LOS objects, 45% free, 84MB/155MB, paused 6.439ms total 52.178ms 12-29 22:46:31.762 8682 8768 I hub.tmo1.sms_i: Waiting for a blocking GC Alloc 12-29 22:46:31.779 8682 8694 I hub.tmo1.sms_i: Background young concurrent copying GC freed 1480(95KB) AllocSpace objects, 0(0B) LOS objects, 0% free, 234MB/234MB, paused 46.817ms total 123.328ms 12-29 22:46:31.780 8682 8768 I hub.tmo1.sms_i: Waiting for a blocking GC Alloc 12-29 22:46:31.826 8682 8694 I hub.tmo1.sms_i: Clamp target GC heap from 330MB to 256MB 12-29 22:46:31.827 8682 8768 I hub.tmo1.sms_i: WaitForGcToComplete blocked Alloc on HeapTrim for 64.829ms 12-29 22:46:31.827 8682 8768 I hub.tmo1.sms_i: Starting a blocking GC Alloc 12-29 22:46:31.827 8682 8768 I hub.tmo1.sms_i: Starting a blocking GC Alloc 12-29 22:46:31.832 8682 8768 I hub.tmo1.sms_i: Alloc young concurrent copying GC freed 0(15KB) AllocSpace objects, 0(0B) LOS objects, 8% free, 234MB/256MB, paused 27us total 5.026ms 12-29 22:46:31.832 8682 8768 I hub.tmo1.sms_i: Starting a blocking GC Alloc 12-29 22:46:31.844 8682 8768 I hub.tmo1.sms_i: Clamp target GC heap from 330MB to 256MB 12-29 22:46:31.844 8682 8768 I hub.tmo1.sms_i: Alloc concurrent copying GC freed 454(30KB) AllocSpace objects, 0(0B) LOS objects, 8% free, 234MB/256MB, paused 31us total 11.722ms 12-29 22:46:31.844 8682 8768 I hub.tmo1.sms_i: Forcing collection of SoftReferences for 51MB allocation 12-29 22:46:31.844 8682 8768 I hub.tmo1.sms_i: Starting a blocking GC Alloc 12-29 22:46:31.853 8682 8768 I hub.tmo1.sms_i: Clamp target GC heap from 330MB to 256MB 12-29 22:46:31.853 8682 8768 I hub.tmo1.sms_i: Alloc concurrent copying GC freed 2537(88KB) AllocSpace objects, 0(0B) LOS objects, 8% free, 234MB/256MB, paused 21us total 9.429ms 12-29 22:46:31.854 8682 8768 W hub.tmo1.sms_i: Throwing OutOfMemoryError "Failed to allocate a 54114896 byte allocation with 22468952 free bytes and 21MB until OOM, target footprint 268435456, growth limit 268435456" (VmSize 14106312 kB) 12-29 22:46:31.854 8682 8768 I hub.tmo1.sms_i: Starting a blocking GC Alloc 12-29 22:46:31.857 8682 8768 I hub.tmo1.sms_i: Starting a blocking GC Alloc 12-29 22:46:31.867 8682 8768 I hub.tmo1.sms_i: Clamp target GC heap from 330MB to 256MB 12-29 22:46:31.867 8682 8768 I hub.tmo1.sms_i: Alloc concurrent copying GC freed 1(15KB) AllocSpace objects, 0(0B) LOS objects, 8% free, 234MB/256MB, paused 19us total 9.412ms 12-29 22:46:31.867 8682 8768 I hub.tmo1.sms_i: Forcing collection of SoftReferences for 51MB allocation 12-29 22:46:31.867 8682 8768 I hub.tmo1.sms_i: Starting a blocking GC Alloc 12-29 22:46:31.876 8682 8768 I hub.tmo1.sms_i: Clamp target GC heap from 330MB to 256MB 12-29 22:46:31.876 8682 8768 I hub.tmo1.sms_i: Alloc concurrent copying GC freed 2(64B) AllocSpace objects, 0(0B) LOS objects, 8% free, 234MB/256MB, paused 18us total 8.951ms 12-29 22:46:31.876 8682 8768 W hub.tmo1.sms_i: Throwing OutOfMemoryError "Failed to allocate a 54114896 byte allocation with 22452568 free bytes and 21MB until OOM, target footprint 268435456, growth limit 268435456" (VmSize 14106312 kB) 12-29 22:46:31.956 8682 8682 E AndroidRuntime: Process: com.github.tmo1.sms_ie, PID: 8682 12-29 22:46:31.956 8682 8682 E AndroidRuntime: at com.github.tmo1.sms_ie.ImportExportMessagesKt.mmsToJSON(ImportExportMessages.kt:225) 12-29 22:46:31.956 8682 8682 E AndroidRuntime: at com.github.tmo1.sms_ie.ImportExportMessagesKt.access$mmsToJSON(ImportExportMessages.kt:1) 12-29 22:46:31.956 8682 8682 E AndroidRuntime: at com.github.tmo1.sms_ie.ImportExportMessagesKt$mmsToJSON$1.invokeSuspend(Unknown Source:17) 12-29 22:46:31.958 1555 1569 W ActivityTaskManager: Force finishing activity com.github.tmo1.sms_ie/.MainActivity 12-29 22:46:32.072 1555 4237 I ActivityManager: Process com.github.tmo1.sms_ie (pid 8682) has died: prcp TOP 12-29 22:46:32.073 1555 3340 I WindowManager: WIN DEATH: Window{4406bfa u0 com.github.tmo1.sms_ie/com.github.tmo1.sms_ie.MainActivity}

I'm guessing all that means even a single copy is too much to store in memory?

I wonder.. since base64 encodes 3 eight bit characters into 4 printable characters, would it be possible to process the base64 encoding of messages in fixed chunks of X megabytes, rather than reading and encoding the entire message in one go? It seems like such an approach could work as long as the buffer size is a multiple of 3.

since base64 encodes 3 eight bit characters into 4 printable characters, would it be possible to process the base64 encoding of messages in fixed chunks of X megabytes, rather than reading and encoding the entire message in one go? It seems like such an approach could work as long as the buffer size is a multiple of 3.

I don't think that would help - we're anyway doing pretty much that by using Base64OutputStream, but the problem is that we need to assemble the whole thing into a single string to feed to the JSONWriter.

Could switching to Gson (which supports streaming) help? https://www.amitph.com/java-parse-large-json-files/

We're already using JSON streaming (via Android's native JSONWriter), but the problem is that each token needs to exist completely in memory before it can be written out to the file. I don't know if GSON would eliminate that requirement, but in any event, I don't want to do a major code rewrite and introduce an external dependency just to get around this problem. I think that handling binary data separately from everything else is a much better solution.

Unfortunately the phone has bricked again and my message history is permanently lost.. (that is, everything after message 2400 or so).

It's disappointing that the Android model seems to artificially limit the memory available to applications so drastically. I'm sure that out of the 128 gb of ram on the device, there was sufficient room to store a ~50mb base64 encoded string.

Unfortunately the phone has bricked again and my message history is permanently lost.. (that is, everything after message 2400 or so).

I'm very sorry that we didn't get this resolved in time.

It's disappointing that the Android model seems to artificially limit the memory available to applications so drastically. I'm sure that out of the 128 gb of ram on the device, there was sufficient room to store a ~50mb base64 encoded string.

Your phone may have 128GB of internal (flash) storage, but it certainly does not have that much RAM. Nevertheless, I take your point. There is a way to ask Android to grant more memory to an app, but I was reluctant to use that method, for various reasons.

Yes, of course not. Was looking at "Memory 128GB/6GB RAM" in the specs and meant to type 6 GB but my fingers went to the wrong keys.

Interesting about the largeHeap option. I would have tried that once just to recover my data if I'd known about it, although it's not clear how to turn it on.

Hello. It seems that I have the same issue : ` type: crash osVersion: google/oriole/oriole:13/TQ1A.230105.002/2023011000:user/release-keys package: com.github.tmo1.sms_ie:13 process: com.github.tmo1.sms_ie processUptime: 6966 + 188 ms

java.lang.OutOfMemoryError: Failed to allocate a 148875448 byte allocation with 33554432 free bytes and 112MB until OOM, target footprint 184253456, growth limit 268435456 at java.util.Arrays.copyOf(Arrays.java:3670) at java.io.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:211) at kotlin.io.ByteStreamsKt.readBytes(IOStreams.kt:137) at com.github.tmo1.sms_ie.ImportExportMessagesKt.mmsToJSON(ImportExportMessages.kt:222) at com.github.tmo1.sms_ie.ImportExportMessagesKt.access$mmsToJSON(ImportExportMessages.kt:1) at com.github.tmo1.sms_ie.ImportExportMessagesKt$mmsToJSON$1.invokeSuspend(Unknown Source:17) at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106) at kotlinx.coroutines.internal.LimitedDispatcher.run(LimitedDispatcher.kt:42) at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:95) at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:570) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:749) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:677) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:664) Suppressed: kotlinx.coroutines.DiagnosticCoroutineContextException: [StandaloneCoroutine{Cancelling}@17e65a8, Dispatchers.Main] `

I apologize - fixing this properly is going to require a substantial change to the export / import format, involving storing MMS binary data outside the main JSON file. I plan to implement this when I get a chance.

Adding another crash report here, even though I think the cause has already been identified.

This is on a Pixel 6 with ~8300 SMS and ~5300 MMS.

2023-03-25 09:39:07.803 14777-14792 hub.tmo1.sms_ie         com.github.tmo1.sms_ie               I  Clamp target GC heap from 268MB to 256MB
2023-03-25 09:39:07.893 14777-14998 hub.tmo1.sms_ie         com.github.tmo1.sms_ie               I  Starting a blocking GC Alloc
2023-03-25 09:39:07.893 14777-14998 hub.tmo1.sms_ie         com.github.tmo1.sms_ie               I  Starting a blocking GC Alloc
2023-03-25 09:39:07.895 14777-14998 hub.tmo1.sms_ie         com.github.tmo1.sms_ie               I  Forcing collection of SoftReferences for 193MB allocation
2023-03-25 09:39:07.895 14777-14998 hub.tmo1.sms_ie         com.github.tmo1.sms_ie               I  Starting a blocking GC Alloc
2023-03-25 09:39:07.900 14777-14998 hub.tmo1.sms_ie         com.github.tmo1.sms_ie               I  Clamp target GC heap from 268MB to 256MB
2023-03-25 09:39:07.900 14777-14998 hub.tmo1.sms_ie         com.github.tmo1.sms_ie               W  Throwing OutOfMemoryError "Failed to allocate a 202530780 byte allocation with 87775184 free bytes and 83MB until OOM, target footprint 268435456, growth limit 268435456" (VmSize 14982200 kB)
2023-03-25 09:39:07.900 14777-14998 hub.tmo1.sms_ie         com.github.tmo1.sms_ie               I  Starting a blocking GC Alloc
2023-03-25 09:39:07.900 14777-14998 hub.tmo1.sms_ie         com.github.tmo1.sms_ie               I  Starting a blocking GC Alloc
2023-03-25 09:39:07.902 14777-14998 hub.tmo1.sms_ie         com.github.tmo1.sms_ie               I  Forcing collection of SoftReferences for 193MB allocation
2023-03-25 09:39:07.902 14777-14998 hub.tmo1.sms_ie         com.github.tmo1.sms_ie               I  Starting a blocking GC Alloc
2023-03-25 09:39:07.906 14777-14998 hub.tmo1.sms_ie         com.github.tmo1.sms_ie               I  Clamp target GC heap from 268MB to 256MB
2023-03-25 09:39:07.906 14777-14998 hub.tmo1.sms_ie         com.github.tmo1.sms_ie               W  Throwing OutOfMemoryError "Failed to allocate a 202530784 byte allocation with 87775208 free bytes and 83MB until OOM, target footprint 268435456, growth limit 268435456" (VmSize 14982200 kB)
2023-03-25 09:39:07.946 14777-14777 AndroidRuntime          com.github.tmo1.sms_ie               E  FATAL EXCEPTION: main
                                                                                                    Process: com.github.tmo1.sms_ie, PID: 14777
                                                                                                    java.lang.OutOfMemoryError: Failed to allocate a 202530784 byte allocation with 87775208 free bytes and 83MB until OOM, target footprint 268435456, growth limit 268435456
                                                                                                        at java.lang.StringFactory.newStringFromBytes(StringFactory.java:86)
                                                                                                        at java.lang.StringFactory.newStringFromBytes(StringFactory.java:65)
                                                                                                        at android.util.Base64.encodeToString(Base64.java:458)
                                                                                                        at com.github.tmo1.sms_ie.ImportExportMessagesKt.mmsToJSON(ImportExportMessages.kt:222)
                                                                                                        at com.github.tmo1.sms_ie.ImportExportMessagesKt.access$mmsToJSON(ImportExportMessages.kt:1)
                                                                                                        at com.github.tmo1.sms_ie.ImportExportMessagesKt$mmsToJSON$1.invokeSuspend(Unknown Source:17)
                                                                                                        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
                                                                                                        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
                                                                                                        at kotlinx.coroutines.internal.LimitedDispatcher.run(LimitedDispatcher.kt:42)
                                                                                                        at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:95)
                                                                                                        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:570)
                                                                                                        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:749)
                                                                                                        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:677)
                                                                                                        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:664)
                                                                                                        Suppressed: kotlinx.coroutines.DiagnosticCoroutineContextException: [StandaloneCoroutine{Cancelling}@f42be5d, Dispatchers.Main]
2023-03-25 09:39:07.948  1600-2243  ActivityTaskManager     system_server                        W    Force finishing activity com.github.tmo1.sms_ie/.MainActivity
2023-03-25 09:39:08.013  1600-3583  WindowManager           system_server                        I  WIN DEATH: Window{8c767c8 u0 com.github.tmo1.sms_ie/com.github.tmo1.sms_ie.MainActivity}
2023-03-25 09:39:08.016  1600-2243  ActivityManager         system_server                        I  Process com.github.tmo1.sms_ie (pid 14777) has died: fg  TOP 
2023-03-25 09:39:08.017  1600-3583  InputManager-JNI        system_server                        W  Input channel object '8c767c8 com.github.tmo1.sms_ie/com.github.tmo1.sms_ie.MainActivity (client)' was disposed without first being removed with the input manager!
2023-03-25 09:39:08.035  1600-1692  WindowManager           system_server                        W  Failed to deliver inset state change to w=Window{8c767c8 u0 com.github.tmo1.sms_ie/com.github.tmo1.sms_ie.MainActivity EXITING}
                                                                                                    android.os.DeadObjectException
                                                                                                        at android.os.BinderProxy.transactNative(Native Method)
                                                                                                        at android.os.BinderProxy.transact(BinderProxy.java:584)
                                                                                                        at android.view.IWindow$Stub$Proxy.insetsControlChanged(IWindow.java:473)
                                                                                                        at com.android.server.wm.WindowState.notifyInsetsControlChanged(WindowState.java:4013)
                                                                                                        at com.android.server.wm.InsetsStateController.lambda$notifyPendingInsetsControlChanged$4(InsetsStateController.java:351)
                                                                                                        at com.android.server.wm.InsetsStateController.$r8$lambda$An2IoiA3BeA5IWc6QwBOjKArM80(Unknown Source:0)
                                                                                                        at com.android.server.wm.InsetsStateController$$ExternalSyntheticLambda3.run(Unknown Source:2)
                                                                                                        at com.android.server.wm.WindowAnimator.executeAfterPrepareSurfacesRunnables(WindowAnimator.java:345)
                                                                                                        at com.android.server.wm.WindowAnimator.animate(WindowAnimator.java:226)
                                                                                                        at com.android.server.wm.WindowAnimator.lambda$new$1(WindowAnimator.java:106)
                                                                                                        at com.android.server.wm.WindowAnimator.$r8$lambda$fo2Nk5bb9hY3lvmHsTnTEwtZMbI(Unknown Source:0)
                                                                                                        at com.android.server.wm.WindowAnimator$$ExternalSyntheticLambda1.doFrame(Unknown Source:2)
                                                                                                        at android.view.Choreographer$CallbackRecord.run(Choreographer.java:1229)
                                                                                                        at android.view.Choreographer$CallbackRecord.run(Choreographer.java:1239)
                                                                                                        at android.view.Choreographer.doCallbacks(Choreographer.java:899)
                                                                                                        at android.view.Choreographer.doFrame(Choreographer.java:827)
                                                                                                        at android.view.Choreographer$FrameDisplayEventReceiver.run(Choreographer.java:1214)
                                                                                                        at android.os.Handler.handleCallback(Handler.java:942)
                                                                                                        at android.os.Handler.dispatchMessage(Handler.java:99)
                                                                                                        at android.os.Looper.loopOnce(Looper.java:201)
                                                                                                        at android.os.Looper.loop(Looper.java:288)
                                                                                                        at android.os.HandlerThread.run(HandlerThread.java:67)
                                                                                                        at com.android.server.ServiceThread.run(ServiceThread.java:44)
2023-03-25 09:39:08.052  1600-1692  WindowManager           system_server                        W  Exception thrown during dispatchAppVisibility Window{8c767c8 u0 com.github.tmo1.sms_ie/com.github.tmo1.sms_ie.MainActivity EXITING}
                                                                                                    android.os.DeadObjectException
                                                                                                        at android.os.BinderProxy.transactNative(Native Method)
                                                                                                        at android.os.BinderProxy.transact(BinderProxy.java:584)
                                                                                                        at android.view.IWindow$Stub$Proxy.dispatchAppVisibility(IWindow.java:536)
                                                                                                        at com.android.server.wm.WindowState.sendAppVisibilityToClients(WindowState.java:3481)
                                                                                                        at com.android.server.wm.WindowContainer.sendAppVisibilityToClients(WindowContainer.java:1228)
                                                                                                        at com.android.server.wm.WindowToken.setClientVisible(WindowToken.java:392)
                                                                                                        at com.android.server.wm.ActivityRecord.setClientVisible(ActivityRecord.java:6725)
                                                                                                        at com.android.server.wm.ActivityRecord.onAnimationFinished(ActivityRecord.java:7537)
                                                                                                        at com.android.server.wm.ActivityRecord.postApplyAnimation(ActivityRecord.java:5416)
                                                                                                        at com.android.server.wm.ActivityRecord.commitVisibility(ActivityRecord.java:5375)
                                                                                                        at com.android.server.wm.ActivityRecord.commitVisibility(ActivityRecord.java:5379)
                                                                                                        at com.android.server.wm.AppTransitionController.handleClosingApps(AppTransitionController.java:1147)
                                                                                                        at com.android.server.wm.AppTransitionController.handleAppTransitionReady(AppTransitionController.java:292)
                                                                                                        at com.android.server.wm.RootWindowContainer.checkAppTransitionReady(RootWindowContainer.java:970)
                                                                                                        at com.android.server.wm.RootWindowContainer.performSurfacePlacementNoTrace(RootWindowContainer.java:834)
                                                                                                        at com.android.server.wm.RootWindowContainer.performSurfacePlacement(RootWindowContainer.java:777)
                                                                                                        at com.android.server.wm.WindowSurfacePlacer.performSurfacePlacementLoop(WindowSurfacePlacer.java:177)
                                                                                                        at com.android.server.wm.WindowSurfacePlacer.performSurfacePlacement(WindowSurfacePlacer.java:126)
                                                                                                        at com.android.server.wm.WindowSurfacePlacer.performSurfacePlacement(WindowSurfacePlacer.java:115)
                                                                                                        at com.android.server.wm.WindowSurfacePlacer$Traverser.run(WindowSurfacePlacer.java:57)
                                                                                                        at android.os.Handler.handleCallback(Handler.java:942)
                                                                                                        at android.os.Handler.dispatchMessage(Handler.java:99)
                                                                                                        at android.os.Looper.loopOnce(Looper.java:201)
                                                                                                        at android.os.Looper.loop(Looper.java:288)
                                                                                                        at android.os.HandlerThread.run(HandlerThread.java:67)
                                                                                                        at com.android.server.ServiceThread.run(ServiceThread.java:44)
2023-03-25 09:39:08.449  1600-1691  ActivityTaskManager     system_server                        W  Activity top resumed state loss timeout for ActivityRecord{7115dac u0 com.github.tmo1.sms_ie/.MainActivity} t-1 f}}

Adding another crash report here, even though I think the cause has already been identified.

Thank you. I have been working on the new JSON / Zip format, which stores MMS binary data outside the JSON and thus entirely bypasses the need for the memory intensive base64 encoding. Message export is largely done; I just have to write the import code and then I'll be able to release a version for testing.

A pre-release version of the app with the new NDJSON / ZIP format is now available. It seems to work well, and should completely resolve this issue, but testing would certainly be appreciated.

tmo1 / sms-ie

Crash during export #84