OutOfMemory error when parsing

[carlos-mg89]

Hi and thanks for sharing your work,

I'm writing this issue since I'm a bit confused when using the minimal json package due to an OutOfMemory error.

I'm using it in an Android project to read a sort of big (10MB) JSON file. And to do that, I thought using a Stream would be the appropriate way, in order to avoid memory issues, but it doesn't seem to be the case. Or at least, I don't know how to make it work like that.

This is my code for reading the JSON file and storing the result in a variable (jsonValue):

val fileName = "json_array.json"
var jsonValue: JsonValue? = null
val returnString = StringBuilder()
var fIn: InputStream? = null
var isr: InputStreamReader? = null
var input: BufferedReader? = null

try {
    fIn = assets.open(fileName)
    isr = InputStreamReader(fIn)

    jsonValue = Json.parse(isr)

} catch (e: Exception) {
    e.message
} finally {
    try {
        if (isr != null)
            isr!!.close()
        if (fIn != null)
            fIn!!.close()
    } catch (e2: Exception) {
        e2.message
    }
}

This code is working on my phone and on an Android Virtual Device, however in some devices is not working due to OutOfMemory errors.

I had understood that to avoid them when reading the JSON file, a Stream reader had to be used, and I'm using an InputStreamReader. Perhaps I'm getting the error since I'm trying to return the parsed JSON and the allocation is the reason of the OutOfMemory error? If that's the case, then I imagine I should be processing the results as I read the JSON file. And store it, for example, in an SQLite database (that is stored in disk and not in memory).

This is the error log:

07-09 03:04:58.120: W/art(20574): Throwing OutOfMemoryError "Failed to allocate a 430 byte allocation with 1173328 free bytes and 1145KB until OOM; failed due to fragmentation (required continguous free 32768 bytes for a new buffer where largest contiguous free 0 bytes)" (recursive case)
07-09 03:04:58.120: W/art(20574): "main" prio=5 tid=1 Runnable
07-09 03:04:58.120: W/art(20574):   | group="main" sCount=0 dsCount=0 obj=0x76e43598 self=0xf4f36500
07-09 03:04:58.120: W/art(20574):   | sysTid=20574 nice=0 cgrp=default sched=0/0 handle=0xf75f7d54
07-09 03:04:58.120: W/art(20574):   | state=R schedstat=( 0 0 0 ) utm=884 stm=14 core=7 HZ=100
07-09 03:04:58.120: W/art(20574):   | stack=0xff5c5000-0xff5c7000 stackSize=8MB
07-09 03:04:58.120: W/art(20574):   | held mutexes= "mutator lock"(shared held)
07-09 03:04:58.120: W/art(20574):   at java.util.ArrayList.add(ArrayList.java:118)
07-09 03:04:58.120: W/art(20574):   at com.eclipsesource.json.JsonObject.add(JsonObject.java:336)
07-09 03:04:58.120: W/art(20574):   at com.eclipsesource.json.Json$DefaultHandler.endObjectValue(Json.java:382)
07-09 03:04:58.120: W/art(20574):   at com.eclipsesource.json.Json$DefaultHandler.endObjectValue(Json.java:331)
07-09 03:04:58.120: W/art(20574):   at com.eclipsesource.json.JsonParser.readObject(JsonParser.java:247)
07-09 03:04:58.120: W/art(20574):   at com.eclipsesource.json.JsonParser.readValue(JsonParser.java:177)
07-09 03:04:58.120: W/art(20574):   at com.eclipsesource.json.JsonParser.readArray(JsonParser.java:212)
07-09 03:04:58.120: W/art(20574):   at com.eclipsesource.json.JsonParser.readValue(JsonParser.java:174)
07-09 03:04:58.120: W/art(20574):   at com.eclipsesource.json.JsonParser.readObject(JsonParser.java:246)
07-09 03:04:58.120: W/art(20574):   at com.eclipsesource.json.JsonParser.readValue(JsonParser.java:177)
07-09 03:04:58.120: W/art(20574):   at com.eclipsesource.json.JsonParser.readArray(JsonParser.java:212)
07-09 03:04:58.120: W/art(20574):   at com.eclipsesource.json.JsonParser.readValue(JsonParser.java:174)
07-09 03:04:58.120: W/art(20574):   at com.eclipsesource.json.JsonParser.parse(JsonParser.java:152)
07-09 03:04:58.120: W/art(20574):   at com.eclipsesource.json.JsonParser.parse(JsonParser.java:114)
07-09 03:04:58.120: W/art(20574):   at com.eclipsesource.json.Json.parse(Json.java:320)
07-09 03:04:58.120: W/art(20574):   at com.myapp.com.MapApplication.readWayPointsFile(MapApplication.kt:93)
07-09 03:04:58.120: W/art(20574):   at com.myapp.com.MapApplication.onCreate(MapApplication.kt:45)
07-09 03:04:58.120: W/art(20574):   at android.app.Instrumentation.callApplicationOnCreate(Instrumentation.java:1037)
07-09 03:04:58.120: W/art(20574):   at android.app.ActivityThread.handleBindApplication(ActivityThread.java:6322)
07-09 03:04:58.120: W/art(20574):   at android.app.ActivityThread.access$1800(ActivityThread.java:222)
07-09 03:04:58.120: W/art(20574):   at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1862)
07-09 03:04:58.120: W/art(20574):   at android.os.Handler.dispatchMessage(Handler.java:102)
07-09 03:04:58.120: W/art(20574):   at android.os.Looper.loop(Looper.java:158)
07-09 03:04:58.120: W/art(20574):   at android.app.ActivityThread.main(ActivityThread.java:7230)
07-09 03:04:58.120: W/art(20574):   at java.lang.reflect.Method.invoke!(Native method)
07-09 03:04:58.120: W/art(20574):   at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1230)
07-09 03:04:58.120: W/art(20574):   at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1120)

Is that true? Or am I doing something wrong related on the way that I'm parsing the JSON with the MinimalJSON library?

Thanks in advance

[carlos-mg89]

I have tried using the GSON package to do the same that I was doing with Minimal JSON, and even though I was forced to perform a careful parsing of the JSON file, it allowed me to read the JSON file, without having the OutOfMemory issues, since I only wanted to read the file to do some processing, which could be done without having to store all the JSON values. Actually, what I wanted is to parse it to store it later in a SQLite database, and thanks to a real buffered reading like GSON offers it was possible to do so.

Example:

val inputStream: InputStream? = assets.open(fileName)
val inputStreamReader: InputStreamReader? = InputStreamReader(inputStream)
val jsonReader = JsonReader(inputStreamReader)

jsonReader.beginArray()

while (jsonReader.hasNext()) {
    // Do the manual parsing here, however it will be with a real buffered reader, so you won't incur on OutOfMemory exceptions
}

jsonReader.endArray()
jsonReader.close()