Excessive memory consumption in parse()

[GoogleCodeExporter]

Summary:
I'm using JSON Simple in our Android application. I recently noticed I was
getting occasional OutOfMemory errors, and it turned out that
JSONParser.parse() is the problem. Parsing this simple JSON document
(http://api.qype.com/v1/places/27539/reviews.json?page=1 -- you need a
consumer key to actually see that) will cause a memory consumption of over
10 MB.

What steps will reproduce the problem?
I'm not sure whether this happens outside the Android environment. But what
I did is this:

    private JSONObject parseJsonReply(InputStream data) throws ParseException,
            ConnectionFailedException {

        if (modelName == null || modelListName == null) {
            String className = getClass().getSimpleName();
            int idx = className.indexOf("Parser");
            modelName = className.substring(0, idx).toLowerCase();
            modelListName = modelName + "s";
        }

        BufferedReader reader = new BufferedReader(new
InputStreamReader(data));
        StringBuilder sb = new StringBuilder();

        try {
            String line = reader.readLine();
            while (line != null) {
                sb.append(line);
                line = reader.readLine();
            }
        } catch (IOException e) {
            throw new ConnectionFailedException(e);
        }

        return (JSONObject) jsonParser.parse(sb.toString());
    }

What version of the product are you using? On what operating system?
json-simple 1.1 on Android 1.5 (HTC Hero)

Original issue reported on code.google.com by m.kaepp...@gmail.com on 27 Nov 2009 at 11:35

[GoogleCodeExporter]

You don't have to convert the stream into a String to pass it to JSONParser.
JSONParser accepts a Reader as its parameter.

Could you comment out the last line and then test it again?

return (JSONObject) jsonParser.parse(sb.toString());

And could you help to print out the result of sb.toString() and post it here 
before
passing it to JSONParser? That will be very helpful to analyze this issue.

Thanks.

Original comment by fangyid...@gmail.com on 28 Nov 2009 at 4:10

[GoogleCodeExporter]

That didn't help, unfortunately. I also can't log the StringBuilder contents, 
because
adb just cuts off strings which are too long to log, but I've attached the JSON 
data
served by our web service.

It's not a problem with that specific document though. I've seen this happening 
all
over the place.

Original comment by m.kaepp...@gmail.com on 30 Nov 2009 at 9:20

Attachments:

• reviews.json

[GoogleCodeExporter]

Thanks a lot for the data.
It's strange that from my initial testing (using a very simple method), nothing 
seems
to be special.(Local JDK is 1.6. What is the version of the JDK/JRE in your 
environment?)

Here's the code:
{{{{
public static void main(String[] args) throws Exception{
        long t = System.currentTimeMillis();
        long m = Runtime.getRuntime().freeMemory();
    Object obj = JSONValue.parse(new FileReader("reviews.json"));
        long delta = System.currentTimeMillis() - t;
        long deltaM = m - Runtime.getRuntime().freeMemory();
        System.out.println(delta + "ms");
        System.out.println(deltaM/1024 + "K");
    System.out.println(obj);
}
}}}}

Here's the result:
79ms
537K
(The resulting JSON text)

I also used YourKit 7.5 to do memory profiling and got the following data:
Overall objects:
All live objects: 13,025 Shallow size: 1,098,824 bytes Retained size: 1,098,824 
bytes

Dominant objects:
char[] 2,902 objects, shallow size: 429,656
java.lang.String 3,132 objects, shallow size: 75,168

Thanks.

Original comment by fangyid...@gmail.com on 30 Nov 2009 at 11:07

[GoogleCodeExporter]

I just replaced JSON simple with the reference JSON implementation from json.org
(which is shipped with Android), and it doesn't suffer from this problem.

There must be something leaking (and massively so -- considering it turns a 40k 
JSON
string into 12 megabytes), in JSONParser.parse().

However, to not waste your time:
It may well be that this is in fact something related to Apache Harmony, the
open-source Java implementation that powers Android. I have been running into
problems on Android several times now where Harmony turned out to be the source 
of
the problem (it's scary sometimes how buggy it can be at its core 
infrastructure).

I guess the best thing to do would be:
Write a simple test case which parses a JSON file, and profile it both using 
the Sun
JVM and the Harmony JVM.

The Android profiler unfortunately isn't detailed enough to tell me exactly 
which
method/object is actually consuming so much memory, but it shows me 10-12MB 
worth of
character data being allocated during a parse(), so it must be something 
related to
reading/buffering character data.

Original comment by m.kaepp...@gmail.com on 30 Nov 2009 at 11:11

[GoogleCodeExporter]

Thanks a lot for your information.
I'll test it in Harmony JVM.

Original comment by fangyid...@gmail.com on 30 Nov 2009 at 11:20

[GoogleCodeExporter]

One solution would be to just use the lexer. I've posted some code to Fang that 
extends the lexer and uses it 
similar to the XMLStreamReader in Java 1.6, effectivly saving lots of RAM when 
processing large amounts of 
JSON. Let me know and I'll post it again.

Original comment by karl.wet...@gmail.com on 30 Nov 2009 at 4:59

[GoogleCodeExporter]

Hi,

The following patch should fix the OutOfMemory problem. At least, it works for 
my JVM.

--- a/src/org/json/simple/parser/Yylex.java
+++ b/src/org/json/simple/parser/Yylex.java
@@ -577,7 +577,7 @@ int getPosition(){
           }
         case 25: break;
         case 4:
-          { sb.delete(0, sb.length());yybegin(STRING_BEGIN);
+          {  sb = null; sb = new StringBuffer(); yybegin(STRING_BEGIN);
           }
         case 26: break;
         case 16:

Glen

Original comment by glen...@gmail.com on 16 May 2012 at 7:25

[GoogleCodeExporter]

Hi glen.tw,

I've applied it to the code. Thanks a lot for your contribution.
Unfortunately I cannot validate it on Android Phone because I don't have one. 
Can any of you help validate it and let me know the result?

Thanks,
Yidong. 

Original comment by fangyid...@gmail.com on 16 May 2012 at 1:56

• Changed state: Fixed