Interning String fields to save memory

[Armaxis]

Hello everyone, I was analyzing memory of our app using memory dump analysis tool in Android Studio and noticed that some of our objects takes a lot of space. For example we have 8000 instances of Recording object, which occupy about 12 Mb of memory - each object has about 30 String fields and 5 more Arrays with Strings. Most of the fields are configuration related, so they use same string values, and those normally should point to the same String instance in String's pool of objects. But it turned out that every single string was unique, and therefore we are wasting a lot of memory on String duplicates. Memory analysis tool showed that we have about 10000 String duplicates there.

So I started digging into problem and found out that Gson converter we use doesn't call .intern() on Strings, quote:

Gson doesn't use String.intern() which can cause unnecessary strain on the garbage collector. (Source)

So, I tried writing a simple intern() method in my Recording object and manually interning every single string and values inside arrays to see if there is a difference. Memory consumption went down from 13Mb to 6Mb. I haven't measured the performance of those method calls yet.

I wonder if there is a way to intern strings automatically using Retrofit (say, after object field are bound from raw JSON) or should it be something I put in custom TypeAdapter and then specify with JsonAdapter annotation?

Some discussions on topic I found: https://groups.google.com/forum/#!topic/google-gson/MVUagkzIUSk

https://github.com/google/gson/issues/618

We use retrofit2:2.0.0 with corresponding converter-gson plugin (gson 2.6.1)

[swankjesse]

I don’t think String.intern() is a good string pool for this purpose. Instead, consider making your own TypeAdapter<String> that uses a LinkedHashMap<String, String> for pooling. Or if you’re on Android, you can use LruCache<String, String>.

We won’t support this in Retrofit because string pooling tends to cause more problems than it solves. You’re a special case with 7 MiB of duplicated strings. Other applications that have that many strings don’t have them for very long – the strings are created and GC’d pretty quickly.

[Armaxis]

Thank you, Jesse, I will look into implementing shared LruCache for storing those. Although as I read more into both LruCache/String implementation it seems that synchronization might become a problem, since it will lock whenever working with that pool, so we might lose performance there. Anyway, worth a look.

We won’t support this in Retrofit because string pooling tends to cause more problems than it solves.

Sorry, I put it wrong, I proposed maybe doing this as optional feature to toggle on/off for specific fields. But since it's really a rare case, it might not worth it. Should I bring this up on Gson issue tracker or nah?

[swankjesse]

I recommend you build the TypeAdapter and then maybe blog about it? The code should be interesting to others in similar situations.

[SeniorZhai]

I need to get a lot of JSON from the file and insert it into the database. Many fields String are not repeated. This process consumes a lot of memory. Is there any good way to do this?