NewVM App Crashes after sitting idle for 10 minutes

[GoogleCodeExporter]

I can reproduce this consistently, but it is a pain because it takes a while.  
I have an app that crashes after 10 minutes or so of sitting idle.  The log 
just shows "Memory Warning", then Xcode loses the connection to the phone 
(because the app crashed).

Currently experiencing this with a real-world app.  Will try to whittle it down 
to a simpler test case.   There is a memory leak somewhere.

Crashes do not occur with the old VM.

Original issue reported on code.google.com by steve.ha...@codenameone.com on 31 Jan 2015 at 1:59

[GoogleCodeExporter]

Notes:  I can see the memory increase steadily over time in Xcode.  Starts at 
about 50Megs.  Goes up to about 160 over the course of about 10 minutes.  Then 
crashes.

Jan 31 06:15:38 iPhone com.apple.xpc.launchd[1] 
(UIKitApplication:ca.weblite.comicchecklist.lite[0x3836][2617]) <Notice>: 
Service exited due to signal: Killed: 9
Jan 31 06:15:38 iPhone SpringBoard[106] <Warning>: Application 
'UIKitApplication:ca.weblite.comicchecklist.lite[0x3836]' exited abnormally via 
signal.
Jan 31 06:15:39 iPhone ReportCrash[2621] <Error>: task_set_exception_ports(B07, 
400, D03, 0, 0) failed with error (4: (os/kern) invalid argument)
Jan 31 06:15:39 iPhone ReportCrash[2621] <Notice>: Saved report to 
/Library/Logs/CrashReporter/JetsamEvent-2015-01-31-061539.ips
Jan 31 06:15:39 iPhone UserEventAgent[17] <Notice>: jetsam: kernel termination 
snapshot being created

Original comment by steve.ha...@codenameone.com on 31 Jan 2015 at 2:20

[GoogleCodeExporter]

I have managed to narrow down the issue in my app, and I have created a 
workaround for the fix.  I believe the issue may be the same as issue #1304.  I 
will be posting a minimal test case shortly.

Original comment by steve.ha...@codenameone.com on 3 Feb 2015 at 3:26

[GoogleCodeExporter]

OK.  Here is a minimal test case that shows the problem.  This runs a 
background thread the goes in an infinite loop (but it waits for 200ms in each 
iteration).  Memory grows without bounds until the app is killed.

Original comment by steve.ha...@codenameone.com on 3 Feb 2015 at 6:36

Attachments:

• Issue1328Test.java

[GoogleCodeExporter]

Original comment by steve.ha...@codenameone.com on 3 Feb 2015 at 6:36

[GoogleCodeExporter]

Thanks for the test case, I made a fix to the per-thread allocator code to 
trigger and wait for a GC when there are too many allocations. This Seems to 
have fixed the issue.

Original comment by shai.almog on 4 Feb 2015 at 8:53

• Changed state: Fixed

[GoogleCodeExporter]

I just pulled the latest and tried another build.  It still seems to grow 
without bounds.  It grows a little slower, and it seems to find plateaus at 
various points.  If I comment out the t.start() (i.e. I don't run the thread), 
memory is flat at about 15 Megs.  If I enable the thread, then it grows 
constantly...  after about 4 or 5 minutes, it's up to 150 Megs.  

Original comment by st...@weblite.ca on 4 Feb 2015 at 4:40

• Changed state: Accepted

[GoogleCodeExporter]

I saw some of that too. I think its related to the 1304 issue and I think that 
is a memory leak not so much a lack of GC problem. Identifying this will be 
tough though...

Original comment by shai.almog on 4 Feb 2015 at 4:51

[GoogleCodeExporter]

OK, I made a fix that solved a major memory leak in the GC and effectively 
stopped memory growth in this test case.

Original comment by shai.almog on 5 Feb 2015 at 12:27

• Changed state: Fixed

[GoogleCodeExporter]

Issue 1303 has been merged into this issue.

Original comment by shai.almog on 5 Feb 2015 at 5:04

[GoogleCodeExporter]

I have confirmed that this fixes the unlimited growing issue.  It now grows up 
until it reaches about 35 megs of memory and holds stable.  If I never start 
the thread, the memory usage is under 20 megs at start and stays that way - 
never growing.  The extra 15 megs just to run this thread seems a little high.  
 Does the GC only kick in after a certain amount of memory is reached?  

Original comment by st...@weblite.ca on 6 Feb 2015 at 5:23

[GoogleCodeExporter]

Yes, the GC is pretty lazy normally. It will run every minute unless memory 
runs low.

Original comment by shai.almog on 6 Feb 2015 at 6:46

[GoogleCodeExporter]

As a comparison, I just ran it on the old VM.  App stays constant throughout 
entire run at 12.5 megs.

Original comment by st...@weblite.ca on 6 Feb 2015 at 8:44

[GoogleCodeExporter]

Interesting. I'll probably try fixing this rather than 1304 since this test 
case is much simpler.

Original comment by shai.almog on 6 Feb 2015 at 8:48

[GoogleCodeExporter]

I've found a big memory leak. Dealloc for images is never invoked including for 
the splash screen. Not sure why this would happen and why it would happen in 
the new VM and not in the old...

Original comment by shai.almog on 8 Feb 2015 at 8:29

[GoogleCodeExporter]

OK, that's 10mb that probably affected the old VM too... Still probably not the 
real reason.

Original comment by shai.almog on 8 Feb 2015 at 10:32

[GoogleCodeExporter]

I'm leaving this issue now. I'm sure there is waste in the new VM vs. the old 
VM e.g. thread overhead is larger and per object overhead is larger too. But 
I'm not so sure its a big deal. Most of that is justified (e.g. tracking the 
object stacks takes RAM), but most of the difference seen in the memory monitor 
is just different GC behavior. Specifically the current GC never deletes young 
objects whereas the old GC makes an effort to delete young objects. 

Original comment by shai.almog on 8 Feb 2015 at 11:52

[GoogleCodeExporter]

I have beefed up the test case to amplify the problem so as to demonstrate that 
this is more than just a case of object stacks taking RAM.

This new test case works as follows:

1. It has a button "Start Threads".  When you click this button, it spawns 10 
threads (of the form included in the original test case).
2.  You can click "Stop Threads" at any time to stop all of the threads that 
were created.  
3. Repeat.

In the new VM, after clicking Start threads the first time, memory steadily 
grows to about 30 megs then holds steady.  Clicking "Stop Threads" will not 
release any memory.  Memory holds firm at 30 megs.  Clicking start threads 
again will grow memory to about 55 megs.

You can repeat this until the program ultimately crashes at around 275 megs of 
RAM (after only 3 or 4 cycles).

On the old VM the memory stays constant around 13 megs no matter how many runs.

Original comment by st...@weblite.ca on 10 Feb 2015 at 6:09

• Changed state: Accepted

Attachments:

• Issue1328Test.java

[GoogleCodeExporter]

Original comment by shai.almog on 11 Feb 2015 at 6:28

• Changed state: Fixed

[GoogleCodeExporter]

Just ran the test again with latest changes and I'm seeing the same behaviour.  
App crashes after 2 to 3 "start/stop" cycles.

Original comment by st...@weblite.ca on 11 Feb 2015 at 4:58

• Changed state: Accepted

[GoogleCodeExporter]

Actually, if I use instruments to check Persistent memory, the memory looks ok 
(goes up to 25 megs while running and back down to 13 when threads stop).  I 
was getting confused by the "Real Memory" in the activity monitor.

Original comment by st...@weblite.ca on 12 Feb 2015 at 12:23

• Changed state: Fixed

[GoogleCodeExporter]

Something about this test wasn't sitting right with me.  When I debug the heap 
I noticed that several hundred thousand ArrayLists were being created.  Since 
the threads had wait(200) calls, there shouldn't have been this many.  So I 
looked into Object.wait() and, sure enough, it wasn't calculating its wait time 
correctly.  Instead of waiting 200ms, it was usually waiting only 2 or 3 ms.  I 
fixed this issue:
https://code.google.com/p/codenameone/source/detail?r=2121

And this makes a dramatic difference.  It may help performance and battery 
consumption in many other contexts also.

Original comment by st...@weblite.ca on 12 Feb 2015 at 1:45