OutOfMemoryErrors on Windows can cause the Java virtual machine to exit

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Run the MemoryAllocationExhaustiveTest on Windows

What's happening is we are deliberately exhausting the entire heap, and
Java is unable to collect new memory.  However, the allocation is in the
following stack:

java failed allocation test
^
JNI transition

^
C++ JNIMemoryAllocator.cpp
^
C++ frames
^
C++ Swig interface
^
JNI transition
^
Java call to IAudioSamples.make

The java allocation on Windows will return null (and strangely not set an
exception).  Our C++ code will throw bad_alloc and the swig layer will
catch that and throw a pre-created OutOfMemoryError.  The problem occurs in
the final JNI transition back.  Java needs to allocate a little memory to
do that transition on Windows (which is actually a JVM bug), but because
the heap is exhausted the JVM throws a std::bad_alloc and since it doesn't
catch it, Windows terminates the JVM.

This is truly unfortunate; there is a potential (theory) way to fix this:
1) At process start-up time, allocate a 'parachute' of memory in C++ that
we keep around for the low-memory situation
2) When we are about to return to Java with a memory error, release the
parachute
3) On the next allocation, attempt to reallocate the parachute before
fullfilling the actual request.  If we fail, then throw out of memory again.

This unfortunately will not work in a multi-threaded scenario without
locking (which thread creates and destroys the parachute), and as of this
bug filing Xuggler thread-locking relies on calling back out to the JVM,
which will attempt a memory allocation, which will fail.  Horrible issue
really.

So for now we're punting.  If you get your JVM terminating because of
bad_alloc on windows, it's most likely because of a memory leak in your
code.  To try to debug, run on another OS, and that JVM should correctly
give you a memory dump.

For now, we're working around this by not running the exhaustive tests on
windows.

- Art

Original issue reported on code.google.com by art.cla...@gmail.com on 14 May 2009 at 9:24

[GoogleCodeExporter]

Actually there appear to be two issues at work here:

1) We were passing a C++ exception from libferry to libxuggler, which on Windows
doesn't work (as it crosses a DLL boundary and G++ 4.2.4 has an issue with 
that). 
This has been fixed for the specific exception to the MemoryAllocationTests now 
work.

2) There appears to be a Java bug when running lots of multiple threads, in low
memory situations, where OutOfMemoryErrors are thrown where the Windows JVM gets
unhappy and either crashes or exits.  The work around for this is to either 
increase
the available JVM memory (e.g. run the SERVER JVM from the JDK) or ensure you 
don't
run out of memory by releasing references).  We're closing this issue because 
really
it only fails now in situations where there is a bug in the users code (i.e. 
they are
leaking memory).

Original comment by art.cla...@gmail.com on 19 May 2009 at 1:38

• Changed state: Fixed