FileSystemdDisk.freeMemoryAndFinalize 'kills' jvm with large heap

[GoogleCodeExporter]

program is trying to open a connection with CREATE=false and upon file not 
found, code forces 16 gc(s) and 16 finalizations in hopes of finalizing that 
one FileInputStream that was forgotten to be closed. When heap is large (>10G 
on 64bit Linux) jvm crawls and becomes unresponsive. Total time to open  (or 
rather legitimately fail with IOException) may be in the order of 20 minutes or 
more.

Original issue reported on code.google.com by kirilz...@gmail.com on 24 May 2011 at 12:35

[GoogleCodeExporter]

Hi,

I couldn't reproduce the problem so far (meaning, I couldn't get it to call 
System.gc()). What exactly should I do to trigger the System.gc() call? What 
database URL do I need to use, and are there other dependencies (what files / 
directories need to exist, does it happen only on one operating system or file 
system or JVM version)?

In any case, I will reduce the number of System.gc() calls to one.

Regards,
Thomas

Original comment by thomas.t...@gmail.com on 25 May 2011 at 6:52

[GoogleCodeExporter]

I think I was pretty specific on how to reproduce, not sure I can more
than that - put create=false in the url and don't have the file on
disk - you will hit IOException and then as per code line it will try
(by design) close all of the InputStreams in hopes of creating a file.

I don't really think reducing number of times you hit gc/finalizations
is good enough. I propose instead, making sure all InputStreams are
closed where they need to be closed and not try to outsmart JVM by
running finalizations in hopes of creating a file. I propose removing
'freeMemoryAndFinalize()' - really if you would be talking on the java
conference with 1000 people attending ,let's say, about this (method),
how would you feel?

-Kiril

Original comment by kirilz...@gmail.com on 26 May 2011 at 3:28

[GoogleCodeExporter]

Well, as I already wrote, I can't reproduce it. So it seems either you were not 
specific enough, or you forgot to say something.

Original comment by thomas.t...@gmail.com on 26 May 2011 at 5:27

[GoogleCodeExporter]

What is your database URL?
What version of H2 do you use?
What exception do you get?

Original comment by thomas.t...@gmail.com on 26 May 2011 at 5:28

[GoogleCodeExporter]

Thomas,

try jdbc:h2:file:<path>;ACCESS_MODE_DATA=r;IFEXISTS=TRUE. I think i
incorrectly stated CREATE=false (this is equivalent hsqldb property, I
am sure you are familiar)
I use 1.3.154
I don't get any exceptions, jvm slows down significantly.

Realistically, you should be able to reproduce the IOException that
triggers the FileSystemdDisk.freeMemoryAndFinalize method. If you have
a good simulator or an application that uses >50G of heap, maybe you
can see similar conditions that I am seeing and see that calculations
do take much longer (100x) when gc is running and perhaps doing
compacting (not sure if that is what causes such a huge slowdown, but
that is the only thing I can think of that makes process that normally
uses 64 cpus to go down to 1 for 10 minutes. Also, we have many
databases  and that is a pattern in our code: try to open a read-only
connection, do not create db, if not found create a database. So every
time we do that we hit gc hard.

I hope this helps you, however, just by looking at code  you may find
yourself asking if a loop that forces jvm to do full gc and
finalization in hopes of forcing some file to be closed is a good idea
or not - regardless of application. What are your thoughts even if you
are unable to reproduce this - can you share?

Original comment by kirilz...@gmail.com on 26 May 2011 at 7:32

[GoogleCodeExporter]

I see, you have used ACCESS_MODE_DATA=r.

Original comment by thomas.t...@gmail.com on 26 May 2011 at 9:07

[GoogleCodeExporter]

Setting the issue to 'invalid' because this is not a bug report.

The bug summary is "FileSystemdDisk.freeMemoryAndFinalize 'kills' jvm with 
large heap", so it sounds like you are 100% sure System.gc() is called, and it 
sounds like that was the reason why your application was slow. But now I 
believe you just *guessed* the method was called, and you just *guessed* this 
was the reason why it was slow. I believe neither was System.gc() called, nor 
calling System.gc() would actually be a problem.

I suggest to provide a reproducible test case, or at least provide enough data 
so I can reproduce it. Don't write in the bug summary what you *think* the 
reason for the problem is if you are not sure. 

Of course you could also debug the code, set a breakpoint, provide a full 
thread dump, use java -verbose:gc. Also, what version of H2 do you use? What 
exception do you get?

Original comment by thomas.t...@gmail.com on 26 May 2011 at 4:43

• Changed state: Invalid