Out of memory always occurs on Runtime.getRuntime().exec - forking duplicates heap!

[rwest]

I noticed that we always run out of memory when starting an external process, like Fame, Dassl, Frankie - one of the fortran programs. It's never the java itself that runs out. I think this is the cause:

Runtime.getRuntime().exec allocates the process with the same amount of memory as the main. If you had you heap set to 1GB and try to exec then it will allocate another 1GB for that process to run.

See http://stackoverflow.com/questions/1124771/how-to-solve-java-io-ioexception-error12-cannot-allocate-memory-calling-runt

If we can solve this, we can double our available RAM.

[rwest]

Other pages that may help: http://developers.sun.com/solaris/articles/subprocess/subprocess.html http://www.thinkplexx.com/blog/linux-runtime-exec-fork-java-lang-processbuilder-start-java-io-ioexception-cannot-allocate-memory

How do we increase the amount of virtual memory available on Monch?

[sreihani]

I use Xmx to set the upper the bound. However, as long as the chemistry grows, the pre-determined upper limit will be hit. Unless pruning can delay this process.

[rwest]

Josh and I have just realised that we won't be able to increase the total (physical + virtual ) memory above 4 gb as Monch is an old 32 bit machine without Physical Address Extension.

Although we were getting into trouble recently for hogging all the RAM on Pharos without using the CPUs efficiently, it could prove a useful testing ground.

Another idea worth investigating is here: http://www.coderanch.com/t/419196/java/java/there-any-way-execute-Linux

Some advice on writing server apps: http://java.sun.com/docs/books/tutorial/networking/sockets/index.html

[sreihani]

I guess you (Richard) and I realized this issue almost one year ago.

[rwest]

Seyed, have you ever been able to set your Xmx to more than half of the available virtual memory? i.e. have you ever got it above 2gb?

[sreihani]

No. I was not able. I got an error message that java could not create VM.

[rwest]

Documented in 92730820de3e49a01332e68bf1b084d397a4a384

[rwest]

See the ideas from @ramanan in 4413bae5834e3b947d174e07fb99c3e4d51cedcb for how this problem could be tackled. (That commit solves it for GATPFit; now only FAME, DASSL, and Frankie to go!)