search

TNC-NMFO / NWLAND

carbon accounting model
0 stars 0 forks source link

caland() run on COGEOPROC – out of memory error when writing output #74

Closed aj1s closed 3 years ago

aj1s commented 3 years ago

Attempted replication of @sbassett's successful run of CALAND.r on COGEOPROC resulted in another java memory error (see also #69 and #62).

write_caland_inputs.r completed in < 1 min.

An initial run of CALAND.r ran then returned: Error: OutOfMemoryError (Java): GC Overhead limit exceeded Although this run took 3 hrs 50 min, it appears the error occurred when loading the input workbooks, as no incremental progress was indicated in the command prompt.

A second attempt of CALAND.r did complete the projections through 2050 before hitting a memory error when writing the outputs: Error in .jcall(“RJavaTools”, “Z”, “hasMethod”, .jast(x, “java/lang/Object”),: java.lang.OutofMemoryError: GC overhead limit exceeded.

> xlcMemoryReport()
Error in .jcall("RJavaTools", "Ljava/lang/Object;", "invokeMethod", cl, : java.lang.OutOfMemoryError: GC Overhead limit exceeded
> xlcFreeMemory()
> xlcMemoryReport()
Amount of free memory in the Java Virtual Machine (JVM):  306.1638MB

Java parameters specified in the run script for CALAND.r are : options(java.parameters = "-d64 -Xms50g -Xmx200g" ) but may be preempted by those specified in the run script for write_caland_inputs.r (if run in the same R session): options(java.parameters = "-Xmx32g" )

CALAND.r run in a new R session completed successfully (on the baseline and an alternative scenario). 

xlcMemoryReport()
Amount of free memory in the JVM: 4294.155 mb
xlcFreeMemory()
xlcMemoryReport()
Amount of free memory in the JVM: 16994.87 mb

plot_caland.r then ran successfully.

aj1s commented 3 years ago

Java memory allocation in 2021_05_28_testingWriteInputsWithPlaceholderData_noHere.r should be changed from:

this enables java to use up to 16GB of memory for reading and writing excel files options(java.parameters = "-Xmx32g" ) # originally set to "-Xmx8g" to

this enables java to use up to 400GB of memory for reading and writing excel files options(java.parameters = "-d64 -Xms50g -Xmx200g" ) # originally set to "-Xmx8g"