search

GEOS-ESM / GEOSctm

Fixture for chemical transport scenarios
Apache License 2.0
0 stars 2 forks source link

Running with GEOSchemchem uses high amounts of memory #19

Open kgerheiser opened 4 years ago

kgerheiser commented 4 years ago

Running on 6 processors and a single node at C48 I am seeing ~18 GB per process, and if emissions are turned on it won't even run. That, coupled with the slow memory leak, means I can only run for several hours before it crashes.

Upping the number of nodes and the number of processes does not help much, and the consumed memory remains constant at around 80% from 1 node and 6 processes or 10 nodes and 280 processes.

Considering this chemistry can run within the AGCM it doesn't seem right that the CTM should consume a whole 128 GB node worth of memory at C48.

I am currently investigating with TotalView's memory checker.

tclune commented 4 years ago

This sounds like a known issue. MAPL develop is much better with regard to leaking memory. (Actually it is the newer version of ESMF that improves it). But if you are seeing 18 GB/proc at the beginning of the run, there is a problem.

kgerheiser commented 4 years ago

Yes, it's using 18 GB at the start of the run