search

OpenModelica / OpenModelica

OpenModelica is an open-source Modelica-based modeling and simulation environment intended for industrial and academic usage.
https://openmodelica.org
Other
851 stars 308 forks source link

Abnormal memory consumption in SimCode with the NB #10231

Open casella opened 1 year ago

casella commented 1 year ago

I recently added HarmonicOscillator_N_819200 to the LargeTestSuite_NB job.

The system has 1.6 million variables, which are all states. Unfortunately the test fails, I guess because the allocated 60 GB are not enough.

Looking at the two smaller cases, HarmonicOscillator_N_204800 allocates 12 GB of memory for the SimCode phase, and HarmonicOscillator_N_409600 almost 25 GB. The scaling is not superlinear, which is good, but the code is probably quite inefficient memory-wise.

Consider that N = 409600 case has 0.8 M variables and 1.6 M non-zero entries in the Jacobian, each equation or non-zero Jacobian entry consumes about 10 kB of RAM. I'm not sure why, but I guess this amount of memory could probably be reduced by at least one order of magnitude without too much effort. Maybe more.

Unfortunately, even though the code to solve the equations is vectorized, the variables and the sparsity pattern in the generated code are still fully scalarized, and so is the init.xml file. Maybe the MetaModelica code that handles that is not very memory efficient.

This is essential to break the 1M state limit and head towards the 10M goal.

casella commented 1 year ago

@perost I'd like to also have your opinion, regarding the efficiency of MetaModelica for SimCode. Why do we consume so much memory (about 10KB) for each individual variable? Could there be some fundamental improvement to make this more efficient?

casella commented 1 year ago

See @adrpo's comment in #10400

casella commented 1 year ago

HarmonicOscillator_N_819200 now works, despite SimCode allocating a whooping 50 GB of memory. Apparently, the garbage collector can recover some of it.

I'll try an even larger model to see what happens. In any case, even if this memory can be recovered, all these allocations take time, so we should try to make the process more efficient.

casella commented 5 months ago

In the meantime, the memory allocation of SimCode for HarmonicOscillator_N_819200 was slightly reduced from 50 to 43 GB. Not much, but at least it's not growing 😅