search

gdtk-uq / gdtk

The Gas Dynamics Toolkit (GDTk) is a set of software tools for simulating high speed fluid flow, maintained at The University of Queensland and the University of Southern Queensland, Australia.
https://gdtk.uqcloud.net/
Other
63 stars 17 forks source link

High ram usage during solution on a moderate mesh #28

Closed UNSWHilbert closed 1 year ago

UNSWHilbert commented 1 year ago

Dear Eilmer Team,

I have been learning Eilmer recently on a relatively simple case. I am only interested in the steady state behavior which is why I am running my case with backward_Euler. My current mesh is 10M cells (imported from .su2 file) and the ram usage while solving it using 30 cores is 124 GB. Is there a way I can decrease the RAM usage? 124GB is manageable at this stage, but it means we might run into trouble on a significant amount of our boxes is we go to to 20 or 30M cell.

Regards, Hilbert

uqngibbo commented 1 year ago

Hi Hilbert,

We have a few compile time flags you can try turning off to save some memory, depending on what you need, try some or all of the following:

MULTI_SPECIES_GAS=0 MULTI_T_GAS = 0 MHD=0 TURBULENCE=0

All of these are on by default but can be turned off when calling make. For example, to build e4shared, e4mpi, and e4zmpi without MHD or two temperature capability, you would do:

$ make WITH_MPI=1 WITH_COMPLEX_NUMBERS=1 MHD=0 MULTI_T_GAS=0 FLAVOUR=fast install

Let me know if you run into any trouble.

UNSWHilbert commented 1 year ago

Hi Nick,

I only just got around to testing your suggestion. It did save me a bit of RAM but nothing significant. I am mainly interested since we are about to buy a couple new boxes and I want to make sure the boxes have enough RAM on board. Given the CPU count we are going after I am hopeful we can run 20-30M cell grids. What would you recommend RAM wise for such grids?

Cheers, Hilbert

uqngibbo commented 1 year ago

That's disappointing.

I am planning to do some memory tuning at some point soon, so hopefully that will help. I guess if you can find 250 GB of RAM for your new boxes you'll be able to make 20M work, and possibly 30M once I can work on the memory.

Nick,

UNSWHilbert commented 1 year ago

HI Nick,

Thanks that Is roughly what I was thinking.

Hilbert,

uqngibbo commented 1 year ago

29ae23f91e9fb194f9ec9e2707edfe2d83bda047 should have improved this issue. Let me know if you notice any improvment.

UNSWHilbert commented 1 year ago

Just letting you know that on a 5.7M cell mesh that I have lying around the RAM usage went from 79.2 to 78.1 Gb, so roughly a 1 Gb improvement.