search

gdtk-uq / gdtk

The Gas Dynamics Toolkit (GDTk) is a set of software tools for simulating high speed fluid flow, maintained at The University of Queensland and the University of Southern Queensland, Australia.
https://gdtk.uqcloud.net/
Other
59 stars 15 forks source link

Prepartitioned grid import for massively parallel job - very slow time integration #21

Closed manukamin closed 1 year ago

manukamin commented 1 year ago

Hello,

We have been testing Eilmer's capabilities in a step-by-step manner, to assess its suitability for our requirements. We have previously conducted 2D supersonic internal flow simulations using Eilmer successfully. The grid as well as block partitioning was done within Eilmer for the simple geometry.

Now, we have created a block structured grid for a 3D scramjet geometry. The grid size is approximately 150 million, divided into approximately 3200 blocks. Grid is generated and also partitioned into blocks externally using GridPro. All blocks have an almost uniform number of grid points (~50000 cells) to ensure uniform load on all processors.

Grid and connectivity information is exported out of GridPro in the SU2 format, and the grid is then imported into Eilmer using the grids = importGridproGrid(gproGrid, scale) function.

While the simulation begins and runs successfully, the progress is prohibitively slow. We were able to run barely 600 time steps in 4 hours of runtime, on 3192 cores!

The code output seems to suggest that MPI threads have been initialized. So I wonder why the progress is so slow. Has the code been used for massively parallel jobs before? My previous experiences have been using Eilmer's internal geometry creation and grid partitioning algorithms. I don't know if I am missing something when I import the grid and block connectivity information from an external source.

I have attached my input deck and the solver on-screen output here for reference.

Thanks config_details.zip

rjgollan-on-github commented 1 year ago

Hi Manu,

It's good you started stepwise on your journey. Can I suggest a few more intermediate steps? For example 1M cells or 10M cells before in 3D before jumping to 150M. Perhaps just truncate your domain.

As for your specific import issue, there are two possible workflows with GridPro grids. I think you have mixed the two workflows.

  1. Using GridPro grids in GridPro native format. There is an example in our collection that shows this approach. That is: gdtk/examples/eilmer/3D/gridpro-import. When you are using importGridproGrid() function call in your script, you are using this approach. This will read all elemental blocks from the original grid you built. You will need that same number of cores for your simulation.
  2. Using GridPro grids in SU2 format. When we do this, we typically partition the SU2 grid using Metis. For this type of workflow, look at the example: gdtk/examples/eilmer/2D/sharp-cone-20-degrees/usg-su2

To answer your question about slow or not, I think you need to step up systematically in cell size to see if there's a scaling issue. It's hard for me to reason about an isolated 150M cell grid.

manukamin commented 1 year ago

Hi Rowan, thanks for your feedback. There is a small correction to my previous comment : I meant to say that I am writing the grid out in format native to GridPro (.tmp), and not in SU2 format. My apologies for that. So no, I don't think I am mixing up the two workflows.

And as far as your suggestions on truncating the geometry is concerned : I shall try that. Perhaps I should also perform a test to see how the code scales.

Thanks again! I shall mark this issue as closed.