Refinement of simplex meshes

[anicusan]

Dear Lethe authors,

Again, thank you for developing this library, we're finding it extremely powerful for our chemical engineering problems at Birmingham. I'm working with a complex case we're trying to run on 2560 CPUs: turbulent air flow through a column packed with pellets, each pellet having holes through them; for such a tortuous geometry, I've only been able to do a tetrahedralization with GMSH. The .msh file alone is 26 GB - and even that's a mesh too coarse for our Reynolds numbers...

For this case, additional node-local refinement would be necessary to not exhaust the RAM - or some other way to split a pre-refined mesh between nodes. From what I gathered off Google, deal.II can refine simplex meshes, right? What work would need to be done to include initial refinement functionality within source/core/grids.cc?

Thank you, Leonard

[blaisb]

Dear @anicusan, As always, we welcome issues such as these. As far as I am aware, refinement of simplex mesh does not work adequately in the context of simplex meshes (@peterrum maybe you can correct me if I am wrong). The fully distributed parallel triangulation is not made to have parts of its triangulation refined, so right now the simplex meshes have to remain static. At least, as far as I understand it, this is a current limitation of the Simplex implementation in deal.II.

An alternative might be to think about using the sharp-edge immersed boundary method that @Luckabarbeau has developed. The issue will be to represent the porous geometry, but we are working on a feature that might be able to achieve this in a nearby future (a month or two)..

For the time being, i'm afraid I don't know what would be a good solution for this problem.

PS: Do you have any images of what you are solving? I have rarely carried out Lethe simulations with over 1k core so I'm always happy to see it perform at this magnitude :)

[blaisb]

By the way, I could not help but notice that you and your colleagues have developed very cool meshes to use with Lethe (e.g. https://github.com/DanW97/Lethe-tools) . If you are interested, you could open a PR in the lethe-utils repository to fork them there :). This latter repo is a sandbox where we keep tons of tools.

The repo is there : https://github.com/lethe-cfd/lethe-utils

[peterrum]

As far as I am aware, refinement of simplex mesh does not work adequately in the context of simplex meshes (@peterrum maybe you can correct me if I am wrong). The fully distributed parallel triangulation is not made to have parts of its triangulation refined, so right now the simplex meshes have to remain static. At least, as far as I understand it, this is a current limitation of the Simplex implementation in deal.II.

@blaisb Yes you are correct.

It is possible to refine locally serial Triangulation objects in deal.II both for 2D and 3D and also to setup the AffineConstraints object for 2D (see Section 2.1 in https://dealii.org/deal94-preprint.pdf). For 3D, the AffineConstraints object cannot set up yet (but if we do not consider mixed meshes, the setup should be similar to 2D, with the difference that subfaces are triangles). What I described should also work for parallel::shared::Triangulation.

For parallel::fulllydistributed::Triangulation (which is the distributed triangulation class that supports simplex and mixed meshes), we don't have yet the full repartitioning functionality. What one can do is to work on locally refined meshes by copying over the cells over from a serial/shared triangulation. Section 2.2 in https://dealii.org/deal94-preprint.pdf shows that one can repartition p:f:T (but solution transfer, ... is missing).

To make adaptive p:f:T working (for simplex meshes), one would refine/coarsen the mesh locally, in the next step the refinement of the ghost cells have to be made consistent between processes (see https://github.com/dealii/dealii/pull/8834), and finally the mesh has to be repartition. Not too much is missing; however unfortunately I won't be able to work on this topic in the next months.

Let me point out three tests:

• https://github.com/dealii/dealii/blob/master/tests/simplex/hanging_nodes_01.cc -> hanging-node constraints for simplex meshes
• https://github.com/dealii/dealii/blob/master/tests/fullydistributed_grids/repartitioning_01.cc -> repartitioning
• https://github.com/dealii/dealii/blob/master/tests/multigrid-global-coarsening/multigrid_p_01.cc -> multigrid on locally-refined simplex meshes

Sorry for being able to help here more.

[anicusan]

I understand, thank you very much for your input! It's one of those projects where you're abusing every single piece of software, but it's almost doable. I'll try tethex to convert tets to hexas, fingers crossed...

We wrote a few geometry and case generation tools, but we weren't entirely happy with the code design - we were thinking of a general purpose interface for these geometries before asking about integration within lethe. If you don't mind bikeshedding, we could submit a pull request with the current state they're in - or include a link to them as we iterate their design. I was also prototyping some automatic Python API generation directly from the parameters C++ source files - but I'll start a conversation about it in a pull request once it's working.

What images would be most useful? The pellet designs are a collaborating company's secret (I know...), but I can share the workflow, usage statistics, perhaps a cutaway image after the packing - plus some difficult-to-reproduce incomplete feedback, such as slightly slow geometry loading; the memory usage becomes constant quite quickly, so perhaps some other step might need more parallelisation. Any suggestions for doing better detective work are more than welcome.

[blaisb]

I understand, thank you very much for your input! It's one of those projects where you're abusing every single piece of software, but it's almost doable. I'll try tethex to convert tets to hexas, fingers crossed...

We wrote a few geometry and case generation tools, but we weren't entirely happy with the code design - we were thinking of a general purpose interface for these geometries before asking about integration within lethe. If you don't mind bikeshedding, we could submit a pull request with the current state they're in - or include a link to them as we iterate their design. I was also prototyping some automatic Python API generation directly from the parameters C++ source files - but I'll start a conversation about it in a pull request once it's working.

What images would be most useful? The pellet designs are a collaborating company's secret (I know...), but I can share the workflow, usage statistics, perhaps a cutaway image after the packing - plus some difficult-to-reproduce incomplete feedback, such as slightly slow geometry loading; the memory usage becomes constant quite quickly, so perhaps some other step might need more parallelisation. Any suggestions for doing better detective work are more than welcome.

Anything is actually useful :) A few images of application (even without the scales) are always useful since it helps us gets more funding, which leads to more work into the code and so on and so forth.

Otherwise, whenever you encounter limitation, please don't hesitate to open issues. I don't think I have ever tries scaling the code to such large systems in the past and there might be some limitations or parts which are not slow at 500-1k cores abut that become at 2-3k. Luckly, the deal.II engine behind is really robust and that saves us most of the time :). Still we had issues in the past where we would, for example, copy the sparsity pattern everywhere before using it. This was not a huge memory hog, but above 100M DOFs it just killed all simulations. Luckly, this is fixed now :).

Anyway, I'm just curious to see how others are using Lethe. We make a lot of uses out of it, but I'm happy to see other people using it and if I can help you achieve what you want to achieve, it makes me even happier. Open-source is fun, but it is a lot, lot more fun to me when others are using it :)!

[DanW97]

By the way, I could not help but notice that you and your colleagues have developed very cool meshes to use with Lethe (e.g. https://github.com/DanW97/Lethe-tools) . If you are interested, you could open a PR in the lethe-utils repository to fork them there :). This latter repo is a sandbox where we keep tons of tools.

The repo is there : https://github.com/lethe-cfd/lethe-utils

Happy to open a PR once I rework the particle packing one, since this was written when lethe couldn't accept a particle file, so the generated .prm file will be significantly smaller now :)

We wrote a few geometry and case generation tools, but we weren't entirely happy with the code design - we were thinking of a general purpose interface for these geometries before asking about integration within lethe.

When you look at these geometry generation tools, there is a silly amount of boilerplate required for each one. With a general-purpose interface this would be more nicely integrated into lethe. However, in their current state they work "well enough" that it wouldn't hurt to include what we have for now, if this is of benefit to others :)