Bottlenecks in creating and loading very large .msh files

[gthyagi]

Bottlenecks with .msh files

Two serial operations involved in creating and loading .msh file. Due to these steps, I recommend not to use .msh files for very large 3D models.

First Bottleneck: https://github.com/underworldcode/underworld3/blob/c953b6fde0ea8526cba87b4d2a720032c9ce5bcb/src/underworld3/meshing.py#L548

Second Bottleneck: https://github.com/underworldcode/underworld3/blob/c953b6fde0ea8526cba87b4d2a720032c9ce5bcb/src/underworld3/discretisation.py#L78

This bug is reported with Underworld:development branch and Petsc:3.21.0

[gthyagi]

@lmoresi what are the pitfalls of doing the second step (above mentioned) in parallel?

Gmsh can not directly write to a h5 file. So, we have to create .msh file and then convert it into .h5 file. I think, using petsc to do this conversion is a better option instead of using other tools like meshio, as it helps mitigate potential compatibility issues.

[lmoresi]

My suggestion is to checkpoint the mesh using underworld (in serial). You'll still go through the .msh creation but you don't need to run this on Gadi which is particularly slow (bottleneck (1) above). This will store the DMPLEX topology information which is responsible for bottleneck (2) above. The h5 file we checkpoint is specific to PETSc so there are not really other tools to do this directly.

The preferred solution is to refine the mesh from some coarse approximation, but we'll need to have a refined case for debugging the issues you are finding with that.

[gthyagi]

Your suggestion to refine the mesh without refinement_callback=None worked in serial. Issue lies in spherical_mesh_refinement_callback function.

[gthyagi]

loading .h5 file takes significantly less time then .msh file