Initial mesh refinement TODO

[jbakosi]

TODO folloing up after #235:

• [x] Non-uniform refinement produces non-positive Jacobians even in serial. (This is probably an easy fix for @rfbird in the AMR object.) Most likely done by #247.
• [x] All this (#235) code will be abstracted away to another class, instead of it being in Partitioner.
• [x] The IC/error-based initial refinement currently still hard-codes the field obtained from the initialization for continuous a Galerkin PDE. This will be done in a discretization-independent fashion so it also works for a discontinuous Galerkin discretization. Done, see 23cd3c8d.
• [x] All this should be made to work for a union of all the scalar depvars given in the refrvar...end block. Done, see 27260ec6.
• [x] Setting Dirichlet boundary conditions using continuous Galerkin (DiagCG and MatCG) discretizations, does not yet work if the mesh is refined: newly added nodes will have to inherit the BCs from their parent's edge. Done but not yet tested by #247.
• [x] Setting BCs also does not work for DG because the side set faces on the physical boundaries must also be augmented/replaced with the new faces created during mesh refinement. Done but not yet tested by #247.

[jbakosi]

Bullet 2 above will be done potentially based on the following:

The problem is how to de-refine which needs the history if some of the children of a tetrahedron cell get assigned to different PEs, i.e., how to communicate histories?

How about we disallow repartitioning during (initial, multiple) refinement steps? Since the original mesh is always our coarsest we allow, that would keep all histories local to the original chare/PE, so the problem with communicating histories is avoided. This would create load imbalances during initial mesh refinement due to nonuniform refinement steps, but that would be okay, because here is how we could do it:

1. Read in mesh (as before)
2. Partition to chares (as before)
3. Global node reorder (as before)

At this point (without initial AMR so far) we used to create workers and would start advancing PDEs. We could inject the initial list of refinement/de-refinement here. Since at this point (after reordering) we have the mesh partitioned to chares, we create a new object that is a chare array (which is allowed to migrate) and do the series of initial mesh refinement steps there. We would do AMR in parallel in these chares (array) and would never repartition but which would allow migration. As a result, due to nonuniform steps, load imbalance would be created but that can be mitigated relying on Charm++ migration during refinement. Since we never repartition, each AMR object keeps its history and there is no need to communicate refinement state history, de-refinement is "simple".

When this is done, we create workers as before, and start PDE computation in chares as usual.

Bonus: The functionality in these new AMR chares could be exactly called the same way also for mesh refinement/de-refinement during time stepping.

[jbakosi]

Update after discussions with @junghans, @rfbird, and @adityakpandare:

The approach described in https://github.com/quinoacomputing/quinoa/issues/236#issuecomment-394910601 appears doable and favorable with the following important points:

1. The disadvantage of doing initial AMR after reordering is the appearance of hash-generated, and in general, non-contiguous global node IDs, which is definitely a problem for MatCG (not for DiagCG and DG). This could be solved by doing another global reordering step after AMR.

2. The above problem can be solved by doing AMR after partitioning to chares but before global node reorder, i.e., between steps 2 and 3 above, instead of after step 3 as https://github.com/quinoacomputing/quinoa/issues/236#issuecomment-394910601 describes.

3. In case of communicating AMR histories is still necessary with this approach for some reason, it appears that there is no need to communicate more than the history of the most recent step.

[jbakosi]

All done, see https://github.com/quinoacomputing/quinoa/pull/262.