pbauman
commented
5 years ago Capturing comments by @roystgnr below from an email thread (note he ran in parallel so some of the issues he points out apply to parallel case):
Things that stand out (note that these are often overlapping), ordered from easy to hard to improve:
1.5% in repartitioning-related stuff is just me being an idiot; that needs to short-circuit in serial. Boris brought up a couple places with similar issues recently.
5.5% in distribute_dofs(), and 5.5% for the combination of dof_indices() and old_dof_indices(), and much of that can probably be shaved off soon with the optimizations I was discussing with Derek, and doing so will disproportionately benefit AMR.
13% within JumpErrorEstimator::estimate_error() that isn't in a subroutine profile is astonishingly high, and I've never tried optimizing that, so there may be easy opportunities I haven't seen.
10% in project_vector(old,new) and 10% in prepare_for_use(); those are still high priority optimization targets but I don't know of any low hanging fruit.
8% in Mesh::find_neighbors()... I think I can actually get rid of most of that, but only with a major redesign. Likewise with the 2% in TopologyMap::init().
13% in inverse_map(), 15% in compute_face_map() - the "map on one element inverse map on the other" process for matching up jump evaluations and projection integration might be replaceable by a more direct calculation, but not easily.
roystgnr
I've attached an example program (main.cpp.txt sigh) and input file (data.txt). This is a modified version originally sent by Simone Rossi and @boyceg. Even running in serial (using superlu for the linear solve via PETSc), AMR is much slower than the uniform grid case, where we limit the AMR grid h level to the same as the uniform. By a factor of about 3x. @roystgnr did profiling in parallel. I'm putting my serial performance log below where the runtime was 60s, but running with a uniform grid no AMR takes about 20s (on my Ivy Bridge workstation, opt mode in libMesh, etc.).
The key difference AFAICT between this example and the transient AMR example we have for convection diffusion is that this layer is "thicker" so we get more elements in the layer so even though the n_dofs is 4-8x lower than the uniform case, the AMR overhead is larger. And as we can see in my perf log below, the vast majority is in the error indicator.
data.txt main.cpp.txt