Reduce memory allocations in AMR

[ranocha]

Currently, we create completely new solution arrays when refining and coarsening (the solver).

• Can we instead just insert/erase some solution data to save memory allocations and some time?
• Is it worth the effort?

[sloede]

Here are my initial thoughts on this issue:

• Memory allocations are in itself no problem (yet). Thus I'd address them only when it's clear that we face some sort of memory bottlenecks, either due to allocation time or out-of-memory errors.
• Inserting and erasing solution data (= the element containers) could theoretically be easily supported by implementing raw_copy!, invalidate! etc. for the DG containers (https://github.com/trixi-framework/Trixi.jl/blob/master/src/auxiliary/containers.jl#L6-L10). However, the surface data structures (interfaces, mortars, and so on) are just too much of a hassle to keep in sync - I think it's better to throw them away and rebuild them each time.
• Whether it makes sense from a wall clock time perspective to reuse the element containers or not is something that would require careful benchmarking to decide. So far, it always seemed to me that the projection/interpolation of solution data is by far the most expensive part of the AMR procedure with respect to the solver. But this might change with your optimizations in place.

[ranocha]

You're definitely right - we should do some careful benchmarks before trying to optimize stuff that isn't worth it.I just thought that allocation 679MiB on average per call looks like it might be worth the effort to reduce the allocations.

[sloede]

allocation 679MiB on average per call looks like it might be worth

No, you're right. This should at least be motivation enough to be investigated once. The downside is that it would require the elements container to be larger than strictly necessary to account for future growth. Thus, e.g., nelements(elements::ElementContainer2D) = length(elements.cell_ids) will not be possible anymore. This might be OK, but we'd have to make sure that anywhere any of the element variables are accessed, no slicing with :, or size(dg.elements.u, <index_for_element_ids>) is used.

[ranocha]

Moreover, we allocate some temporary arrays - we could also move them to some cache.