Rework of adapt_tree and synch_ghosts

[Arcadia197]

I am continuing my work to waltz through the code and do preparations for CVS, while covering groundwork topics along the way. This time there are some hefty changes for adapt_tree and sync _ghosts / the underlying data transfer routines.

Here the major parts:

• reworked adapt_tree. It now starts with wavelet decomposition and reconstructs as the last step. This makes the wavelet decomposed values available for all the routines in between. Especially:
  • Coarse extension boils down simply to the modification of the SC or WC
  • AddSecurityZone and CoarseninIndicator have access to wavelet decomposed values and could be adapted
  • The actual coarsening step becomes a send operation from the SC of the wavelet decomposed blocks to the domain of the mother block
  • Double coarse extension becomes really cheap, in fact we do it three times just to avoid one sync
  • All syncs can be more streamlined, currently we only need three syncs but this could maybe be even reduced to two (one for normal values and one for wavelet decomposed, updated values can be synced with next iteration or once at the end?)
• Reworked sync ghosts. All details for this can be found in PR #57 . It there was not directly a speedup but already helped me to quickly design the coarsening with patch updating.
• Introduced family relations, those are already used for non-cvs for the coarsening step.

The work is not compleeeetely finished but there is much yet to read for you, thomas. Here some leftovers:

• adaptive fails for two tests still and the error is in the CE of the most coarse CE-boundaries. Will need to check within the SC and WC for old and new implementation what is going on
• Speedup to be comfirmed, This time I am sure though that there is something to gain!
• Check JMax speedup if it was implemented sufficiently or we can gain more
• cleanup work: commenting where we have WD-values, removing all testing variables
• Security zone is next on the list, maybe you can already use this code to start? Should be not too much work I guess

Broader cleaning I came up with along the way, maybe I move this to issues:

• cleanup wavelet module: remove inflated mallat
• cleanup wavelet module: correctly identify normal mallat in the routines and assign it where we have it, use our conversion routines
• cleanup wavelet module: cleanup conversion modules and combine this even-odd case as I have done it for some parts
• cleanup sync ghosts: rewrite whole-block transfer to work with xfer_block_data, just need to write one function to create the lists and the rest is already there. Maybe there I can assess if mpi likes it more in buffered fashion or one mpi call per block
• cleanup all the loose temporary buffers - there are so many and I am pretty sure currently we need more than twice as many memory as we might need. I want to collect all temporary buffers and create an overview so we can discuss what we need and what not

At last my presentation from today to have the algorithm in front of you 24-05-23 short update.odp

[Arcadia197]

I checked and the discrepancy in the wavelet/adaptive tests actually arises from a small bug in the old version. Somehow CE for level 3 was not executed, actually the whole adapt tree loop. This arises from the fact that 3 is Jmin_active and no blocks were coarsened in step 4 (as for indicator everywhere and random this was done in the first iteration), so the condition at the end of the loop exits early.

The new version sets all flags for the indicator in the first iteration but keeps them and still only coarsens level wise so values are correct. We need to update the test results though ... Possibly after changing the security zone settings anyways.

[tommy-engels]

I checked and the discrepancy in the wavelet/adaptive tests actually arises from a small bug in the old version. Somehow CE for level 3 was not executed, actually the whole adapt tree loop. This arises from the fact that 3 is Jmin_active and no blocks were coarsened in step 4 (as for indicator everywhere and random this was done in the first iteration), so the condition at the end of the loop exits early.

The new version sets all flags for the indicator in the first iteration but keeps them and still only coarsens level wise so values are correct. We need to update the test results though ... Possibly after changing the security zone settings anyways.

Okay, so you found the "error", which is actually a bugfix in the new version! Congratulations!! I will go ahead and test this new version on the cluster now

[Arcadia197]

Another point: We might prevent the overcreation of blocks when merging (creating the mother block before the daughters are deleted) by setting one of them to the mother block and just shifting the SC inside itself. It's not too much to code and would circumvent the Block-increase for the non-CVS cases.

[Arcadia197]

Did some preparations to adapt the security zone, to be discussed together as there is one puzzle piece missing to do it

[Arcadia197]

Added security zone changes after our discussions today. I went for the encoding approach and we actually do not sync more as before.

Obviously now all adaptive lifted convection test as well as the acm test fail.

Here some visualization of the difference in the results:

1. Absolute difference between pressure fields from acm test at last timestep (t=1) in logarithmic form. The red squares are where the grid differs. The rest ranges from $2\times10^{-5}$ downwards. This difference is to show that in the tests no substantial deterioration is directly visible. How can we further test that the implementation doesn't eradicate too much?
2. Pressure field as reference

[tommy-engels]

I merged the PR into the main code, let us discuss the changes to the unit tests on monday. In addition to your tests, I did the compression test for all wavelets with the new securityZone, and the compression behavior is correct. This is a good indicator that the new security zone works as intended.

[tommy-engels]

new.compression-test-fortran.pdf old.compression-test-fortran.pdf

here are the tests for the new (your) vs old version