Major: reworked synchronize ghost, performance gain visible and more clarity

[Arcadia197]

In preparation to give synchronize ghosts more flexible ways of synching I deep dived into the current implementation.

I noticed the following:

• there were two individual loops present to apply the logic of who synchs with whom
• two actual synching loops were made, one for metadata and one for the actual data

I reworked the structure and now we do the following:

1. Loop over all neighbor relations and apply synching logic, save all metadata in one array (meta_send_all). Keep track of the sizes for metadata and data as before.
2. Loop over meta_send_all and fill the sendBuffer with the necessary data. Here we save for each processor:
   • first entry is the amount of patches we synchronize with this processor
   • afterwards follows all metadata, we need send_hvy_id, neighborhood, level_diff and data_size, possibly rank for development
   • after that we have all actual data
3. Do the transfer logic as before but only transfer the one buffer from which we then extract the metadata and data
4. Internally loop over meta_send_all and copy if this neighborhood relation is internal

This enables us to:

• Apply synching logic at one place to enhance the functionality
• Save time because we have less mpi communication of close-to-none data

Performance gain is visible, easy tests show 15% with blob_adaptive_2D_CDF62 which I normally use. I know 62 is probably not representative but probably this scales anyways better in 3D?

[Arcadia197]

Additionally, I thought about the no-stage synching along the way with correcting internally. Are you sure this can be made working? If I look at it for example with CDF42 there should be points missing without enlarging the boundary. In the attached pdf the left orange point cannot be reconstructed on the receiver side without more points. Obviously we could send different points from the coarse side but this would be really tedious at the edges with reconstruction with level difference. Am I missing something? To be discussed together ..

synch_ghost_stagefree_CDF42.pdf

[Arcadia197]

Also just dumping my ideas that I had to implement new ways of synching:

• (Normal synch between all neighbours with 2 stages)
• (Same level only (for CVS SC and WC))
• Send to lower level, send to upper level (one-sided updates for CVS)
• Finer to same or same to coarser level (CE)
• SC to mother (CVS, redundant grid) - different patches are needed
• update SC of daughters (CVS, redundant grid) - different patches are needed
• Do not update edges or corners (for RHS after refinement)
• send to lower and receive from upper (for CE) with interpolation from full ghost layer to avoid second CE, this needs a new flag to indicate which cells have the correct boundary values to perform this

[Arcadia197]

As further tests showed no direct benefit I will close the PR for now as I would like to push further work into my branch and it is to be merged later.