Currently, the default domain decomposition leads to a majority of the time being spend on communication for a higher number of ranks. Hence, we ought to implement a smarter one. Polar coordinates are probably a better starting point for the decomposition but the may require either a lot of conversion with Cartesian (i.e. a lot of trigonometric function calls) or fundamental changes in the code (like having everything in polar).
An easier alternative might be to start with splitting the particles by altitude above Earth? Thus, each rank would be still be a shell around Earth but we avoid all the conversion? As I see it
Pros
Easy to ignore domain inside Earth to avoid empty ranks there
Altitude computation should be much cheaper than trigonometric functions
Easy intuition to check for collisions between ranks during transit (more below)
Quick to implement
Cons
At some point scaling will break down again as ranks get very small altitude ranges leading to a lot of migration again. At that point some rudimentary form of polar coordinates could be used, e.g., splitting the domain additionally, e.g. in particles with either x,y,z > or < 0, i.e. basically cutting each shell in half / four. This will lead to migrations every few hundred iterations but not as many as in Cartesian boxes as these are still spherical.
What needs to be done
[x] Implement new domain decomposition for altitude
[ ] Identify smart place to compute altitude for all particles in rank (we are probably using it somewhere else already, I think?)
[x] Come up with a nice scheme to handle migrating particle collisions (I would suggest to look at altitude change in last time step to identify a tighter bound of considering collisions. E.g. a particle at 150km and one at 152km will not collide if their altitude change in the last time step was very small. This should be a pretty legitimate smoothness assumption we can make, I think and most particles have small altitude changes due to low eccentricity.
Feature
Desired behavior
Currently, the default domain decomposition leads to a majority of the time being spend on communication for a higher number of ranks. Hence, we ought to implement a smarter one. Polar coordinates are probably a better starting point for the decomposition but the may require either a lot of conversion with Cartesian (i.e. a lot of trigonometric function calls) or fundamental changes in the code (like having everything in polar).
An easier alternative might be to start with splitting the particles by altitude above Earth? Thus, each rank would be still be a shell around Earth but we avoid all the conversion? As I see it
Pros
Cons
What needs to be done
How it can be tested
TBD
What do you think, @FG-TUM ?