Reverse order of MPI ranks

[AlexanderSinn]

Since the MPI communication is now all done in HiPACE++, we can reverse the order of the ranks to better reflect the timestep. Additionally, this PR prints the time of the start of a time step relative to the start of the simulation.

00:00:00 Rank 0 started step 0 at time = 0 with dt = 1.676676018e-13
00:00:00 Rank 1 started step 1 at time = 1.676676018e-13 with dt = 1.676676018e-13
00:00:00 Rank 2 started step 2 at time = 3.353352035e-13 with dt = 1.676676018e-13
00:00:00 Rank 3 started step 3 at time = 5.030028053e-13 with dt = 1.676676018e-13
00:00:19 Rank 0 started step 4 at time = 6.70670407e-13 with dt = 1.676676018e-13
00:00:23 Rank 1 started step 5 at time = 8.383380088e-13 with dt = 1.676676018e-13
00:00:29 Rank 2 started step 6 at time = 1.006005611e-12 with dt = 1.676676018e-13
00:00:34 Rank 3 started step 7 at time = 1.173673212e-12 with dt = 1.676676018e-13
00:00:39 Rank 0 started step 8 at time = 1.341340814e-12 with dt = 1.676676018e-13
00:00:44 Rank 1 started step 9 at time = 1.509008416e-12 with dt = 1.676676018e-13
00:00:49 Rank 2 started step 10 at time = 1.676676018e-12 with dt = 1.676676018e-13
00:00:55 Rank 3 started step 11 at time = 1.844343619e-12 with dt = 1.676676018e-13
00:01:00 Rank 0 started step 12 at time = 2.012011221e-12 with dt = 1.676676018e-13
00:01:05 Rank 1 started step 13 at time = 2.179678823e-12 with dt = 1.676676018e-13
00:01:10 Rank 2 started step 14 at time = 2.347346425e-12 with dt = 1.676676018e-13
00:01:15 Rank 3 started step 15 at time = 2.515014026e-12 with dt = 1.676676018e-13

Note that this changes the results of functions that use the RNG because the initial random seed depends on the rank ID.

• [ ] Small enough (< few 100s of lines), otherwise it should probably be split into smaller PRs
• [ ] Tested (describe the tests in the PR description)
• [ ] Runs on GPU (basic: the code compiles and run well with the new module)
• [ ] Contains an automated test (checksum and/or comparison with theory)
• [ ] Documented: all elements (classes and their members, functions, namespaces, etc.) are documented
• [ ] Constified (All that can be const is const)
• [ ] Code is clean (no unwanted comments, )
• [ ] Style and code conventions are respected at the bottom of https://github.com/Hi-PACE/hipace
• [ ] Proper label and GitHub project, if applicable

[MaxThevenet]

Thanks for this PR! Could we set the RNG seed so the results do not change? So when users re-run a previous simulation, they do not get different results.

[AlexanderSinn]

The default random seed is set in https://github.com/AMReX-Codes/amrex/blob/3fe7aadc23c94a038b306af54ff732ce5564c3d1/Src/Base/AMReX.cpp#L625 It depends on the ID of the rank. This means that with the old rank order the random seed of the head rank that initializes the beams depended on the number of ranks. In the spin tracking simulations I made, I added a bit of vacuum before the plasma profile and also changed the number of ranks. The result was different and at the time I thought it was because of the vacuum but it probably was caused by the change to the random seed. With this PR the head rank always has ID 0 so it will fix this issue for the first time step. The random seed of later time steps will still depend on the number of ranks but that doesn't affect the beams, only plasma, collisions and ionization. The seed issue for the first time step cannot be fixed while being RNG backwards compatible. Note that users that use random_seed = … are unaffected.

[MaxThevenet]

Good point. Could you add an option for the users to easily recover the previous behavior? e.g. use_previous_rng (false by default)? Then we'll announce this breaking change on the channel, and hopefully this will be modest pain for them