Different MPI ranks restarts

[mtwest2718]

One of the reasons for moving to HDF5 was that a consolidated single file would allow restarts at different rank numbers.

• [x] Changes in IO_rot.f90 to get it to ignore info in config.txt file and just use the new settings.
• [x] Check that the output is close to what one would get if you ran it all the way through from the start.

It didn't work out of the box, so I need to go hunting for the what's giving me errors.

[mtwest2718]

I found the few lines to change in IO_rot.f90 [471, 475-7] to get the simulation to run with a different number of MPI ranks. So then I started to check consistency.

1. repeated runs (initial & restart) with the exact same settings
   • Yes this gives consistent results
2. same number of ranks, different restart positions
   • Rarely
3. different ranks, same starting position
   • Haven't found a pairing yet that gives the same results at all time steps

Have been checks of 1 & 2 in previous versions of the software? @AstroSnow

[mtwest2718]

So talking to @AstroSnow today about restarts.

1. The primary use of this is to continue the simulation after its initial stopping point
2. One can restart from a fixed point in the initial sim, though its uncomon
   1. Because the test sims have low resolution, I am getting diverges
   2. These discrepancies also have something to do with the ratio of the save time to step time dt

With the sim restart code working from the last time-step, I would say this is good to go.

One can continue a simulation with a different number of MPI ranks from what it was started.