File i/o for dfn in 2D code and beyond

[mrhardman]

Currently we write all time-dependent data (fields, moments, dfn) to file every nwrite timesteps. This is OK for the 1D-1V code as the distribution functions are manageable in size (nvpa*nz). For the 2D-3V code, the neutral distribution function is significantly larger (nz*nr*nvz*nvr*nvzeta), as is the ion distribution function (nz*nr*nvpa*nvperp). We need to decide how to write distribution function data to file in the future.

It would be desirable to 1) write slices of the dfn as a function of time (e.g. f(z,v||) at fized r, or f(r,v||) at the wall) 2) keep the whole distribution for restart files 3) write out the distribution at fewer time steps than nwrite, but more than once. 4) have a separate .cdf file for distribution function data

This would require reworking the post processing diagnostics and file i/o routines, and so should not be undertaken lightly without a plan.

@mabarnes @johnomotani Do you have preferred solutions or features that we should keep or add going forward? I have in mind that we should make any changes now that make life easy for us to go 3D in the future.

[johnomotani]

Another thing we want to think about for I/O is what to do when we implement distributed-MPI. The simplest in terms of usability is to write one file per 'block', but this leads to a large number of files. The nicest solution is to have a bunch of different ranks all write to the same file. This is possible with NetCDF or HDF5 (through the C interfaces). The Julia interface to NetCDF does not seem to support the parallel version yet (see https://github.com/Alexander-Barth/NCDatasets.jl/issues/122). However, it looks like the HDF5 interface does https://juliaio.github.io/HDF5.jl/stable/, if you link it to a version of the HDF5 library that was compiled with MPI. On a quick skim, it sounds like this works because HDF5.jl exposes pretty directly the low-level HDF5 interface. Unfortunately this would mean re-writing all the file I/O code to use HDF5, but I think it would probably be worth doing (assuming we can make parallel HDF5 work!).

[mabarnes]

Implementing the I/o with NetCDF was not that arduous a task, so switching to HDF5 should not be too bad (though I've never worked directly with HDF5 myself, so cannot promise). I agree with the diagnostics you propose, Michael H: basically 2D slices of the pdf, in some cases averaged over the other variables and in others simply evaluated at special phase space points such as boundaries. Also writing out the low-order moments every nwrite seem sensible + the full pdf much less often. Most of the things we might want should be covered by that, and we can then add in any other special diagnostics as the need arises.

[mrhardman]

Closed by https://github.com/mabarnes/moment_kinetics/pull/90 https://github.com/mabarnes/moment_kinetics/pull/94 https://github.com/mabarnes/moment_kinetics/pull/96.