Saving options

[johnomotani]

Options that were actually used in a run are saved into BOUT.settings.

• BOUT.settings is only fully created if the run finishes cleanly

  • the initial version is created before the init() method of a PhysicsModel is called, so is missing many options
  • I often let runs time out, and then the final version is never created
  • I think it would be more reliable (and useful at least for me) if writeSettingsFile() was called when the output monitor is called for the first time (which is after rhs() has been evaluated at least once, so we can be pretty sure that all the needed options have been read and used).

• It would be nice to write the options into the dump files as well

  • BOUT.settings is nice because it can be used as an input file to explicitly re-create an old run, even if defaults, etc. have changed since then, so I think is definitely a feature to keep
  • but for keeping track of run output, it's much nicer to have everything in a single file where it can't be separated. (I say 'single file' assuming the BOUT.dmp.*.nc files will be squashed to a single netcdf file somehow or other to avoid wasting disk space and network bandwidth.)

• What is the best way of saving options into the dump files?

  • saving each option into a variable is simple, but might risk name conflicts with existing variables?
  • it would be natural to create a 'group' in the netCDF file for options, and maybe sub-groups for sub-sections. Unfortunately xarray doesn't support groups at the moment

A backport would be nice to have, but I guess how feasible it is depends on implementation details...

[johnomotani]

For the 'every option as a separate variable' scheme, we would need to make variable names with nested section names. netCDF allows underscore '_', period '.', plus '+', hyphen '-', or at sign '@' (https://www.unidata.ucar.edu/software/netcdf/docs/netcdf_data_set_components.html#object_name). Hopefully at least one of those is not allowed in option names? So we could use say - as a separator?

If we prefixed every variable with "Options-" we'd know which variables were options (so we could add some special handling in xBOUT) and could be pretty sure of avoiding name conflicts.

[ZedThree]

Yes, we should definitely try and save the complete settings as soon as possible.

OptionsNetCDF does some of this already: https://bout-dev.readthedocs.io/en/latest/user_docs/bout_options.html#reading-and-writing-to-netcdf

That uses groups, which I think is the more natural way of saving the hierarchical structure. If I understand the xarray docs correctly, xarray can't load a group structure directly, but you can pass it individual groups to load. xBOUT might then need to iterate over the groups with another library and load them individually perhaps?

I'd really like to be able to deprecate the current DataFile stuff and just use OptionsNetCDF instead, as it's much nicer to use.