Putting diffusivities in "reference" and creating readable file "input" for parameters in main_input

[illorenzo7]

Correct me if I'm wrong, but I don't believe there are currently diffusivities in the "reference" file that is created by Rayleigh. I am working on including them, so that the object ref (ref = ReferenceState('reference') will have attributes

ref.nu ref.kappa ref.eta

when applicable. Also, I think it would be nice to have

ref.dpdr

...the radial derivative of the reference pressure. This way, we can immediately check hydrostatic balance when incorporating custom reference states. If there is interest for this, I will create a pull request.

Additionally, I have written a script that manually parses the text in "main_input" (in Python) and associates it with the numerical values of the parameter. I think it would be helpful for Rayleigh to generate a binary file

input

And a diagnostics function

InputParams

that can immediately read input parameters to Python as in

input = InputParams('input')

with attributes like

input.n_theta, input.angular_velocity, input.nu_top, etc.

Again if there is interest in this I will make a pull request.

Also please let me know if such functionality already exists and if so, how to use it! Otherwise, I will modify away.

[gassmoeller]

Are you suggesting that input would be a file that contains only the values of the input parameters in a binary format? This will be somewhat problematic in case we add or remove new input parameters in a future version (new versions of the script could not read old files and vice-versa). i like the idea of storing a copy of the input parameters of a model together with the output for documentation and reproducibility purposes, but I would think that a documented text version (at least containing the name and value for each parameter) is more useful, maybe just a copy of main_input. Since you seem to have a script to read this already, I currently do not see additional value in a binary file. Or do I misunderstand your idea?

But in general yes, adding these features (the script and archived input file) would be appreciated :+1:

[illorenzo7]

My idea was that the binary file would contain both the parameter names and and values of all parameters in the namelists of main_input. Then, in keeping with the format for rayleigh_diagnostics.py, there would be a diagnostics function to read in an "input" object to Python, which would have attributes identical to the individual parameter names in main_input. Writing to a binary file is probably unnecessary; I just wanted a diagnostics object (like Shell_Slices or ReferenceState) that would contain the input parameters. My current script can read the value of a parameter in main_input one at a time, but you need to know the parameter name a priori. What's nice about ReferenceState and Shell_Slices is that you can read in the data before knowing exactly what the individual pieces are (revealed later by the 'names' parameter of ReferenceState and the lookup table of Shell_Slices).

After doing some research, however, I am realizing that modifying my script is probably unnecessary. Someone has already written the module f90nml (which can be installed using pip) exactly designed to import Fortran namelists from a textfile into Python dictionaries. So, my new idea is:

1. When running Rayleigh, there is an option to create a copy of "main_input." (Perhaps with the name "main_input" and some prepending text to indicate the time/date it was run).

2. I make a "wrapper" function that calls the f90nml.read function and makes some tweaks to the resulting dictionary in order to parse the main_input file in a way identical to the other diagnostics. I'd like to have this function be called "InputParams" and be placed in Rayleigh/post_processing/rayleigh_diagnostics.py. This will require users who want to use the functionality to install f90nml via pip or equivalent. Alternatively, I could place the f90nml.read function (which doesn't appear to be that long) directly inside rayleigh_diagnostics so that everything is self-contained. I believe this is allowed as long we mention where the source code came from and include the url to the Apache license (which f90nml falls under) in rayleigh_diagnostics. If including such a license is undesirable, I can modify my original script to perform the same functionality, though it seems easier to just use the already existing f90nml module.

3. This is more of a question: when the "copy main_input" option is specified (if having such an option is agreeable), can we also generate a minimalistic logfile associated with the run? Pleiades already does this to some extent by generating logfiles for the particular job, which usually contain things like "resources used (number of cpus)" "total wall-time," the exact time and date the job was run, as well as failure reports from individual nodes if something goes wrong. My idea is that this logfile and the "main_input" copy can together describe when the simulation was run, the resources it used (and whether it was successful or not), and exactly what inputs were used. I think this would be useful for debugging purposes when jobs fail. Finally, to complete the history of the job run, perhaps when the "copy main_input" option is specified, we could automatically redirect "stdout" to a second logfile, so that the particular simulation iterations are also stored. Ultimately, I am envisioning three files organized by the start time of the simulation, which would together describe exactly how a given simulation was run. For example:

2018-10-21_23:05_main_input 2018-10-21_23:05_run_specs 2018-10-21_23:05_stdout

(The format for the date/time ensures the files would be stored in chronologically increasing order within the same directory).

Sorry for the long wind! Anyone's feedback on these ideas would be much appreciated.

[illorenzo7]

On a different note, I have modified "Transport_Coefficients.F90" to write the radial profiles of the diffusivities to a binary file called "transport" as well as a corresponding class "TransportCoeffs" in rayleigh_diagnostics so that the binary file can be read into python. Still testing to make sure it works.

[feathern]

A few comments: (1) I think having the transport_coeffs file is good idea. You should definitely initiate a pull request with that functionality, but you may want to wait until Ryan's recent pull request has been accepted. That leads me to comment #2 (2) Ryan has essentially implemented making a copy of main_input

(3) I would prefer that we don't alter the reference state structure to include transport coefficients just yet. That's a pretty widely-used data structure, as are the transport coefficients, and so altering either potentially constitutes a fundamental change to the code's internal workings. I think we should give some thought to what makes the most sense here, bearing in mind that there are other transport-related arrays that we may want to include. It might be that the most sensible thing to do is to create an equation coefficient data structure that holds all constant and non-constant coefficients used in the system of PDEs solved by Rayleigh.

[orvedahl]

This has been mostly resolved. The transport file is currently written out and my pull request that basically copies the input file was merged. The reference state does not currently include the transport coefficients, but the entire reference structure will be changing to only hold constants and non-constant coefficients...stay tuned.