Modification of XML files from within esm_tools yaml files

[JanStreffing]

Is your feature request related to a problem? Please describe. We are starting to use AWI-CM3 with the XIOS IO server. The output is no longer configured in f90 namelist files, but in xml files. It would be nice to be able to modify the xml files from within esm_tools, so as to not have to make new xml template files for every small change.

Describe the solution you'd like Something that looks like for fortran namelist changes would be nice:

        add_namelist_changes:
                fort.4:
                        NAMCLDP:
                                SCLCT_SWITCH: 1

Do we already have this for FOCI-OpenIFS? Is this on your wishlist @joakimkjellsson

@denizural

[joakimkjellsson]

Hi @JanStreffing

This is definitely on my wishlist for Christmas. I looked into this, and it's pretty easy to translate yaml to xml. That means we could define our XIOS output using yaml files and then let ESM-Tools do the modification. A typical file_def.yaml could be:

file_definition:
   _type: "one_file"
   _format: "netcdf4"
   file:
      _id: "file1"
      field:
         _field_ref: "z_pl"

which would make the following file_def.xml

<?xml version="1.0" ?>
<file_definition type="one_file" format="netcdf4">
    <file id="file1">
        <field field_ref="z_pl"/>
    </file>
</file_definition>

which would feed into XIOS.

Furthermore, XIOS supports grouping of fields, i.e we could define something like "AMIP_sfc" in fields_def.xml

<field_group id="AMIP_sfc">
      <field id="2t"      long_name="2m air temperature"   unit="degK"  />
      <field id="tp"      long_name="total precipitation"   unit="m"    />
      <field id="u10"    long_name="10m wind"   unit="m/s"    />
     etc....
</field_group>

Then we would have the following in file_def.xml

<file_definition type="one_file" format="netcdf4">
    <file id="file1">
        <field field_ref="AMIP_sfc"/>
    </file>
</file_definition>

That way we could activate standard output like "Aday" or "Amon" by adding just a single line in file_def.xml.

So ESM-Tools need either: a) a package that can modify XML files directly or b) to modify some template file_def.yaml that we have prepared ourselves.

I've got a very simple python code that translates the above yaml -> xml. It builds on the "xml" and "yaml" packages.
But I haven't sunk deep enough into ESM-Tools that I know where and how we could create / modify a yaml file.

I would really like your thoughts on this.

Cheers Joakim

[pgierz]

XLM is in the Python standard library: https://docs.python.org/3/library/xml.dom.html

But somehow from Joakim's comment I do not think that is what you need, at least not without some logic around it. The nice thing about the namelist package was that it just directly converted everything to a dictionary, which made further use super easy. That was purely heading, key, value. XML looks like it has some more stuff:

<file_definition type="one_file" format="netcdf4">
    <file id="file1">
        <field field_ref="AMIP_sfc"/>
    </file>
</file_definition>

So, heading, heading ID A, heading ID B, inner heading, key, value? I can't really tell without looking at the most complex example (but just typing that also scares me of what the most complex example could look like)

I have never worked with XIOS, and while I would love to make @joakimkjellsson christmas more merry, this likely isn't going to be in the cards until the new year. I do however have a day long train journey to get through before Christmas, so maybe we are lucky. @JanStreffing, could you perhaps post some examples of "plain" templates, "corrected" templates, and the sort of yaml you would write for going from A to B? Even just hypothetically. @joakimkjellsson, just post your code as a comment, I can use that as a jumping off point.

Perhaps someone could look at the following answers and let me know if something like that could be useful for your case? https://stackoverflow.com/questions/2148119/how-to-convert-an-xml-string-to-a-dictionary

[JanStreffing]

This came up in discussion again today. The feature is still on the TODO list, and one of the applications would be to change the splitting frequency of XIOS output files: file_def.xml: split_freq="1y" based on the restart frequency of the esm_tools:

    nday: 0
    nmonth: 1
    nyear: 0

[JanStreffing]

bump, as @pgierz is working on this

[mandresm]

Sorted! Find the documentation here: https://esm-tools.readthedocs.io/en/v6.41.0/jinja.html

[joakimkjellsson]

Wow @mandresm! Christmas sure did come early! I'll try this out at some point.

Best wishes /Joakim

[seb-wahl]

Indeed, this is really great news. Congratulations.

[JanStreffing]

Will try this out tomorrow morning :)