Automated post-processing

ACCESS-NRI / access-esm1.5-configs

Standard ACCESS-ESM1.5 configurations released and supported by ACCESS-NRI

Creative Commons Attribution 4.0 International

0 stars 0 forks source link

Automated post-processing #3

Open aidanheerdegen opened 2 months ago

aidanheerdegen commented 2 months ago

The UM outputs are in a proprietary format (PP files) that should be automatically post-processed to netCDF.

This has traditionally been done with python scripts like um2netcdf.py.

This functionality (amongst others) has been subsumed into a new package:

https://github.com/ACCESS-NRI/amami

which will provide a um2nc tool.

This will require having this package available in the payu conda-environment and adding a payu userscript hook to model configurations to call the um2nc function at the appropriate time.

truth-quark commented 2 months ago

Do we have any indication if um2nc will be run from the command line/scripts and/or called as a library from workflow code?

aidanheerdegen commented 2 months ago

Do we have any indication if um2nc will be run from the command line/scripts and/or called as a library from workflow code?

Likely called as a command line script at least in the first instance.

aidanheerdegen commented 1 month ago

What are the requirements for post-processing? Review of command line options. What options can we exclude?

Is everything that is currently done necessary:

Coordinate rounding
Removing unreliable cell methods
Metadata fixes (coordinate naming,
Masking pressure level field
Fixing pressure coordinates
Converting to 32 bit
Calendar fixes

Any additional metadata required to meet FAIR data standards?

Pinging @MartinDix for feedback.

atteggiani commented 1 month ago

What should be defined here are all the requirements for the ACCESS-ESM1.5 post-processing conversion to netCDF.

Most importantly, I need to understand if there are any requirements that currently cannot be performed in amami um2nc, so I could work on the implementation of them.

Some examples could be:

Do we expect any processed um fieldsfile to contain time-series fields?
Do we expect the STASH variables lookup for the naming of the converted netCDF variables to use the same STASHmaster file or we need to allow the choice of a different STASHmaster files to be used for it?

truth-quark commented 1 month ago

It would also be useful to review other details such as any redundant um2nc edge cases. Ideally, anything that should be removed would be done "now", before retrofitting amami unit tests.

aidanheerdegen commented 4 weeks ago

@MartinDix can you select which of these are required for the ESM1.5 post-processing?

[x] Coordinate rounding
[x] Removing unreliable cell methods
[x] Metadata fixes (coordinate naming)
[x] Masking pressure level field
[x] Fixing pressure coordinates
[x] Converting to 32 bit
[x] Calendar fixes (proleptic Gregorian conversion)
[ ] Support time-series
[x] Support choice of STASHmaster file

truth-quark commented 2 weeks ago

@MartinDix can you select which of these are required for the ESM1.5 post-processing?

[x] Coordinate rounding

[x] Removing unreliable cell methods

[x] Metadata fixes (coordinate naming)

[x] Masking pressure level field

[x] Fixing pressure coordinates

[x] Converting to 32 bit

[x] Calendar fixes (proleptic Gregorian conversion)

[ ] Support time-series

[x] Support choice of STASHmaster file

These are raised as issues/requirements here: https://github.com/ACCESS-NRI/um2nc-standalone/issues