Definition of standard for output format of diagnostics

[bjoernbroetz]

In a recent conversation with @nielsdrost and @bouweandela it was discussed if we could come up with a standard for the output format of diagnostics. Once implemented this would allow to separate the visualization (i.e. generation of static plots, interactive web visualization, etc. ) from the computation in the diagnostics.

This conversation started facing the problem of using the output of the ESMValTool in ADAGUC or in other web based visualizations to create interactive maps and other plot types.

As a starting point it was proposed to use the netcdf format with the cf-convention. A defined set of metadata would be needed to keep provenance information as well as the tagging.

[bouweandela]

The code available in esmvaltool/diag_scripts/shared/plot probably covers most of the required functionality/plot types, though we will probably want to make use of the iris plotting capabilities to simplify using and maintaining the scripts.

[mattiarighi]

For NCL routines, the separation between diagnostic and graphic scripts is to some extent already implemented, although it would need to be improved and simplified.

In particular, the way NCL plot resources are passed from the diagnostic scripts to the graphic routines is over elaborate and has been created some confusion to the users in the past.

This is however a big task, since it would require rewriting (and testing) all plot scripts with a different approach. I'm not sure we can achieve this in v2.0. @axel-lauer

[bouweandela]

The way I see it, a good way to implement this in esmvaltool would be to add a single plot.py or plot.ncl diagnostic script, that receives the output from the diagnostic script(s), i.e. a list of directories containing netcdf files adhering to some standard, and some plotting options as specified in the namelist and then does all plotting. That way the same plot scripts can be used independent of the language that the diagnostic is in. The plot diagnostic script can then be added explicitly to the namelist (using the ancestors keyword to specify which diagnostic scripts it should plot the output of).

For a quick implementation based on the existing ncl code, plot.ncl could be a thin wrapper script around the existing ncl plot scripts, while a future file plot.py could be based on the plotting functionality available in iris.

[bjoernbroetz]

As @mattiarighi wrote:

This is however a big task, since it would require rewriting (and testing) all plot scripts with a different approach. I'm not sure we can achieve this in v2.0.

The implementation might be well to big for v2.0. I will remove the label. But here it is more about finding a strategy and defining a standard.

[nielsdrost]

For ADAGUC there is a work-in-progress supported file format document https://knmi.github.io/adaguc-server/, and file checker. This could be a good basis for this discussion.

[bouweandela]

@schlunma Here is some previous discussion about generic plotting tools. A bit dated though..