Changes in doc concerning the use of CMIP6-like custom tables

MartinCussacMF commented 6 months ago

Description

Modifications to the 'configure' documentation, to better reflect the possibility to use both CMIP5-like and CMIP6-format for custom CMOR tables. Added an example of a .json CMIP-like custom table.

Closes #2364

Before you get started

[x] ☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

[x] 🧪 The new functionality is relevant and scientifically sound
[x] 🛠 This pull request has a descriptive title and labels
[x] 🛠 Code is written according to the code quality guidelines
[x] 🧪 and 🛠 Documentation is available
[ ] 🛠 Unit tests have been added
[ ] 🛠 Changes are backward compatible
[ ] 🛠 Any changed dependencies have been added or removed correctly
[ ] 🛠 The list of authors is up to date
[ ] 🛠 All checks below this pull request were successful

To help with the number pull requests:

🙏 We kindly ask you to review two other open pull requests in this repository

codecov[bot] commented 6 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 94.07%. Comparing base (4615da7) to head (98a36fa).

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #2368 +/- ## ======================================= Coverage 94.07% 94.07% ======================================= Files 241 241 Lines 13420 13420 ======================================= Hits 12625 12625 Misses 795 795 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

MartinCussacMF commented 5 months ago

Hi @valeriupredoi ! Good catch for the mistake, I accepted the sugested changes.

valeriupredoi commented 5 months ago

many thanks @MartinCussacMF :beer: One thing I'd be very grateful if you did, please would be fixing the PR description boxes - check the ones that are relevant to this PR (docs and tests, and all tests pass, can remove all the others). Let me know if you're unsure what to keep, will do it then :+1:

bouweandela commented 5 months ago

@schlunma or @bouweandela do we have implementation for JSON-loading of the CMOR custom variable tables

I'm not very up to date with this part of the code. It looks like the features were introduced by @schlunma in #1625 and #2279. @schlunma Would you have time to review this?

schlunma commented 5 months ago

In these PRs I only made the location of these tables customizable. This did NOT change the code that actually reads the tables.

To my knowledge the custom tables need to be in CMIP5 format (the custom table class inherits from the CMIP5 class). Also, it looks like only files in the form *.dat are considered, so I am really surprised that this works with *.json files in CMIP6 format.

@MartinCussacMF are you 100% sure that works with *.json files? Please keep in mind that ESMValCore might also look in other tables than the one specified by the mip attribute if a project has cmor_strict: false in the config-developer file.

MartinCussacMF commented 5 months ago

@schlunma I'm pretty sure that works with *.json files, and I did so following the advice from @ehogan (see issue #2364). @valeriupredoi I checked some of the boxes I thought were relevant, don't hesitate to correct me if I'm wrong about some.

schlunma commented 5 months ago

All right, I will have a look at that tomorrow.

valeriupredoi commented 5 months ago

cheers fellas! Boxes look good @MartinCussacMF :+1: Now, the less important thing is to see if JSONs are actually loadable :grin:

schlunma commented 5 months ago

I tried to reproduce this, but unfortunately couldn't get it to work. I keep getting errors like

Unable to load CMOR table (project) 'native6' for variable 'thisisacustomvariable' with mip 'HOMP39Amon'

even though I have a custom table CMOR_HOMP39Amon.json:

{
    "Header": {
        "data_specs_version": "01.00.21",
        "table_id": "Table CNRM_HOMP39Amon",
        "realm": "atmos",
        "cmor_version": "3.2",
        "table_date": "27 Feb 2018",
        "missing_value": "1e20",
        "product": "model-output",
        "approx_interval": "0.250000",
        "generic_levels": "",
        "mip_era": "CMIP6",
        "Conventions": "CF-1.7 CMIP-6.0"
    },
    "variable_entry": {
        "thisisacustomvariable": {
            "frequency": "mon",
            "modeling_realm": "atmos",
            "standard_name": "air_temperature",
            "units": "K",
            "cell_methods": "area: time: mean",
            "cell_measures": "area: areacella",
            "long_name": "THIS IS REALLY THE VARIABLE I AM LOOKING FOR",
            "comment": "Air Temperature",
            "dimensions": "longitude latitude plev7hm time",
            "out_name": "thisisacustomvariable",
            "type": "real",
            "positive": "",
            "valid_min": "",
            "valid_max": "",
            "ok_min_mean_abs": "",
            "ok_max_mean_abs": "",
            "cnrm_priority": "1"
        }
    }
}

(This is actually what I would have expected).

@MartinCussacMF could you please send me the

recipe
config-user.yml
config-developer.yml
custom tables
log files

you were using so I can try to reproduce this?

valeriupredoi commented 5 months ago

many thanks for testing @schlunma :beer: Very possible we'll have to plop a JSON loader there

ehogan commented 5 months ago

We use custom tables in CMIP6 format; cmor_type: 'CMIP6' just needs specifying in a new project section in the config-developer.yml file 🥳

schlunma commented 5 months ago

@ehogan could you maybe send your corresponding config-developer.yml file? I am not entirely sure if I understand you correctly.

ehogan commented 5 months ago

@ehogan could you maybe send your corresponding config-developer.yml file? I am not entirely sure if I understand you correctly.

Sure thing! Here's the diff of the changes we made compared to the v2.9.0 developer config file (we created an ESMVal project to use for CMEW); hopefully it makes sense?:

25,26d24
< # custom:
< #   cmor_path: ~/my/own/custom_tables
28a27,42
> custom:
>   cmor_path: /mip_tables/GCModelDev/0.0.9
> 
> ESMVal:
>   cmor_strict: true
>   input_dir:
>     default: '/'
>     BADC: '{activity}/{institute}/{dataset}/{exp}/{ensemble}/{mip}/{short_name}/{grid}/{version}'
>     DKRZ: '{activity}/{institute}/{dataset}/{exp}/{ensemble}/{mip}/{short_name}/{grid}/{version}'
>     ESGF: '{project}/{activity}/{institute}/{dataset}/{exp}/{ensemble}/{mip}/{short_name}/{grid}/{version}'
>     ETHZ: '{exp}/{mip}/{short_name}/{dataset}/{ensemble}/{grid}/'
>     SYNDA: '{activity}/{institute}/{dataset}/{exp}/{ensemble}/{mip}/{short_name}/{grid}/{version}'
>   input_file: '{short_name}_{mip}_{dataset}_{exp}_{ensemble}_{grid}*.nc'
>   output_file: '{project}_{dataset}_{mip}_{exp}_{ensemble}_{short_name}_{grid}'
>   cmor_type: 'CMIP6'
>   cmor_default_table_prefix: 'GCModelDev_'

It works for us! 🤪

schlunma commented 5 months ago

Hmm...I'm even more confused now 😆

Sorry for all the questions, but would you also be able to share an example recipe, and one of the table files? These table files are then only stored in /mip_tables/GCModelDev/0.0.9? There is no esmvalcore/cmor/tables/esmval? I really don't get how the ESMVal project can access the cmor_path defined in the custom table, especially since it's defined with cmor_strict: true...

ehogan commented 5 months ago

Hmm...I'm even more confused now 😆

Oh no, sorry! 😝

Sorry for all the questions, but would you also be able to share an example recipe, and one of the table files? These table files are then only stored in /mip_tables/GCModelDev/0.0.9?

Ah, yes, we did update the project in one of the datasets in the recipe (we're using the radiation budget recipe for the CMEW prototype) from CMIP6 to ESMVal (to match with what was defined in the config-developer.yml file). The table files are a direct copy and duplicate of all the CMIP6 MIP tables (including the CVs), with some extra variables added to some of the tables (that the MO are using for model development purposes, but that CMEW isn't using right now), so if you made a copy of all the CMIP6 tables into a new location and added that to the cmor_path then that should reflect our setup 👍 (Edited to add that cmor_default_table_prefix: 'GCModelDev_' would need removing from the config-developer.yml file if a direct copy of the CMIP6 tables is used; the MIP tables in /mip_tables/GCModelDev/0.0.9 are named, e.g. GCModelDev_Amon.json.)

There is no esmvalcore/cmor/tables/esmval?

Correct 👍

I really don't get how the ESMVal project can access the cmor_path defined in the custom table, especially since it's defined with cmor_strict: true...

That I don't know! 🤔 I assumed cmor_strict referred to the CMOR checks, which would be just as strict, since we are using exact copies of the CMIP6 MIP tables 🤷

ehogan commented 5 months ago

I also meant to add that we could demo CMEW at the upcoming workshop, if that would be helpful? 😊

schlunma commented 5 months ago

Thanks @ehogan, that already clears things up! 😄

One more (hopefully last) question: since you mentioned that you are using exact copies of the CMIP6 tables with some additional variables, but you are not using those additional variables right now, are you sure that the tables in /mip_tables/GCModelDev/0.0.9 are used? With the way you set up the ESMVal project (cmor_type: CMIP6), the CMIP6 tables would be read by default (i.e., the ones in esmvalcore/cmor/tables/cmip6). You could for example change an entry from a variable you are using and check if that is reflected in your output.

From the code, I think you need to specify cmor_path for the ESMVal project, i.e.,

ESMVal:
  cmor_path: /mip_tables/GCModelDev/0.0.9
  cmor_strict: true
  ...

I assumed cmor_strict referred to the CMOR checks, which would be just as strict, since we are using exact copies of the CMIP6 MIP tables 🤷

No, cmor_strict only controls if other variables or the custom tables may be searched. I also find the name slightly misleading 😬

From the doc: "cmor_strict: if this is set to false, the CMOR table will be extended with variables from the Custom CMOR tables (by default loaded from the esmvalcore/cmor/tables/custom directory) and it is possible to use variables with a mip which is different from the MIP table in which they are defined. Note that this option is always enabled for derived variables."

ehogan commented 5 months ago

Thanks @ehogan, that already clears things up! 😄

Yay! 🥳

One more (hopefully last) question: since you mentioned that you are using exact copies of the CMIP6 tables with some additional variables, but you are not using those additional variables right now, are you sure that the tables in /mip_tables/GCModelDev/0.0.9 are used? With the way you set up the ESMVal project (cmor_type: CMIP6), the CMIP6 tables would be read by default (i.e., the ones in esmvalcore/cmor/tables/cmip6). You could for example change an entry from a variable you are using and check if that is reflected in your output.

I believe they are, since the CVs in GCModelDev contain changes that are reflected in our outputs; compare some of the global attributes in the output that uses the GCModelDev MIP tables:

% ncdump -h recipe_radiation_budget_20240409_083347/preproc/single_value_radiation_budget/rsdt/ESMVal_UKESM1-0-LL_Amon_amip_r1i1p1f1_rsdt_gn_1993-1993.nc
netcdf ESMVal_UKESM1-0-LL_Amon_amip_r1i1p1f1_rsdt_gn_1993-1993 {
[...]
// global attributes:
        :activity_id = "ESMVal" ;
        :cmor_version = "3.7.3" ;
        :cv_version = "GCModelDev v0.0.9" ;
        :data_specs_version = "GCModelDev v0.0.9" ;
        :further_info_url = "none" ;
        :license = "GCModelDev model data is licensed under the Open Government License v3 (https://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/)" ;
        :mip_era = "GCModelDev" ;
        :table_info = "Creation Date:(28 May 2020) MD5:136a5527cb291b78498f710725f53511" ;
        :title = "UKESM1-0-LL output prepared for GCModelDev" ;

with the global attributes in the output that uses the CMIP6 MIP tables:

% ncdump -h recipe_radiation_budget_20240409_083347/preproc/single_value_radiation_budget/rsdt/CMIP6_HadGEM3-GC31-LL_Amon_historical_r1i1p1f3_rsdt_gn_1993-1993.nc
netcdf CMIP6_HadGEM3-GC31-LL_Amon_historical_r1i1p1f3_rsdt_gn_1993-1993 {
[...]
// global attributes:
        :activity_id = "CMIP" ;
        :cmor_version = "3.4.0" ;
        :cv_version = "6.2.20.1" ;
        :data_specs_version = "01.00.29" ;
        :further_info_url = "https://furtherinfo.es-doc.org/CMIP6.MOHC.HadGEM3-GC31-LL.historical.none.r1i1p1f3" ;
        :license = "CMIP6 model data produced by the Met Office Hadley Centre is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License (https://creativecommons.org/licenses). Consult https://pcmdi.llnl.gov/CMIP6/TermsOfUse for terms of use governing CMIP6 output, including citation requirements and proper acknowledgment. Further information about this data, including some limitations, can be found via the further_info_url (recorded as a global attribute in this file) and at https://ukesm.ac.uk/cmip6. The data producers and data providers make no warranty, either express or implied, including, but not limited to, warranties of merchantability and fitness for a particular purpose. All liabilities arising from the supply of the information (including any liability arising in negligence) are excluded to the fullest extent permitted by law." ;
        :mip_era = "CMIP6" ;
        :table_info = "Creation Date:(13 December 2018) MD5:2b12b5db6db112aa8b8b0d6c1645b121" ;
        :title = "HadGEM3-GC31-LL output prepared for CMIP6" ;

I assumed cmor_strict referred to the CMOR checks, which would be just as strict, since we are using exact copies of the CMIP6 MIP tables 🤷

No, cmor_strict only controls if other variables or the custom tables may be searched. I also find the name slightly misleading 😬

Ah, so maybe the variable tables aren't being used, but the CVs are? I will check this ...

From the code, I think you need to specify cmor_path for the ESMVal project, i.e.,
ESMVal:
  cmor_path: /mip_tables/GCModelDev/0.0.9
  cmor_strict: true
  ...

Hopefully this is the solution if I find the variable tables aren't being used! 😊

schlunma commented 5 months ago

Sorry, one more comment/question from me 😄

I think these attributes are not written by ESMValTool. For example, if I look at a preprocessed ERA5 file (project native6), I get:

$ ncdump -h native6_ERA5_reanaly_v1_Amon_tas_1999-2005.nc
netcdf native6_ERA5_reanaly_v1_Amon_tas_1999-2005 {
dimensions:
        time = 84 ;
        lat = 721 ;
        lon = 1440 ;
        bnds = 2 ;
variables:
        float tas(time, lat, lon) ;
                ...

// global attributes:
                :comment = "Contains modified Copernicus Climate Change Service Information 2024" ;
                :Conventions = "CF-1.7" ;
                :software = "Created with ESMValTool v2.11.0.dev83+g3f37b31da.d20240320" ;
}

An easy way to test if the tables are really read is to change a long_name in a variable/table that you are actually using an check if that is reflected in the output.

MartinCussacMF commented 5 months ago

Hello @schlunma ! Sorry I haven't really been available last few days. Are you still interested in me uploading the configuration in wich I'm able to perform a recipe with custom CMIP6 like tables?

schlunma commented 5 months ago

@MartinCussacMF, yes please! Thanks!

ehogan commented 5 months ago

Sorry, one more comment/question from me 😄

I think these attributes are not written by ESMValTool.

You are absolutely correct, apologies for my slowness! 😴

I have proven to myself that my custom MIP tables are not being used by ESMValTool. When I move cmor_path into my ESMVal project in the config-developer.yml file I get the error:

Unable to load CMOR table (project) 'ESMVal' for variable 'hfls' with mip 'Amon'

which I what you are expecting (I'm guessing?) 🤔 I have tried to play around with the naming (since our MIP tables have the prefix GCModelDev) but I haven't gotten it working yet.

Are you still interested in me uploading the configuration in wich I'm able to perform a recipe with custom CMIP6 like tables?

@MartinCussacMF, I am now also interested! 🤪

valeriupredoi commented 5 months ago

well this escalated quickly :rofl:

schlunma commented 5 months ago

Thanks Emma! I am relieved that the code does what I expected it to do (not that you got an error!) 😄

MartinCussacMF commented 5 months ago

Well here is the awaited recipe/config I'm using atm:

the recipe: recipe_zonal_sig_emich4.yml.txt
config files: config-user.yml.txt & config-developer.yml.txt
custom table and coordinates: CMOR_HOMP39Amon.json & CMOR_coordinate.json
and finally the log: main_log.txt

Let me know if you have questions/want some more info.

ehogan commented 5 months ago

I figured it out! 🥳 There were two issues:

1) I did need to move cmor_path into the ESMVal project in the config-developer.yml file, as @schlunma suggested in https://github.com/ESMValGroup/ESMValCore/pull/2368#issuecomment-2044556436; thank you! 😁 Other than that, the differences to my config-developer.yml file are still as in https://github.com/ESMValGroup/ESMValCore/pull/2368#issuecomment-2043038135 (is it possible that cmor_strict: true works in this case because cmor_path is specified directly within the project and provides a complete copy of all MIP tables?) 1) ESMValTool requires that the MIP tables are stored in a Tables directory within the path provided by cmor_path; would it be possible to relax this / make it configurable?

Evidence:

% ncdump -h recipe_radiation_budget_20240409_181752/preproc/single_value_radiation_budget/rsdt/ESMVal_UKESM1-0-LL_Amon_esmval-dummy-a_r1i1p1f1_rsdt_gn_1993-1993.nc
netcdf ESMVal_UKESM1-0-LL_Amon_esmval-dummy-a_r1i1p1f1_rsdt_gn_1993-1993 {
dimensions:
    bnds = 2 ;
variables:
    float rsdt ;
        rsdt:_FillValue = 1.e+20f ;
        rsdt:standard_name = "toa_incoming_shortwave_flux" ;
        rsdt:long_name = "Will my edit show in the output netCDF file? I hope so!" ;
        rsdt:units = "W m-2" ;
        rsdt:cell_methods = "area: time: mean longitude: latitude: mean time: mean" ;
        rsdt:coordinates = "lat lon time" ;

🤪🥳

schlunma commented 5 months ago

Well here is the awaited recipe/config I'm using atm:

* the recipe: [recipe_zonal_sig_emich4.yml.txt](https://github.com/ESMValGroup/ESMValCore/files/14920052/recipe_zonal_sig_emich4.yml.txt)

* config files: [config-user.yml.txt](https://github.com/ESMValGroup/ESMValCore/files/14920061/config-user.yml.txt) & [config-developer.yml.txt](https://github.com/ESMValGroup/ESMValCore/files/14920063/config-developer.yml.txt)

* custom table and coordinates: [CMOR_HOMP39Amon.json](https://github.com/ESMValGroup/ESMValCore/files/14920094/CMOR_HOMP39Amon.json) & [CMOR_coordinate.json](https://github.com/ESMValGroup/ESMValCore/files/14920097/CMOR_coordinate.json)

* and finally the log: [main_log.txt](https://github.com/ESMValGroup/ESMValCore/files/14920104/main_log.txt)

Let me know if you have questions/want some more info.

Thanks for that @MartinCussacMF!

Sorry for being so stubborn, but with this evidence I am pretty sure that your custom tables are also NOT read (similar to @ehogan's case).

Here's what's happening (I think):

You defined your custom project CNRM with cmor_type: 'CMIP6' (note that this is very different to what we call custom tables; custom tables are simply another directory which can be used by all derived variables and any other project defined with cmor_strict: false). Since you did not specify a cmor_path for this, the tables located at esmvalcore/cmor/tables/cmip6/Tables are read by default.
ESMValTool will now look for a table esmvalcore/cmor/tables/cmip6/Tables/CMIP6_HOMP39Amon (your mip defined in the recipe), which it does not find. However, since project CNRM is defined with cmor_strict: false, ESMValTool is allowed to search other mips. One appearance of the variable ch4 is for example in the CMIP6 Amon table, which will now be used.

As I've already mentioned to @ehogan, it's easy to verify this: could you please alter the long_name of ch4 in your CMOR_HOMP39Amon.json table and check if that is reflected in the output of the tool (the files located in the preproc directory). Alternatively, you could remove the entire ch4 block in your custom tables or use cmor_strict: true for your CNRM project. If the run fails, your tables have been read. If the run still works, your tables are not read.

schlunma commented 5 months ago

I figured it out! 🥳 There were two issues:

That's great to hear @ehogan!

is it possible that cmor_strict: true works in this case because cmor_path is specified directly within the project and provides a complete copy of all MIP tables?

The main reason it works is that you have a complete copy of the MIP tables. Without cmor_path, the default CMIP6 tables would be read, so it would still work as long as you only use CMIP6 variables (you involuntary tested this earlier 😁). cmor_strict: true means that the tool will only check the MIP you defined in the recipe, e.g., for mip: Amon, only the table ..._Amon.json is searched. If a variable is not there, an error will be raised. With cmor_strict: false, this is relaxed, and also other MIP tables of the same project (e.g., ..._Lmon.json) and the custom tables are searched, but never tables from another project.

ESMValTool requires that the MIP tables are stored in a Tables directory within the path provided by cmor_path; would it be possible to relax this / make it configurable?

I think it's certainly possible, but let's discuss that in another issue 👍

MartinCussacMF commented 5 months ago

Well here is the awaited recipe/config I'm using atm:

* the recipe: [recipe_zonal_sig_emich4.yml.txt](https://github.com/ESMValGroup/ESMValCore/files/14920052/recipe_zonal_sig_emich4.yml.txt)

* config files: [config-user.yml.txt](https://github.com/ESMValGroup/ESMValCore/files/14920061/config-user.yml.txt) & [config-developer.yml.txt](https://github.com/ESMValGroup/ESMValCore/files/14920063/config-developer.yml.txt)

* custom table and coordinates: [CMOR_HOMP39Amon.json](https://github.com/ESMValGroup/ESMValCore/files/14920094/CMOR_HOMP39Amon.json) & [CMOR_coordinate.json](https://github.com/ESMValGroup/ESMValCore/files/14920097/CMOR_coordinate.json)

* and finally the log: [main_log.txt](https://github.com/ESMValGroup/ESMValCore/files/14920104/main_log.txt)

Let me know if you have questions/want some more info.

Thanks for that @MartinCussacMF!

Sorry for being so stubborn, but with this evidence I am pretty sure that your custom tables are also NOT read (similar to @ehogan's case).

No worries, I'm rather new to the code so there are things I don't get right away.

Here's what's happening (I think):
* You defined your custom **project** CNRM with `cmor_type: 'CMIP6'` (note that this is very different to what we call custom **tables**; custom tables are simply another directory which can be used by all derived variables and any other project defined with `cmor_strict: false`). Since you did not specify a `cmor_path` for this, the tables located at `esmvalcore/cmor/tables/cmip6/Tables` are read by default.

* ESMValTool will now look for a table `esmvalcore/cmor/tables/cmip6/Tables/CMIP6_HOMP39Amon` (your `mip` defined in the recipe), which it does not find. However, since project CNRM is defined with `cmor_strict: false`, ESMValTool is [allowed to search other mips](https://docs.esmvaltool.org/projects/ESMValCore/en/latest/quickstart/configure.html#project-cmor-table-configuration). One appearance of the variable `ch4` is for example in the [CMIP6 Amon](https://github.com/ESMValGroup/ESMValCore/blob/d128753fd6f6d2a10ba9b2b0c500b369fff1dd95/esmvalcore/cmor/tables/cmip6/Tables/CMIP6_Amon.json#L107-L124) table, which will now be used.
As I've already mentioned to @ehogan, it's easy to verify this: could you please alter the long_name of ch4 in your CMOR_HOMP39Amon.json table and check if that is reflected in the output of the tool (the files located in the preproc directory). Alternatively, you could remove the entire ch4 block in your custom tables or use cmor_strict: true for your CNRM project. If the run fails, your tables have been read. If the run still works, your tables are not read.

Indeed, I can't see the modified long name in the preproc directory even though I changed it in CMOR_HOMP39Amon.json, and my run crashes if I use cmor_strict: true for my CNRM project. However there are still hints that esmvaltool is using the right HOMP39Amon files (sorry, I just want to undestand what esmvaltool is doing...):

ncdump return the following, where you can clearly see that the plev has a lenght of 39 which corresponds to the HOMP39Amon grid and not the Amon one, which only has 19 vertical levels.

netcdf CNRM_CNRM-ESM2-1_HOMP39Amon_AESM-amip_ARPV7-1.03c_piclim_racmobus_emich4_2.5TgN_r1i1p1f2_ch4_gr_1850-1879 {
dimensions:
time = 360 ;
plev = 39 ;
lat = 90 ;
bnds = 2 ;
variables:
float ch4(time, plev, lat) ;
    ch4:_FillValue = 1.e+20f ;
    ch4:standard_name = "mole_fraction_of_methane_in_air" ;
    ch4:long_name = "Mole Fraction of CH4" ;
    ch4:units = "mol mol-1" ;
    ch4:cell_methods = "area: time: mean longitude: mean" ;
    ch4:coordinates = "lon" ;

I have been able to plot some variables such as no2 for which I only have output on the HOMP39Amon table.

Is it possible that esmvaltool uses the Amon table for the variable attributes but still uses the HOMP39Amon files found in my experiment directory ? Thanks for your patience.

schlunma commented 5 months ago

Thanks for confirming 👍

ncdump return the following, where you can clearly see that the plev has a lenght of 39 which corresponds to the HOMP39Amon grid and not the Amon one, which only has 19 vertical levels.

That's because you are using the following preprocessor:

pp27:
    extract_levels:
      levels: {cmor_table: CMIP6, coordinate: plev39}
      coordinate: air_pressure
      scheme: linear

This will extract the levels defined by the plev39 coordinate of CMIP6, which has 39 levels. If you don't use this preprocessor, ESMValTool will simply use the input levels without modifying them. If they don't match the number of desired levels in the table, you will only get a warning (no error).

I have been able to plot some variables such as no2 for which I only have output on the HOMP39Amon table.

As mentioned, since cmor_strict: false, ESMValTool will search every other mip in the CMIP6 project, and it happens to find no2 in the AERmon table.

Is it possible that esmvaltool uses the Amon table for the variable attributes but still uses the HOMP39Amon files found in my experiment directory ? Thanks for your patience.

Nope, I am 100% sure now that the HOMP39Amon table is not used in this case. If you want to use it, please have a look at Emma's comment. It basically boils down to adding a cmor_path to your CNRM project.

I am closing this now since these changes are not necessary anymore (custom tables need to be in CMIP5 format). Feel free to reopen if you think that other changes in the documentation are necessary to clear this up!

MartinCussacMF commented 5 months ago

Thanks for the explanation. I'm not sure I get everything since i still am ploting some fields that I only output through the HOMP39Amon table and no other one like Amon or AERmon. Anyway, is it worth keeping issue #2364 open ? Loading .json custom table would be nice since they are the CMIP6 standard.

schlunma commented 5 months ago

Can you post a list of those variables here?

Anyway, is it worth keeping issue #2364 open ? Loading .json custom table would be nice since they are the CMIP6 standard.

Yes, definitely!

MartinCussacMF commented 5 months ago

For instance, for the following recipe: recipe_zonal_sig_no2.yml.txt I plot some zonal means for no2, and in my experiment directory I only have the following file for this variable (this is the result of ls no2*):

no2_HOMP39Amon_CNRM-ESM2-1_AESM-amip_r1i1p1f2_gr_197901-201412.nc

The behaviour is the same for the co variable for instance, the recipe is successful and I get the expected plots.

schlunma commented 5 months ago

You have to differentiate between finding the data (i.e., the files) and finding an appropriate variable in the CMOR tables:

In your recipe, you have mip: HOMP39Amon. Thus, the tool will find data located at no2_HOMP39Amon_CNRM-ESM2-1_AESM-amip_r1i1p1f2_gr_197901-201412.nc.
When looking for CMOR tables, the tool will first look for a table CMIP6_HOMP39Amon.json in the default CMIP6 tables, which it does NOT find. However, since cmor_strict: false, it will also look into other tables. It will then find the variable in the AERmon table, which it will use. Note that this only affects the variables; for ESMValTool, the mip facet will stay HOMP39Amon.

MartinCussacMF commented 5 months ago

Ok I get it now, thanks again for your time @schlunma !

ESMValGroup / ESMValCore