recipe_ocean_amoc.yml using CMIP6 data

[helenelang]

Describe the bug I downloaded the recipe from here (https://github.com/ESMValGroup/ESMValTool/blob/main/esmvaltool/recipes/recipe_ocean_amoc.yml), and tried to use CMIP6 data instead of CMIP5, but I ran into problems.

I’m using this version: ESMValTool: 2.4.0

main_log_debug.txt recipe_ocean_amoc.txt

@ledm

[ledm]

It looks like the problem is wth the msftmyz time series section. That is a CMIP5 variable which doesn't exist in CMIP6, and was replaced by msftyz or msftmz.

This is a problem that we discussed and solved for IPCC-AR6, but it doesn't look like it ever got merged into the master branch of ESMValCore.

The solution that we used was in the following amoc.py variable derivation: https://github.com/ESMValGroup/ESMValCore/blob/development_amoc_cmip6/esmvalcore/preprocessor/_derive/amoc.py

For reference, you're likely using this code: https://github.com/ESMValGroup/ESMValCore/blob/main/esmvalcore/preprocessor/_derive/amoc.py

[helenelang]

Thank you!

I have now updated the amoc.py, but I'm still running into problems. There seems to be problem with diag_transects/Atlantic_Stream_function_zonal_mean (and 'msftmyz'). Please see attached file.

main_log_debug.txt

[ledm]

Nice work! I think you're still hitting the same problem though:

2022-05-06 07:26:27,110 UTC [34195] ERROR   esmvalcore._recipe:1051 Unable to load CMOR table (project) 'CMIP6' for variable 'msftmyz' with mip 'Omon'

The problem is that there's no msftmyz variable in CMIP6. You will need to edit the diag_transects section of your recipe to reflect the difference in naming between CMIP5 and CMIP6.

[helenelang]

Thank you, but I'm unfortunately not able to find the diag_transects. I'm looking in these locations: esmvaltool-python-2.4.0-pyhd8ed1ab_0 --> site-packages-->esmvaltool-->diag_scripts esmvalcore-2.4.0-pyhd8ed1ab_2 --> site-packages-->esmvalcore-->preprocessors

Do I look in the wrong places? (amoc.py was found and changed following the second line above)

[zklaus]

@ledm is talking about the recipe, i.e. the yaml file. There, in the diagnostics section, you will find the diag_transect here and in there the variable msftmyz, which was used in CMIP5. It was replaced in CMIP6 by the two variables msftmz and msftyz to make the distinction between "strictly northwards" (meridional, msftmz) and "roughly northwards following the grid" (y, msftyz) clear.

So you will have to change that variable to the applicable one for your selection of models.

[dhohn]

Hi all,

I happened to also run across this issue. Thanks for the pointers so far. I think theres more to it:

1. in the amoc.py linked by @ledm theres still reference to msftmyz
2. Deprecated iris method extract_strict is used there too.
3. The region/basin coordinate doesnt pass cmor checks for a couple of datasets I tried. Here im not sure if the cmip6 vocab or the datasets are the problem...

I've collected the changes I made in my fork. In that state I can run the following datasets:

  - {dataset: CanESM5,        project: CMIP6, exp: 1pctCO2,                          ensemble: r1i1p2f1, grid: gn, timerange: "1850/1999"}
  - {dataset: ACCESS-ESM1-5,  project: CMIP6, exp: 1pctCO2,                          ensemble: r1i1p1f1, grid: gn, start_year: 101, end_year: 200}
  - {dataset: MPI-ESM1-2-LR,  project: CMIP6, exp: 1pctCO2,                              ensemble: r1i1p1f1, grid: gn, timerange: "1850/1999"}
  - {dataset: UKESM1-0-LL,    project: CMIP6, exp: 1pctCO2,                          ensemble: r1i1p1f2, grid: gn, timerange: "1850/1999"}

In all of them the coord looks like:

cube.coord("region").summary()
AuxCoord :  region / (unknown)
    points: [
        atlantic_arctic_ocean, global_ocean         ,
        indian_pacific_ocean ]
    shape: (3,)
    dtype: <U21
    standard_name: 'region'
    long_name: 'ocean basin'
    var_name: 'sector'

NB that I changed the cmip6 Table json to get this working which expects the var_name to be "basin". I guess thats not the proper way? Would this then rather warrant dataset fixes instead?

This one doesnt work as the region/basin coord is different:

  - {dataset: CESM2,          project: CMIP6, exp: 1pctCO2,                          ensemble: r1i1p1f1, grid: gn, timerange: "0001/150"}

Coord looks like:

cube.coord("region").summary()
DimCoord :  region / (1)
    points: [0, 1, 2]
    shape: (3,)
    dtype: int32
    standard_name: 'region'
    var_name: 'basin'
    attributes:
        coords     'region'
        requested  'atlantic_arctic_ocean=0, indian_pacific_ocean=1, global_ocean=2'
        title      'ocean basin'
        type       'int'

Theres no error. But the region constraints in the amoc derivation simply returns None.

Any hint how to deal with it would also be welcome.

[helenelang]

Thanks for your responses!

The recipe is corrected with the correct variable, but it still stops running:

2022-05-11 10:38:15,290 UTC [4863] INFO esmvalcore._recipe:607 Found input files for CanESM5 2022-05-11 10:38:15,718 UTC [4863] INFO esmvalcore._recipe:880 PreprocessingTask diag_transects/msftmz created. 2022-05-11 10:38:15,719 UTC [4863] DEBUG esmvalcore._recipe:881 PreprocessingTask diag_transects/msftmz will create the files: /mnt/lustre02/work/bk1088/b381145/esmvaltool_output/recipe_ocean_amoc_20220511_103807/preproc/diag_transects/msftmz/CMIP6_CanESM5_Omon_historical-ssp126_r1i1p1f1_msftmz_gn_1993-2014.nc 2022-05-11 10:38:15,719 UTC [4863] INFO esmvalcore._recipe:1491 Creating diagnostic task diag_transects/Atlantic_Stream_function_zonal_mean 2022-05-11 10:38:15,720 UTC [4863] ERROR esmvalcore._recipe:1049 Could not create all tasks 2022-05-11 10:38:15,720 UTC [4863] ERROR esmvalcore._recipe:1051 Unable to load CMOR table (project) 'CMIP6' for variable 'msftmyz' with mip 'Omon'

Should I change something in the cmip6 Table json?

[valeriupredoi]

@helenelang msftmyz is a CMIP5-only Omon variable - you have an extra "m" in its name - try mstfyz and also see an old issue that raised the issue for the change of name https://github.com/ESMValGroup/ESMValCore/issues/310

[valeriupredoi]

Hi all,

I happened to also run across this issue. Thanks for the pointers so far. I think theres more to it:

1. in the `amoc.py` linked by @ledm theres still reference to `msftmyz`

2. Deprecated iris method `extract_strict` is used there too.

3. The region/basin coordinate doesnt pass cmor checks for a couple of datasets I tried. Here im not sure if the cmip6 vocab or the datasets are the problem...

I've collected the changes I made in my fork. In that state I can run the following datasets:

  - {dataset: CanESM5,        project: CMIP6, exp: 1pctCO2,                          ensemble: r1i1p2f1, grid: gn, timerange: "1850/1999"}
  - {dataset: ACCESS-ESM1-5,  project: CMIP6, exp: 1pctCO2,                          ensemble: r1i1p1f1, grid: gn, start_year: 101, end_year: 200}
  - {dataset: MPI-ESM1-2-LR,  project: CMIP6, exp: 1pctCO2,                              ensemble: r1i1p1f1, grid: gn, timerange: "1850/1999"}
  - {dataset: UKESM1-0-LL,    project: CMIP6, exp: 1pctCO2,                          ensemble: r1i1p1f2, grid: gn, timerange: "1850/1999"}

In all of them the coord looks like:

cube.coord("region").summary()
AuxCoord :  region / (unknown)
    points: [
        atlantic_arctic_ocean, global_ocean         ,
        indian_pacific_ocean ]
    shape: (3,)
    dtype: <U21
    standard_name: 'region'
    long_name: 'ocean basin'
    var_name: 'sector'

NB that I changed the cmip6 Table json to get this working which expects the var_name to be "basin". I guess thats not the proper way? Would this then rather warrant dataset fixes instead?

This one doesnt work as the region/basin coord is different:

  - {dataset: CESM2,          project: CMIP6, exp: 1pctCO2,                          ensemble: r1i1p1f1, grid: gn, timerange: "0001/150"}

Coord looks like:

cube.coord("region").summary()
DimCoord :  region / (1)
    points: [0, 1, 2]
    shape: (3,)
    dtype: int32
    standard_name: 'region'
    var_name: 'basin'
    attributes:
        coords     'region'
        requested  'atlantic_arctic_ocean=0, indian_pacific_ocean=1, global_ocean=2'
        title      'ocean basin'
        type       'int'

Theres no error. But the region constraints in the amoc derivation simply returns None.

Any hint how to deal with it would also be welcome.

@dhohn cheers! Could you please open a separate issue for that if you reckon needs addressing? Helps us better tackle different things in different places - if it proves out to be an ESMValCore issue then I'll move the issue there :beer:

[helenelang]

1) I have updated the amoc.py:

@staticmethod
def required(project):
    """Declare the variables needed for derivation."""
    if project == 'CMIP5':
        required = [{'short_name': 'msftmyz'}]
    if project == 'CMIP6':
    #     required = [{'short_name': 'msftyz'}]
        required = [{'short_name': 'msftmyz', 'optional': True, },
                    {'short_name': 'msftyz', 'optional': True, },
                    {'short_name': 'msftmz', 'optional': True, }]

    return required

2) The recipe is updated (using either 'msftyz' or 'msftmz', depending on the model) 3) I still run into problems, please see the attached files. main_log_debug.txt recipe_ocean_amoc.txt

[valeriupredoi]

@helenelang cheers for moving the comment here! Two points wrt chaning amoc.py:

• I see required = [{'short_name': 'msftmyz', 'optional': True, }, in your CMIP6 optional - that should be msftyz instead
• if you are using an ESMValCore installed from conda as a dependency of ESMValTool that change needs to happen in the conda-installed source, make sure that that the code change is in the package that the environment uses

And a general point - please format your code inserted in GH comments with triple reverse quotes like this:

import something

def func():
    return stuff

so one can read nicely what the code does :beer:

@ledm could you please open an issue about fixing amoc.py to account for CMIP6 variables derivation too?

[dhohn]

Thanks @valeriupredoi I see that the issue you linked Core#310 and Core#359 basically describe @helenelang and my problems. Im afraid some (most?) of the work in https://github.com/ESMValGroup/ESMValCore/tree/development_amoc_cmip6 has gone stale by now. A shame it wasn't merged! (I didn't see an open PR)

If its welcome I can clean up the fixes in my fork a bit and submit a PR (probably next week)

[valeriupredoi]

@dhohn @helenelang I have opened a PR to fix this https://github.com/ESMValGroup/ESMValCore/pull/1577 - hopefully it'll get through quickly and then you guys can use the development branch of ESMValCore to get your CMIP6 amoc stuff going :+1: