OBS `Tier1/ISCCP/clisccp_ISCCP_L3_V1.0_*` on esmeval/JASMIN is slightly dodgy according to esmvaltool's CMOR checks

[valeriupredoi]

running recipe_autoassess_radiation_rms_cfMon_all.yml gives me this complaint:

esmvalcore.cmor.check.CMORCheckError: There were errors in variable clisccp:
Coordinate plevs has var name plev7 instead of plev
 tau: standard_name should be atmosphere_optical_thickness_due_to_cloud, not None
 clisccp: does not match coordinate rank
 in cube:
ISCCP Cloud Area Fraction / (%)     (time: 1; cloud optical depth: 6; air_pressure: 7; latitude: 72; longitude: 144)

the files are all in /gws/nopw/j04/esmeval/obsdata-v2/Tier1/ISCCP/clisccp_ISCCP_L3_V1.0_* and I see they have been updated on April 8 by our OBS expert @remi-kazeroni - could you have a look please, mate :beer:

[valeriupredoi]

yeah sorry, not cmor-related, I mean I though it is, but even if one uses ignore flag it still goes to the fishes :fish:

[remi-kazeroni]

yeah sorry, not cmor-related, I mean I though it is, but even if one uses ignore flag it still goes to the fishes 🐟

sorry I guess I removed that label accidentally without noticing it 😬

Nevertheless, I have not added that dataset to Mistral myself and I have never worked with that. @axel-lauer could you please have a look whenever possible? Have you used this dataset successfully in the past? The data were added on Mistral on October 2019 and the symbolic links (clisccp_ISCCP_L3V1.0*) last March. I only transfered that to Jasmin using some rsync.

[valeriupredoi]

aha cheers @remi-kazeroni - it might be only some of those datasets are dodgy, and we've not found them until re-running that particular recipe :+1: I'll have a closer look

[axel-lauer]

Some of the variables provided by this obs4mips dataset are used by recipe_autoassess_radiation_rms_cfMon_all.yml. Variable clisccp, however, is currently not used. If I remember correctly, this variable was used in ESMValTool v1.0 but the diagnostics have not been ported completely. So my guess is that this is why this issue has not been found before. As I think it would be good to keep this variable, I tried to extend esmvalcore/cmor/tables/custom/CMOR_coordinates.dat to include the missing coordinate plev7:

!============
axis_entry: plev7
!============
axis: Z
long_name: pressure
must_have_bounds: yes
out_name: plev7
positive: down
requested: 90000. 74000. 62000. 50000. 37500. 24500. 9000.
requested_bounds: 100000. 80000. 80000. 68000. 68000. 56000. 56000. 44000. 44000. 31000. 31000. 18000. 18000. 0.
standard_name: air_pressure
stored_direction: decreasing
units: Pa

and changed the coordianate definition (plevs --> plev7) in esmvalcore/cmor/tables/custom/CMOR_clisccp.dat:

dimensions:        longitude latitude plev7 tau time

For testing purposes, I also removed the standard_name of coordinate tau in esmvalcore/cmor/tables/custom/CMOR_coordinates.dat. When running this example recipe_check_obs_isccp.yml.txt, I get beyond the plev and tau related errors reported by @valeriupredoi but I still this get this error:

esmvalcore.cmor.check.CMORCheckError: There were errors in variable clisccp:
clisccp: does not match coordinate rank
 in cube:
ISCCP Cloud Area Fraction / (%)     (time: 1; cloud optical depth: 6; air_pressure: 7; latitude: 72; longitude: 144)

Not sure why, though. It would be nice to also fix this if possible.

By the way, the symbolic links have been introduced as the naming convention used for the files containing some variables does not follow the naming convention used by most other obs4mips datasets. No idea why this is the case. A possible solution might be to make the data finder more flexible for the obs4mips class.

[valeriupredoi]

cheers @axel-lauer :beer: I'll be having a closer look too, in the near future :grin:

[zklaus]

Hm. Looking at this, there seem to be at least two issues that prevent a timely resolution:

• clisccp is not in the obs4MIPs tables (as far as I can see). The closes one is cl, but that has no dependence on cloud optical depth.
• Our obs4MIPs tables are old compared to upstream at https://github.com/PCMDI/obs4MIPs-cmor-tables

As such, I am bumping this to 2.5.0. Let's try to work on it rather early in the cycle.

[valeriupredoi]

cheers @zklaus - just a mention that there are currently 5 derived variables that depend on clisccp:

esmvalcore/preprocessor/_derive/clhtkisccp.py:        required = [{'short_name': 'clisccp'}]
esmvalcore/preprocessor/_derive/cllmtisccp.py:        required = [{'short_name': 'clisccp'}]
esmvalcore/preprocessor/_derive/clltkisccp.py:        required = [{'short_name': 'clisccp'}]
esmvalcore/preprocessor/_derive/clmtkisccp.py:        required = [{'short_name': 'clisccp'}]
esmvalcore/preprocessor/_derive/clmmtisccp.py:        required = [{'short_name': 'clisccp'}]

and these are all in esmvaltool/recipes/recipe_autoassess_radiation_rms_cfMon_all.yml and the utility esmvaltool/recipes/examples/recipe_preprocessor_derive_test.yml - nothing else is affected :+1:

[schlunma]

We can check the affected recipes before the release, but in case the error persists I will move this to v2.6

[schlunma]

This is still an issue for v2.5.0rc:

esmvalcore.cmor.check.CMORCheckError: Coordinate plevs has var name plev7 instead of plevThere were errors in variable clisccp:
 tau: standard_name should be atmosphere_optical_thickness_due_to_cloud, not NoneThere were errors in variable clisccp:
 clisccp: does not match coordinate rank
in cube:
ISCCP Cloud Area Fraction / (%)     (time: 1; cloud optical depth: 6; air_pressure: 7; latitude: 72; longitude: 144)
    Dimension coordinates:
        time                             x                       -                -            -              -
        cloud optical depth              -                       x                -            -              -
        air_pressure                     -                       -                x            -              -
        latitude                         -                       -                -            x              -
        longitude                        -                       -                -            -              x
    Attributes:
        cmor_version                2.6.1
        comments                    The equal-area monthly mean data are generated from the ISCCP-D1 data by...
        contact                     zhang24@llnl.gov and Robert.Pincus@colorado.edu
        conventions                 CF-1.4
        experiment_id               obs
        frequency                   mon
        institute                   PCMDI (Program for Climate Model Diagnosis and Intercomparison, LLNL, California,...
        institute_id                PCMDI/LLNL
        instrument                  ISCCP
        instrument_id               ISCCP
        invalid_standard_name       isccp_cloud_area_fraction_in_atmosphere_layer
        mip_specs                   CMIP5
        modeling_realm              atmos
        obs_project                 GCM simulator-oriented ISCCP Cloud Product
        obs_project_id              ISCCP
        obs_structure               grid
        obs_type                    Satellite retrieval
        processing_level            L3
        processing_version          V1.0
        product                     observations
        project_id                  obs4MIPs
        reference                   http://climserv.ipsl.polytechnique.fr/cfmip-obs/
        source                      The GCM simulator-oriented ISCCP cloud product derived from the ISCCP D1...
        source_file                 /mnt/lustre02/work/bd0854/DATA/ESMValTool2/OBS/Tier1/ISCCP/clisccp_ISC...
        source_url                  http://climserv.ipsl.polytechnique.fr/cfmip-obs.html
        title                       ISCCP level 3 data prepared for obs4MIPs observation
        valid_max                   100.0
        valid_min                   0.0
loaded from file /mnt/lustre02/work/bd0854/DATA/ESMValTool2/OBS/Tier1/ISCCP/clisccp_ISCCP_L3_V1.0_198501-198501.nc

I suggest we move this to v2.6.

BTW, these error message has a really weird formatting right now. I will open an issue about that.

[valeriupredoi]

yes, I got the same when running recipe_autoassess_radiation_rms_cfMon_all at JASMIN just an hour ago

[valeriupredoi]

looks like this issue gets unearthed everytime one is doing a release, I am faced with the same problem yet again for v2.7 - I think this recipe, namely recipe_autoassess_radiation_rms_cfMon_all.yml is a bit dead since if nobody is using it to come back and nag us here then why even bother keeping it? I'm gonna link this issue to the list of recipes I am running for 2.7 then promptly bump it to M2.8 since why not :grin:

[remi-kazeroni]

Hi @valeriupredoi and @bouweandela, could you please let me know if you think this could be solved in time for v2.8? I reckon we discussed this and were hoping for a update of the OBS4MIPs CMOR tables before addressing the issue, right? If that has not happened yet, shall we shift this issue to the next milestone?

[valeriupredoi]

the only viable solution I can think of that'll fix this is the updating of the obs4mips tables as @zklaus mentions in an ancient reply above - @bouweandela - have we updated those tables? If not, let's do that now? (note the question mark - am not sure how much stuff will hit the fan if we do that, from other recipes)

[bouweandela]

No, they have not been updated because they have no version numbers. I created an issue about it here: https://github.com/ESMValGroup/ESMValCore/issues/1890.

[remi-kazeroni]

I'm bumping this to the next milestone since it seems very unlikely that things can be resolved in time for v2.8

[remi-kazeroni]

Since the recipe recipe_autoassess_radiation_rms_cfMon_all.yml has been retired in https://github.com/ESMValGroup/ESMValTool/pull/3115 and the data is not used in any public recipe, I'm closing this. Feel free to reopen if needed.