search

ESMValGroup / ESMValCore

ESMValCore: A community tool for pre-processing data from Earth system models in CMIP and running analysis scripts.
https://www.esmvaltool.org
Apache License 2.0
42 stars 38 forks source link

Loading CMIP6 data #34

Closed ledm closed 4 years ago

ledm commented 5 years ago

Hi everone,

Related to the PR ESMValGroup/ESMValTool#1015. That PR address the problems with UKESM & MIROC6. The problem both of these models was that they used the terms latitude and longitude instead of lat & lon.

The other models that are available now on JASMIN are:

These three all seem to work fine with ESMValTool at the moment!

The models below don't work at the moment, and I'm not sure how to address their problems.

The two BCC models use irregular grids, however they haven't used the standard terms to describe latitude, longitude, cell index along first dimension and cell index along second dimension (lat, lon and i ,j). Instead they have used: lat, lon and latitude ,longitude). Basically, this is rather confusing because latitude is a 2D array and lat is a one D array. I don't think iris is very happy with this dataset. This causes the following error message:

esmvaltool.cmor.check.CMORCheckError: There were errors in variable tos:
tos: does not match coordinate rank: 3 != 1

The IPSL dataset includes the terms nav_lat and nav_lon instead of lat and lon, which is annoying but not the end of the world. The files I'm looking at also include an area array (instead of areacello) which causes trouble when loading the cube, because it wants to load the netcdf as a list of cubes instead of as a single cube. This produces the error:

ValueError: Cubes were not reduced to one afterfixing: 
0: cell_area / (m2)                    (-- : 332; -- : 362)
1: sea_surface_temperature / (degC)    (time: 1980; -- : 332; -- : 362)

Anyone have any ideas how to address these issues?

ledm commented 5 years ago

I've found a way to add the IPSL fix into PR ESMValGroup/ESMValTool#1015, here. However, I want to know whether there is a way to take an cell_area cube and set it as the area_cello attribute of another cube. Any ideas?

ledm commented 5 years ago

Okay, I've managed a fix for the two BCC models. These fixes have only been tested with the recipe esmvaltool/recipes/recipe_ocean_temperature_cmip6.yml

valeriupredoi commented 5 years ago

so here's a question for you @ledm - what kind of UKESM1 did you look at Jasmin BADC that you noticed the var name issues? I am looking at the UKESM1 data pushed to ESGF on April 6 and all looks fine to me:

In [1]: import iris                                                                                                                

In [2]: c = iris.load_cube('/badc/cmip6/data/CMIP6/CMIP/MOHC/UKESM1-0-LL/historical/r1i1p1f2/Amon/tas/gn/latest/tas_Amon_UKESM1-0-L
   ...: L_historical_r1i1p1f2_gn_195001-201412.nc')                                                                                
/home/users/valeriu/anaconda3Feb19/lib/python3.7/site-packages/iris/fileformats/cf.py:798: UserWarning: Missing CF-netCDF measure variable 'areacella', referenced by netCDF variable 'tas'
  warnings.warn(message % (variable_name, nc_var_name))

In [3]: c.coord('latitude').var_name                                                                                               
Out[3]: 'lat'

In [4]: ls -la /badc/cmip6/data/CMIP6/CMIP/MOHC/UKESM1-0-LL/historical/r1i1p1f2/Amon/tas/gn/latest/tas_Amon_UKESM1-0-LL_historical_
   ...: r1i1p1f2_gn_195001-201412.nc                                                                                               
lrwxrwxrwx 1 badc badcint 79 Apr 10 02:43 /badc/cmip6/data/CMIP6/CMIP/MOHC/UKESM1-0-LL/historical/r1i1p1f2/Amon/tas/gn/latest/tas_Amon_UKESM1-0-LL_historical_r1i1p1f2_gn_195001-201412.nc -> ../files/d20190406/tas_Amon_UKESM1-0-LL_historical_r1i1p1f2_gn_195001-201412.nc
valeriupredoi commented 5 years ago

and same for MIROC:

In [6]: c = iris.load_cube('/badc/cmip6/data/CMIP6/CMIP/MIROC/MIROC6/historical/r1i1p1f1/Amon/tas/gn/latest/tas_Amon_MIROC6_histori
   ...: cal_r1i1p1f1_gn_195001-201412.nc')                                                                                         
/home/users/valeriu/anaconda3Feb19/lib/python3.7/site-packages/iris/fileformats/cf.py:798: UserWarning: Missing CF-netCDF measure variable 'areacella', referenced by netCDF variable 'tas'
  warnings.warn(message % (variable_name, nc_var_name))

In [7]: c.coord('longitude').var_name                                                                                              
Out[7]: 'lon'
ledm commented 5 years ago

Hi V,

it was the ocean temperature tos fields, historical, r1i1p1f1, monthly, latest.

ie:

/badc/cmip6/data/CMIP6/CMIP/MOHC/UKESM1-0-LL/historical/r1i1p1f2/Omon/tos/gn/latest/tos_Omon_UKESM1-0-LL_historical_r1i1p1f2_gn_185001-194912.nc

valeriupredoi commented 5 years ago

cheers @ledm I can see that behavior looking at the file you pointed me to. What still baffles me is why, if both creation processes used the same CMOR tables, the coords diffe. They indeed used the same specs as per the files metadata:

     Attributes:
          Conventions: CF-1.7 CMIP-6.2
          activity_id: CMIP
          branch_method: standard
          branch_time_in_child: 0.0
          branch_time_in_parent: 144000.0
          cmor_version: 3.4.0
          comment: Temperature of upper boundary of the liquid ocean, including temperatures...
          creation_date: 2019-04-05T16:11:26Z
          cv_version: 6.2.20.1
          data_specs_version: 01.00.29

Anyhoo, it's Friday, so won't beat it too much :beer:

npgillett commented 4 years ago

@ledm Is your fix for the BCC models now included in the standard release of esmvalcore? I installed it from conda and updated last week, but when I try to read in tos from BCC-CSM2-MR, I get:

esmvalcore.cmor.check.CMORCheckError: There were errors in variable tos:
tos: does not match coordinate rank
in cube:
sea_surface_temperature / (degC)    (time: 1980; latitude: 232; longitude: 360)
     Dimension coordinates:
          time                           x               -               -
          latitude                       -               x               -
          longitude                      -               -               x
     Auxiliary coordinates:
          latitude                       -               x               x
          longitude                      -               x               x

Looking at the thread above, it looks like you fixed this on 15th April last year. Thanks for any help anyone can give on this. My log file is attached for reference. main_log_debug.txt

schlunma commented 4 years ago

@npgillett a fix for this will be available soon, see #560

bouweandela commented 4 years ago

This is a very old issue that deals with many different topics. @ledm If any of this is still relevant, can you please open a new issue? Preferably using the new template for data issues?