Closed valeriupredoi closed 3 years ago
@valeriupredoi Ohh very interesting, I'll take a peek for you...
@valeriupredoi Okay, so this took a wee bit of digging, but the mystery is thankfully solved :detective:
Firstly, I can recreate your issue in iris
3.0.0, and I can also confirm that iris
2.4.0 does indeed realise derived coordinates for your example dataset :+1:
This change in behaviour between iris
3.0.0 and 2.4.0 directly relates to PR #3795, where the units
of an AuxCood
or DimCoord
that is not explicitly specified defaults to unknown
rather then 1
(dimensionless).
We thought about this change in behaviour long and hard, as the treatment of default units
was inconsistent across iris
for different objects, and the CF Metadata standard did not add any clarity (no surprises there, indeed it was inconsistent), but in the end we unified the behaviour for consistency sake. That is, erring on the side that it correct and safer to assign units
as unknown
in the case when they are not specified, whereas it is potentially incorrect and not safe to assume default dimensionless units instead.
This change in behaviour is mentioned in the iris
3.0.0 release notes in the Incompatible Changes section, fifth bullet point.
Where this is stinging you is for the missing units
for the sigma
variable associated with the atmosphere_hybrid_sigma_pressure_coordinate
i.e., given formula_terms = "ap: ap_bnds b: b_bnds ps: ps"
, sigma
in this case is the NetCDF b
variable, and this is defined within the example file simply as double b(lev) ;
As the b
variable has no units
associated with it iris
3.0.0 assigns the default units
of unknown
(which they are). However, during the creation of the auxiliary factory that creates the derived coordinate it enforces the contract that the sigma
variable must be dimensionless
, which in this case it's not, as it's unknown
. Note that, enforcing that sigma
is dimensionless has not changed from 2.4.0 to 3.0.0.
This failure for sigma
units
results in iris
raising a warning
and not creating the associated HybridPressureFactory
auxiliary factory i.e., during loading you see the following warning:
UserWarning: Invalid units: sigma must be dimensionless.
Also, note the example given in the CF Metadata Conventions, where their sigma
variable has explicit units = "1"
.
So in order to fix this, my advise is to add the correct units
to your sigma
variable in the file. In addition to this, you could also add the missing bounds
attribute for the sigma
(b
) and delta
(ap
) variables e.g.,
netcdf common_cl_ap {
dimensions:
time = 1 ;
lev = 2 ;
lat = 3 ;
lon = 4 ;
bnds = 2 ;
variables:
double time(time) ;
time:standard_name = "time" ;
time:units = "days since 6543-2-1" ;
double lev(lev) ;
lev:bounds = "lev_bnds" ;
lev:standard_name = "atmosphere_hybrid_sigma_pressure_coordinate" ;
lev:units = "1" ;
lev:formula_terms = "ap: ap b: b ps: ps" ;
double lev_bnds(lev, bnds) ;
lev_bnds:standard_name = "atmosphere_hybrid_sigma_pressure_coordinate" ;
lev_bnds:units = "1" ;
lev_bnds:formula_terms = "ap: ap_bnds b: b_bnds ps: ps" ;
double lat(lat) ;
lat:standard_name = "latitude" ;
lat:units = "degrees_north" ;
double lon(lon) ;
lon:standard_name = "longitude" ;
lon:units = "degrees_east" ;
double b(lev) ;
b:units = "1" ; <== add "units" for sigma
b:bounds = "b_bnds" ; <== add "bounds" for sigma
double b_bnds(lev, bnds) ;
double ps(time, lat, lon) ;
ps:standard_name = "surface_air_pressure" ;
ps:units = "Pa" ;
ps:additional_attribute = "xyz" ;
float cl(time, lev, lat, lon) ;
cl:standard_name = "cloud_area_fraction_in_atmosphere_layer" ;
cl:units = "%" ;
double ap(lev) ;
ap:units = "Pa" ;
ap:bounds = "ap_bnds" ; <== add "bounds" for delta
double ap_bnds(lev, bnds) ;
}
Hope this helps :smile:
The alternative is that we relax the contract in the auxiliary factories to accept coordinates that are dimensionless or unknown. However that could cause other undesirable side effects or complications e.g., The computation of the derived coordinate would break due to the use of sigma
with unknown
units (I think) or at least you'd end up with a derived coordinate also with units
of unknown
.
However, iris
could override the unknown
units to be dimensionless. This may affect load/save round-tripping... that would need to be investigated. Regardless, this would be an iris
3.0.1 release
@valeriupredoi Done a quick hack to see, if in principle, this promotion of units
of unknown
to dimensionless (1
) within auxiliary factories works...
Python 3.7.8 | packaged by conda-forge | (default, Nov 17 2020, 23:45:15)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.12.0 -- An enhanced Interactive Python. Type '?' for help.
PyDev console: using IPython 7.12.0
Python 3.7.8 | packaged by conda-forge | (default, Nov 17 2020, 23:45:15)
[GCC 7.5.0] on linux
>>> import iris
>>> iris.__version__
'3.0.0'
>>> fname = "/downloads/common_cl_ap.nc"
>>> cubes = iris.load(fname)
/net/home/h05/itwl/projects/git/iris/lib/iris/fileformats/cf.py:1207: UserWarning: Ignoring formula terms variable 'ps' referenced by data variable 'ap_bnds' via variable 'lev': Dimensions ('time', 'lat', 'lon') do not span ('lev', 'bnds')
warnings.warn(msg)
/net/home/h05/itwl/projects/git/iris/lib/iris/fileformats/cf.py:1207: UserWarning: Ignoring formula terms variable 'ps' referenced by data variable 'b_bnds' via variable 'lev': Dimensions ('time', 'lat', 'lon') do not span ('lev', 'bnds')
warnings.warn(msg)
/net/home/h05/itwl/projects/git/iris/lib/iris/fileformats/netcdf.py:685: UserWarning: Unable to find coordinate for variable 'ps'
"{!r}".format(name)
/net/home/h05/itwl/projects/git/iris/lib/iris/fileformats/netcdf.py:685: UserWarning: Unable to find coordinate for variable 'ps'
"{!r}".format(name)
>>> print(cubes)
0: ap_bnds / (unknown) (atmosphere_hybrid_sigma_pressure_coordinate: 2; -- : 2)
1: b_bnds / (unknown) (atmosphere_hybrid_sigma_pressure_coordinate: 2; -- : 2)
2: cloud_area_fraction_in_atmosphere_layer / (%) (time: 1; atmosphere_hybrid_sigma_pressure_coordinate: 2; latitude: 3; longitude: 4)
3: surface_air_pressure / (Pa) (time: 1; latitude: 3; longitude: 4)
print(cubes[2])
cloud_area_fraction_in_atmosphere_layer (time: 1; atmosphere_hybrid_sigma_pressure_coordinate: 2; latitude: 3; longitude: 4)
Dimension coordinates:
time x - - -
atmosphere_hybrid_sigma_pressure_coordinate - x - -
latitude - - x -
longitude - - - x
Auxiliary coordinates:
surface_air_pressure x - x x
ap - x - -
b - x - -
Derived coordinates:
air_pressure x x x x
seems possible
@bouweandela and @valeriupredoi How much of a deal breaker is this for your guys? I'm assuming an iris
3.0.1 would be preferable?
@bjlittle cheers for the super quick and detailed answers, mate! We appreciate very much your very quick response times to issues we raise :beer: About this, I am afraid it generates a chicken-and-egg problem - the chicken being faulty model data that is already on the ESGF and the egg is us needing to fix it via our CMOR fixes before we put it through the analysis. The file I linked above is a replication of such a faulty model data file that we use in a test of a fix: if we can't load the derived coordinates it means we can't fix them in ESMValCore and that means we will not be able to run ESMValTool on it, well, at least without not turning off all the CMOR checks that, in turn, will generate a whole lot of other headaches. So yeah, I think this is something we kind of really need :grin:
@valeriupredoi Okay, no worries dude. Leave it with me and I'll get back to you asap... just need to speak to the team about this :+1:
@valeriupredoi Okay, no worries dude. Leave it with me and I'll get back to you asap... just need to speak to the team about this +1
you're legend @bjlittle cheers muchly! :beer:
@valeriupredoi Okay, I'm dropping everything today to service this - I can totally appreciate the situation you're in, so I'm going to push an iris
3.0.1 release today.
Hopefully, I can turn this around quickly for you, as I know you're working to a tight deadline.
I'll ping you here @valeriupredoi with my progress on this to keep you in the loop :+1:
@bjlittle that's fantastic, sorry to have done this to you :grin: We are planning on frezing ESMValCore on February 1st, as per our release schedule, but the freeze lasts a week and in extraordinary circumstances we can still push to main during that week - so - this would be good to be in so we can push iris=3
in the release, but that means it's not a race to the sea kind of thing. @jvegasbsc is the release manager this time around - heads up, Javi :+1: Cheers muchly for your efforts, Bill and team! :beer:
@bouweandela and @valeriupredoi How much of a deal breaker is this for your guys? I'm assuming an iris 3.0.1 would be preferable?
@schlunma Could you comment on this?
From a quick look at the issue I'm pretty sure that we can easily work around this issue in our fixes by setting the units
to '1'
like @bjlittle suggested (we actually do that in one example). I'll can have a detailed look at this later in the afternoon today.
By the way I really like the new behavior of iris
(using unknown
as units
if no unit is specified), since I had troubles with the old behavior before :+1:
From a quick look at the issue I'm pretty sure that we can easily work around this issue in our fixes by setting the units to '1' like @bjlittle suggested (we actually do that in one example). I'll can have a detailed look at this later in the afternoon today.
yes, but the coordinate is not loaded at load point!
The particular file here is so messed up that we have to fix it with netCDF4
anyway, so we can do it with that.
Nevertheless, even though the derived coordinate is not there we should be able to add it after loading with iris.cube.Cube.add_aux_factory
.
how about the case where the file is a model file off the ESGF and needs fixes - this very case that we test? Don't assume any level of clean data, you have seen what messed up files can still be out there on ESGF
@schlunma seems like a reasonable work around, but it's really causing you guys quite a lot of extra work to jump through hoops.
I've got a fix that I'm testing just now that would gracefully promote any formula terms variable that must be dimensionless to be dimensionless if it has units
of unknown
- it's doing this within the aux factory itself.
Also, this behavioural change applies to 6 aux factories - so for me, it makes more sense to fix this in iris
rather than force this upon all users.
I'm going to roll with this fix regardlessly, as my expectation is that this will affect many users.
Question: do you have any other ESMValTool netCDF files that also show this failure that I can test my fix with?
If this fix is good, and there are no other side-effects, then I believe that i can push an iris
3.0.1 release today and get it on conda-forge
for you guys.
I have @trexfeathers on hand as a review buddy to expedite the review and release.
@bjlittle hang on, I'm rerunning the tests, try find more files, if @schlunma doesn't beat me to it :)
@bjlittle Thanks, that sounds like a great solution!
@schlunma Cool, let's roll with that then.
If you guys find any extra "real world messed up" files that hit the same problem, then share them with me so that I can check this fix is good before I tag a release - just to give us extra confidence.
cirrus-ci is certainly happy with the fix i.e., no side-effects that our testing coverage can pick up... that's encouraging (or we have a hole in our test coverage :laughing:)
I'm just going to check our unit tests and ensure there is test coverage for at least this change... then we should be good to release.
massive :clap: @bjlittle - unfortunately both @schlunma and me couldn't find anymore of these messed files, but there is a good probability there may be out there, given that these are like ants - once you see one they're bound to be more
No worries, that's not a problem. Thanks for checking, much appreciated :+1:
This has turned out to be such a simple change, I'm pretty confident that similar files with no units
attributes on NetCDF variables that should have explicit units = 1
(dimensionless) will work (pride before a fall :face_with_head_bandage:)
Ha! Found one: /CMIP6/CMIP/BCC/BCC-ESM1/historical/r1i1p1f1/Amon/cl/gn/v20181217/cl_Amon_BCC-ESM1_historical_r1i1p1f1_gn_185001-201412.nc
netcdf cl_Amon_BCC-ESM1_historical_r1i1p1f1_gn_185001-201412 {
dimensions:
time = UNLIMITED ; // (1980 currently)
lev = 26 ;
lat = 64 ;
lon = 128 ;
bnds = 2 ;
variables:
double time(time) ;
time:bounds = "time_bnds" ;
time:units = "days since 1850-01-01" ;
time:calendar = "365_day" ;
time:axis = "T" ;
time:long_name = "time" ;
time:standard_name = "time" ;
double time_bnds(time, bnds) ;
double lev(lev) ;
lev:bounds = "lev_bnds" ;
lev:units = "1" ;
lev:axis = "Z" ;
lev:positive = "down" ;
lev:long_name = "hybrid sigma pressure coordinate" ;
lev:standard_name = "atmosphere_hybrid_sigma_pressure_coordinate" ;
lev:formula = "p = a*p0 + b*ps" ;
lev:formula_terms = "p0: p0 a: a b: b ps: ps" ;
double lev_bnds(lev, bnds) ;
lev_bnds:formula = "p = a*p0 + b*ps" ;
lev_bnds:standard_name = "atmosphere_hybrid_sigma_pressure_coordinate" ;
lev_bnds:units = "1" ;
lev_bnds:formula_terms = "p0: p0 a: a_bnds b: b_bnds ps: ps" ;
double p0 ;
p0:long_name = "vertical coordinate formula term: reference pressure" ;
p0:units = "Pa" ;
double a(lev) ;
a:long_name = "vertical coordinate formula term: a(k)" ;
double b(lev) ;
b:long_name = "vertical coordinate formula term: b(k)" ;
float ps(time, lat, lon) ;
ps:long_name = "Surface Air Pressure" ;
ps:units = "Pa" ;
double a_bnds(lev, bnds) ;
a_bnds:long_name = "vertical coordinate formula term: a(k+1/2)" ;
double b_bnds(lev, bnds) ;
b_bnds:long_name = "vertical coordinate formula term: b(k+1/2)" ;
double lat(lat) ;
lat:bounds = "lat_bnds" ;
lat:units = "degrees_north" ;
lat:axis = "Y" ;
lat:long_name = "latitude" ;
lat:standard_name = "latitude" ;
double lat_bnds(lat, bnds) ;
double lon(lon) ;
lon:bounds = "lon_bnds" ;
lon:units = "degrees_east" ;
lon:axis = "X" ;
lon:long_name = "Longitude" ;
lon:standard_name = "longitude" ;
double lon_bnds(lon, bnds) ;
float cl(time, lev, lat, lon) ;
cl:standard_name = "cloud_area_fraction_in_atmosphere_layer" ;
cl:long_name = "Cloud Area Fraction" ;
cl:comment = "Percentage cloud cover, including both large-scale and convective cloud." ;
cl:units = "%" ;
cl:original_name = "CLOUD" ;
cl:cell_methods = "area: time: mean (interval: 20 minutes)" ;
cl:cell_measures = "area: areacella" ;
cl:missing_value = 1.e+20f ;
cl:_FillValue = 1.e+20f ;
cl:history = "2018-12-17T02:20:59Z altered by CMOR: Inverted axis: lev." ;
@schlunma Awesome find :eyes: ... I'll test with that now... hang tight...
@schlunma Dah! Where does that file exist? Can you provide a link so that I can download it...
I though that it might live in https://github.com/ESMValGroup/ESMValCore/tree/master/tests/integration/cmor/_fixes/test_data
told you they're like ants :laughing: That's on the ESGF node (at DKRZ I assume, since Manu works on that one), let me try grab it off CEDA-JASMIN and put it somewhere where you can download it
Thanks V.!
I found another one with hybrid height: MOHC/UKESM1-0-LL/historical/r1i1p1f2/Amon/cl/gn/v20190406/cl_Amon_UKESM1-0-LL_historical_r1i1p1f2_gn_185001-189912.nc
.
You can check cl
of all CMIP6 models, I guess there's more of them. I have a meeting now, can help you guys later! :+1:
I've put two of those files on JASMIN at /home/users/valeriu/badData
which is open to all JASMIN users, unfortunately the local server that I use to get into JASMIN is read-only today (as it happens, when you need something done), and I can't scp
them anywhere w/o first going through it - any chance any of yous can grab them straight from my JASMIN location?
@valeriupredoi Nuts. Sorry, I don't have access to JASMIN.
Hmmm is it possible for you to create a new personal GitHub repo and upload the files into there? Then I can get them, and afterwards simply nuke that GitHub repo when we're finished...
@bjlittle You can just download the files directly from ESGF:
wget http://esgf3.dkrz.de/thredds/fileServer/cmip6/CMIP/BCC/BCC-ESM1/historical/r1i1p1f1/Amon/cl/gn/v20181217/cl_Amon_BCC-ESM1_historical_r1i1p1f1_gn_185001-201412.nc
wget http://esgf3.dkrz.de/thredds/fileServer/cmip6/CMIP/MOHC/UKESM1-0-LL/historical/r1i1p1f2/Amon/cl/gn/v20190406/cl_Amon_UKESM1-0-LL_historical_r1i1p1f2_gn_185001-189912.nc
yes, that's what I just did, but I had to chop them (in iris2.4) otherwise they'd be 2G >> 100M max upload limit :grin: Here's the first, am adding the second in a jiffy https://github.com/valeriupredoi/badData/blob/main/cl_Amon_BCC-ESM1_chopped.nc
and here's the second file (complete with gitHub barking at me it's more than 50M hehe) https://github.com/valeriupredoi/badData/blob/main/cl_Amon_UKESM1-0-LL_chopped.nc - chopping is on time (I kept only the first 10 time points, all else should be there)
or @bouweandela 's solution which is elegant, I must confess I have never downloaded from ESGF :grin:
Thanks guys, downloading now...
Blimey it's a sloooooow connection... :sleeping:
can grab me chopped files, they're much smaller :+1:
These are opendap urls, so you can also load the files directly without downloading, maybe that's faster?
import iris
cubes = iris.load('http://esgf3.dkrz.de/thredds/fileServer/cmip6/CMIP/BCC/BCC-ESM1/historical/r1i1p1f1/Amon/cl/gn/v20181217/cl_Amon_BCC-ESM1_historical_r1i1p1f1_gn_185001-201412.nc')
I've got one file, the other is almost there... so glad we did this as the file cl_Amon_BCC-ESM1_historical_r1i1p1f1_gn_185001-201412
contains the alternative formula_terms = "p0: p0 a: a b: b ps: ps"
for a atmosphere_hybrid_sigma_pressure_coordinate
, where ap
(delta
) is formed by performing p0 * a
.
In the example file a
has no units
and p0
has units
of Pa
(which does and must match ps
), however the p0 * a
results in a delta
with units
of unknown
and causes the same failure, but in a different way.
I've located the fix in iris/fileformats/netcdf.py
before it then creates the aux factory, as it's too late for the patch in the aux factory to fix this particular nuance.
Re-testing now...
Tis a beautiful thing...
>>> import iris
>>> fname = "/downloads/cl_Amon_BCC-ESM1_historical_r1i1p1f1_gn_185001-201412.nc"
>>> cubes = iris.load(fname)
/net/home/h05/itwl/projects/git/iris/lib/iris/fileformats/cf.py:862: UserWarning: Missing CF-netCDF measure variable 'areacella', referenced by netCDF variable 'cl'
message % (variable_name, nc_var_name)
/net/home/h05/itwl/projects/git/iris/lib/iris/fileformats/cf.py:1207: UserWarning: Ignoring formula terms variable 'ps' referenced by data variable 'b_bnds' via variable 'lev': Dimensions ('time', 'lat', 'lon') do not span ('lev', 'bnds')
warnings.warn(msg)
/net/home/h05/itwl/projects/git/iris/lib/iris/fileformats/cf.py:1207: UserWarning: Ignoring formula terms variable 'ps' referenced by data variable 'a_bnds' via variable 'lev': Dimensions ('time', 'lat', 'lon') do not span ('lev', 'bnds')
warnings.warn(msg)
/net/home/h05/itwl/projects/git/iris/lib/iris/fileformats/netcdf.py:685: UserWarning: Unable to find coordinate for variable 'ps'
"{!r}".format(name)
/net/home/h05/itwl/projects/git/iris/lib/iris/fileformats/netcdf.py:685: UserWarning: Unable to find coordinate for variable 'ps'
"{!r}".format(name)
>>> cubes
[<iris 'Cube' of vertical coordinate formula term: b(k+1/2) / (unknown) (atmosphere_hybrid_sigma_pressure_coordinate: 26; -- : 2)>,
<iris 'Cube' of vertical coordinate formula term: a(k+1/2) / (unknown) (atmosphere_hybrid_sigma_pressure_coordinate: 26; -- : 2)>,
<iris 'Cube' of Surface Air Pressure / (Pa) (time: 1980; latitude: 64; longitude: 128)>,
<iris 'Cube' of cloud_area_fraction_in_atmosphere_layer / (%) (time: 1980; atmosphere_hybrid_sigma_pressure_coordinate: 26; latitude: 64; longitude: 128)>]
>>> print(cubes[3])
cloud_area_fraction_in_atmosphere_layer (time: 1980; atmosphere_hybrid_sigma_pressure_coordinate: 26; latitude: 64; longitude: 128)
Dimension coordinates:
time x - - -
atmosphere_hybrid_sigma_pressure_coordinate - x - -
latitude - - x -
longitude - - - x
Auxiliary coordinates:
Surface Air Pressure x - x x
vertical coordinate formula term: a(k) - x - -
vertical coordinate formula term: b(k) - x - -
vertical pressure - x - -
Derived coordinates:
air_pressure x x x x
Scalar coordinates:
vertical coordinate formula term: reference pressure: 100000.0 Pa
Attributes:
Conventions: CF-1.7 CMIP-6.2
activity_id: CMIP
branch_method: Standard
branch_time_in_child: 0.0
branch_time_in_parent: 2110.0
cmor_version: 3.3.2
comment: Percentage cloud cover, including both large-scale and convective clou...
contact: Dr. Tongwen Wu(twwu@cma.gov.cn)
creation_date: 2018-12-17T02:20:59Z
data_specs_version: 01.00.27
description: DECK: historical
experiment: all-forcing simulation of the recent past
experiment_id: historical
external_variables: areacella
forcing_index: 1
frequency: mon
further_info_url: https://furtherinfo.es-doc.org/CMIP6.BCC.BCC-ESM1.historical.none.r1i1...
grid: T42
grid_label: gn
history: 2018-12-17T02:20:59Z altered by CMOR: Inverted axis: lev.
initialization_index: 1
institution: Beijing Climate Center, Beijing 100081, China
institution_id: BCC
license: CMIP6 model data produced by BCC is licensed under a Creative Commons Attribution...
mip_era: CMIP6
nominal_resolution: 250 km
original_name: CLOUD
parent_activity_id: CMIP
parent_experiment_id: piControl
parent_mip_era: CMIP6
parent_source_id: BCC-ESM1
parent_time_units: days since 1850-01-01
parent_variant_label: r1i1p1f1
physics_index: 1
product: model-output
realization_index: 1
realm: atmos
references: Model described by Tongwen Wu et al. (JGR 2013; JMR 2014; submmitted to...
run_variant: forcing: greenhouse gases,aerosol emission,solar constant,volcano mass,land...
source: BCC-ESM 1 (2017): aerosol: none atmos: BCC_AGCM3_LR (T42; 128 x 64 longitude/latitude;...
source_id: BCC-ESM1
source_type: AER AOGCM CHEM
sub_experiment: none
sub_experiment_id: none
table_id: Amon
table_info: Creation Date:(30 July 2018) MD5:e53ff52009d0b97d9d867dc12b6096c7
title: BCC-ESM1 output prepared for CMIP6
tracking_id: hdl:21.14100/6f5b873f-9609-4be8-99d6-bfde65468e67
variable_id: cl
variant_label: r1i1p1f1
Cell methods:
mean: area (20 minutes), time (20 minutes)
Just checking the second file cl_Amon_UKESM1-0-LL_historical_r1i1p1f2_gn_185001-189912.nc
...
Sweet, this looks good too...
>>> fname2 = "/downloads/cl_Amon_UKESM1-0-LL_historical_r1i1p1f2_gn_185001-189912.nc"
>>> cubes = iris.load(fname)
/net/home/h05/itwl/projects/git/iris/lib/iris/fileformats/cf.py:862: UserWarning: Missing CF-netCDF measure variable 'areacella', referenced by netCDF variable 'cl'
message % (variable_name, nc_var_name)
/net/home/h05/itwl/projects/git/iris/lib/iris/fileformats/cf.py:1207: UserWarning: Ignoring formula terms variable 'ps' referenced by data variable 'b_bnds' via variable 'lev': Dimensions ('time', 'lat', 'lon') do not span ('lev', 'bnds')
warnings.warn(msg)
/net/home/h05/itwl/projects/git/iris/lib/iris/fileformats/cf.py:1207: UserWarning: Ignoring formula terms variable 'ps' referenced by data variable 'a_bnds' via variable 'lev': Dimensions ('time', 'lat', 'lon') do not span ('lev', 'bnds')
warnings.warn(msg)
/net/home/h05/itwl/projects/git/iris/lib/iris/fileformats/netcdf.py:685: UserWarning: Unable to find coordinate for variable 'ps'
"{!r}".format(name)
/net/home/h05/itwl/projects/git/iris/lib/iris/fileformats/netcdf.py:685: UserWarning: Unable to find coordinate for variable 'ps'
"{!r}".format(name)
>>> print(cubes)
0: vertical coordinate formula term: b(k+1/2) / (unknown) (atmosphere_hybrid_sigma_pressure_coordinate: 26; -- : 2)
1: vertical coordinate formula term: a(k+1/2) / (unknown) (atmosphere_hybrid_sigma_pressure_coordinate: 26; -- : 2)
2: Surface Air Pressure / (Pa) (time: 1980; latitude: 64; longitude: 128)
3: cloud_area_fraction_in_atmosphere_layer / (%) (time: 1980; atmosphere_hybrid_sigma_pressure_coordinate: 26; latitude: 64; longitude: 128)
>>> print(cubes[3])
cloud_area_fraction_in_atmosphere_layer (time: 1980; atmosphere_hybrid_sigma_pressure_coordinate: 26; latitude: 64; longitude: 128)
Dimension coordinates:
time x - - -
atmosphere_hybrid_sigma_pressure_coordinate - x - -
latitude - - x -
longitude - - - x
Auxiliary coordinates:
Surface Air Pressure x - x x
vertical coordinate formula term: a(k) - x - -
vertical coordinate formula term: b(k) - x - -
vertical pressure - x - -
Derived coordinates:
air_pressure x x x x
Scalar coordinates:
vertical coordinate formula term: reference pressure: 100000.0 Pa
Attributes:
Conventions: CF-1.7 CMIP-6.2
activity_id: CMIP
branch_method: Standard
branch_time_in_child: 0.0
branch_time_in_parent: 2110.0
cmor_version: 3.3.2
comment: Percentage cloud cover, including both large-scale and convective clou...
contact: Dr. Tongwen Wu(twwu@cma.gov.cn)
creation_date: 2018-12-17T02:20:59Z
data_specs_version: 01.00.27
description: DECK: historical
experiment: all-forcing simulation of the recent past
experiment_id: historical
external_variables: areacella
forcing_index: 1
frequency: mon
further_info_url: https://furtherinfo.es-doc.org/CMIP6.BCC.BCC-ESM1.historical.none.r1i1...
grid: T42
grid_label: gn
history: 2018-12-17T02:20:59Z altered by CMOR: Inverted axis: lev.
initialization_index: 1
institution: Beijing Climate Center, Beijing 100081, China
institution_id: BCC
license: CMIP6 model data produced by BCC is licensed under a Creative Commons Attribution...
mip_era: CMIP6
nominal_resolution: 250 km
original_name: CLOUD
parent_activity_id: CMIP
parent_experiment_id: piControl
parent_mip_era: CMIP6
parent_source_id: BCC-ESM1
parent_time_units: days since 1850-01-01
parent_variant_label: r1i1p1f1
physics_index: 1
product: model-output
realization_index: 1
realm: atmos
references: Model described by Tongwen Wu et al. (JGR 2013; JMR 2014; submmitted to...
run_variant: forcing: greenhouse gases,aerosol emission,solar constant,volcano mass,land...
source: BCC-ESM 1 (2017): aerosol: none atmos: BCC_AGCM3_LR (T42; 128 x 64 longitude/latitude;...
source_id: BCC-ESM1
source_type: AER AOGCM CHEM
sub_experiment: none
sub_experiment_id: none
table_id: Amon
table_info: Creation Date:(30 July 2018) MD5:e53ff52009d0b97d9d867dc12b6096c7
title: BCC-ESM1 output prepared for CMIP6
tracking_id: hdl:21.14100/6f5b873f-9609-4be8-99d6-bfde65468e67
variable_id: cl
variant_label: r1i1p1f1
Cell methods:
mean: area (20 minutes), time (20 minutes)
Derived coordinates:
air_pressure x
yay! welcome to the absolute nightmare on ESGF street @bjlittle :rofl:
Okay, I think we're good guys :+1:
I defo need to add some unit test coverage for these changes, but that shouldn't take too long.
I've already updated the docs for the v3.0.1 patch release, bumped the version et al.
Once I nail the tests, I'll try to expedite this through review. Tag a release, and the conda-forge
bot should automagically create a PR on the feedstock. So iris
3.0.1 should be up on conda-forge
in a couple of hours from now :crossed_fingers:
If any gotchas happen between now and then I'll let you know here asap
The UKESM
model should have an altitude
derived coordinate, that's weird!
EDIT: You need to use fname2
instead of fname
in the beginning :smile:
that's terrific! Thanks very much @bjlittle - a very nice example of cross-package collaboration when you have cool devvs on both sides, remind me to buy you a :beer: when all this mess is over (the pandemic not the ESGF data issues, the latter one won't be over soon) :grin:
LOL what a muppet... hang tight...
Great spot @schlunma, thanks... here we go:
>>> fname2 = "/downloads/cl_Amon_UKESM1-0-LL_historical_r1i1p1f2_gn_185001-189912.nc"
>>> cubes = iris.load(fname2)
/net/home/h05/itwl/projects/git/iris/lib/iris/fileformats/cf.py:862: UserWarning: Missing CF-netCDF measure variable 'areacella', referenced by netCDF variable 'cl'
message % (variable_name, nc_var_name)
/net/home/h05/itwl/projects/git/iris/lib/iris/fileformats/cf.py:1207: UserWarning: Ignoring formula terms variable 'orog' referenced by data variable 'b_bnds' via variable 'lev': Dimensions ('lat', 'lon') do not span ('lev', 'bnds')
warnings.warn(msg)
/net/home/h05/itwl/projects/git/iris/lib/iris/fileformats/netcdf.py:685: UserWarning: Unable to find coordinate for variable 'orog'
"{!r}".format(name)
>>> print(cubes)
0: vertical coordinate formula term: b(k+1/2) / (unknown) (atmosphere_hybrid_height_coordinate: 85; -- : 2)
1: Surface Altitude / (m) (latitude: 144; longitude: 192)
2: cloud_area_fraction_in_atmosphere_layer / (%) (time: 600; atmosphere_hybrid_height_coordinate: 85; latitude: 144; longitude: 192)
>>> print(cubes[2])
cloud_area_fraction_in_atmosphere_layer / (%) (time: 600; atmosphere_hybrid_height_coordinate: 85; latitude: 144; longitude: 192)
Dimension coordinates:
time x - - -
atmosphere_hybrid_height_coordinate - x - -
latitude - - x -
longitude - - - x
Auxiliary coordinates:
vertical coordinate formula term: b(k) - x - -
Surface Altitude - - x x
Derived coordinates:
altitude - x x x
Attributes:
Conventions: CF-1.7 CMIP-6.2
activity_id: CMIP
branch_method: standard
branch_time_in_child: 0.0
branch_time_in_parent: 144000.0
cmor_version: 3.4.0
comment: Percentage cloud cover, including both large-scale and convective clou...
creation_date: 2019-04-05T15:57:16Z
cv_version: 6.2.20.1
data_specs_version: 01.00.29
experiment: all-forcing simulation of the recent past
experiment_id: historical
external_variables: areacella
forcing_index: 2
frequency: mon
further_info_url: https://furtherinfo.es-doc.org/CMIP6.MOHC.UKESM1-0-LL.historical.none....
grid: Native N96 grid; 192 x 144 longitude/latitude
grid_label: gn
history: 2019-04-05T15:57:16Z altered by CMOR: Converted units from '1' to '%'....
initialization_index: 1
institution: Met Office Hadley Centre, Fitzroy Road, Exeter, Devon, EX1 3PB, UK
institution_id: MOHC
license: CMIP6 model data produced by the Met Office Hadley Centre is licensed under...
mip_era: CMIP6
mo_runid: u-bc179
nominal_resolution: 250 km
original_name: mo: (stash: m01s02i261, lbproc: 128)
original_units: 1
parent_activity_id: CMIP
parent_experiment_id: piControl
parent_mip_era: CMIP6
parent_source_id: UKESM1-0-LL
parent_time_units: days since 1850-01-01-00-00-00
parent_variant_label: r1i1p1f2
physics_index: 1
product: model-output
realization_index: 1
realm: atmos
source: UKESM1.0-LL (2018):
aerosol: UKCA-GLOMAP-mode
atmos: MetUM-HadGEM3-GA7.1...
source_id: UKESM1-0-LL
source_type: AOGCM AER BGC CHEM
sub_experiment: none
sub_experiment_id: none
table_id: Amon
table_info: Creation Date:(13 December 2018) MD5:2b12b5db6db112aa8b8b0d6c1645b121
title: UKESM1-0-LL output prepared for CMIP6
tracking_id: hdl:21.14100/7507e315-b264-43ff-a7c6-479946c0a033
variable_id: cl
variant_label: r1i1p1f2
Cell methods:
mean: area, time
do you still need my chopped files on GitHub @bjlittle - or I can remove them? Might get Github to banish me for uploading junk :grin:
📰 Custom Issue
Hey guys, one more for me today, I am using this file and loading it simple:
with iris=3 I am not seeing the derived coordinates no more:
iris=2.4
:iris=3.0
:where did the Derived coordinates go?