search

EC-Earth / ece2cmor3

Post-processing and cmorization of ec-earth output
Apache License 2.0
13 stars 6 forks source link

Cmorizing error: siitdconc - SImon has 2 dimensions, but it should have 4 #281

Closed treerink closed 5 years ago

treerink commented 5 years ago

C Traceback: ! In function: cmor_variable !

!!!!!!!!!!!!!!!!!!!!!!!!! ! ! Error: You are defining variable 'siitdconc' (table SImon) with 2 dimensions, when it should have 4 ! !!!!!!!!!!!!!!!!!!!!!!!!!

Traceback (most recent call last): File "/usr/people/reerink/anaconda2/envs/ece2cmor3/bin/ece2cmor", line 11, in load_entry_point('ece2cmor3==1.0.0', 'console_scripts', 'ece2cmor')() File "/usr/people/reerink/anaconda2/envs/ece2cmor3/lib/python2.7/site-packages/ece2cmor3-1.0.0-py2.7.egg/ece2cmor3/ece2cmor.py", line 115, in main ece2cmorlib.perform_nemo_tasks(args.datadir, args.exp, startdate, length) File "/usr/people/reerink/anaconda2/envs/ece2cmor3/lib/python2.7/site-packages/ece2cmor3-1.0.0-py2.7.egg/ece2cmor3/ece2cmorlib.py", line 176, in perform_nemo_tasks nemo2cmor.execute(nemo_tasks) File "/usr/people/reerink/anaconda2/envs/ece2cmor3/lib/python2.7/site-packages/ece2cmor3-1.0.0-py2.7.egg/ece2cmor3/nemo2cmor.py", line 96, in execute execute_netcdf_task(dataset, task) File "/usr/people/reerink/anaconda2/envs/ece2cmor3/lib/python2.7/site-packages/ece2cmor3-1.0.0-py2.7.egg/ece2cmor3/nemo2cmor.py", line 135, in execute_netcdf_task varid = create_cmor_variable(task, dataset, axes) File "/usr/people/reerink/anaconda2/envs/ece2cmor3/lib/python2.7/site-packages/ece2cmor3-1.0.0-py2.7.egg/ece2cmor3/nemo2cmor.py", line 198, in create_cmor_variable original_name=str(srcvar)) File "/usr/people/reerink/anaconda2/envs/ece2cmor3/lib/python2.7/site-packages/cmor/pywrapper.py", line 546, in variable missing_value, tolerance, positive, original_name, history, comment) _cmor.CMORError: Problem with 'cmor.variable'. Please check the logfile (if defined).

goord commented 5 years ago

Thomas I can't access your system from outside KNMI, can you post the ncdump -h result of the monthly sea ice xios output?

treerink commented 5 years ago

Hi Gijs, it concerns the T_3D_ncatice grid variables. Thus something I expect this needs still to be addressed (might be the last one?).

Here is the ncdump -h: so look for siitdconc: (time_counter, ncatice, y, x) 

ncdump -h /net/bhw463/nobackup/users/reerink/cmip6-ec-earth-output/t150/001/t150_1m_19900101_19901231_lim_grid_T_3D_ncatice.nc 
netcdf t150_1m_19900101_19901231_lim_grid_T_3D_ncatice {
dimensions:
    axis_nbounds = 2 ;
    x = 362 ;
    y = 292 ;
    ncatice = 5 ;
    time_counter = UNLIMITED ; // (12 currently)
variables:
    float nav_lat(y, x) ;
        nav_lat:standard_name = "latitude" ;
        nav_lat:long_name = "Latitude" ;
        nav_lat:units = "degrees_north" ;
    float nav_lon(y, x) ;
        nav_lon:standard_name = "longitude" ;
        nav_lon:long_name = "Longitude" ;
        nav_lon:units = "degrees_east" ;
    float ncatice(ncatice) ;
        ncatice:name = "ncatice" ;
        ncatice:long_name = "Ice category" ;
        ncatice:units = "1" ;
    double time_centered(time_counter) ;
        time_centered:standard_name = "time" ;
        time_centered:long_name = "Time axis" ;
        time_centered:calendar = "gregorian" ;
        time_centered:units = "seconds since 1800-01-01 00:00:00" ;
        time_centered:time_origin = "1800-01-01 00:00:00" ;
        time_centered:bounds = "time_centered_bounds" ;
    double time_centered_bounds(time_counter, axis_nbounds) ;
    double time_counter(time_counter) ;
        time_counter:axis = "T" ;
        time_counter:standard_name = "time" ;
        time_counter:long_name = "Time axis" ;
        time_counter:calendar = "gregorian" ;
        time_counter:units = "seconds since 1800-01-01 00:00:00" ;
        time_counter:time_origin = "1800-01-01 00:00:00" ;
        time_counter:bounds = "time_counter_bounds" ;
    double time_counter_bounds(time_counter, axis_nbounds) ;
    float siitdconc(time_counter, ncatice, y, x) ;
        siitdconc:standard_name = "sea_ice_area_fraction_over_categories" ;
        siitdconc:long_name = "Sea-ice area fractions in thickness categories" ;
        siitdconc:units = "%" ;
        siitdconc:online_operation = "average" ;
        siitdconc:interval_operation = "2700 s" ;
        siitdconc:interval_write = "1 month" ;
        siitdconc:cell_methods = "time: mean (interval: 2700 s)" ;
        siitdconc:_FillValue = 1.e+20f ;
        siitdconc:missing_value = 1.e+20f ;
        siitdconc:coordinates = "time_centered ncatice nav_lat nav_lon" ;
    float siitdthick(time_counter, ncatice, y, x) ;
        siitdthick:standard_name = "sea_ice_thickness_over_categories" ;
        siitdthick:long_name = "Sea-ice thickness in thickness categories" ;
        siitdthick:units = "m" ;
        siitdthick:online_operation = "average" ;
        siitdthick:interval_operation = "2700 s" ;
        siitdthick:interval_write = "1 month" ;
        siitdthick:cell_methods = "time: mean (interval: 2700 s)" ;
        siitdthick:_FillValue = 1.e+20f ;
        siitdthick:missing_value = 1.e+20f ;
        siitdthick:coordinates = "time_centered ncatice nav_lat nav_lon" ;
    float siitdsnthick(time_counter, ncatice, y, x) ;
        siitdsnthick:standard_name = "snow_thickness_over_categories" ;
        siitdsnthick:long_name = "Snow thickness in thickness categories" ;
        siitdsnthick:units = "m" ;
        siitdsnthick:online_operation = "average" ;
        siitdsnthick:interval_operation = "2700 s" ;
        siitdsnthick:interval_write = "1 month" ;
        siitdsnthick:cell_methods = "time: mean (interval: 2700 s)" ;
        siitdsnthick:_FillValue = 1.e+20f ;
        siitdsnthick:missing_value = 1.e+20f ;
        siitdsnthick:coordinates = "time_centered ncatice nav_lat nav_lon" ;

// global attributes:
        :name = "t150_1m_19900101_19901231_lim_grid_T_3D_ncatice" ;
        :description = "Created by xios" ;
        :title = "Created by xios" ;
        :Conventions = "CF-1.6" ;
        :timeStamp = "2018-Oct-12 21:09:11 GMT" ;
        :uuid = "26f70abd-a59c-4074-8fbc-6d39d432b6f2" ;
}
goord commented 5 years ago

Hi Thomas, just added the 'iceband' variable to nemo2cmor, so you can try once more....

treerink commented 5 years ago

hi Gijs, could it be you forgot to push your commits?

cmorizing only siitdconc still gives:

INFO:ece2cmor3.nemo2cmor:Looking up variables in files...
INFO:ece2cmor3.nemo2cmor:Creating NEMO grids in CMOR...
INFO:ece2cmor3.nemo2cmor:Executing 1 NEMO tasks...
INFO:ece2cmor3.nemo2cmor:Cmorizing NEMO tasks...
INFO:ece2cmor3.nemo2cmor:Creating time axes for table SImon from data in /net/bhw463/nobackup/users/reerink/cmip6-ec-earth-output/t150/001/t150_1m_19900101_19901231_lim_grid_T_3D_ncatice.nc...
INFO:ece2cmor3.nemo2cmor:Creating depth axes for table SImon from data in /net/bhw463/nobackup/users/reerink/cmip6-ec-earth-output/t150/001/t150_1m_19900101_19901231_lim_grid_T_3D_ncatice.nc ...
INFO:ece2cmor3.nemo2cmor:Creating extra axes for table SImon from data in /net/bhw463/nobackup/users/reerink/cmip6-ec-earth-output/t150/001/t150_1m_19900101_19901231_lim_grid_T_3D_ncatice.nc ...

C Traceback:
! In function: cmor_variable
! 

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: You are defining variable 'siitdconc' (table SImon)  with 2 dimensions, when it should have 4
!
!!!!!!!!!!!!!!!!!!!!!!!!!

Traceback (most recent call last):
  File "/usr/people/reerink/anaconda2/envs/ece2cmor3/bin/ece2cmor", line 11, in <module>
    load_entry_point('ece2cmor3', 'console_scripts', 'ece2cmor')()
  File "/usr/people/reerink/cmorize/ece2cmor3/ece2cmor3/ece2cmor.py", line 115, in main
    ece2cmorlib.perform_nemo_tasks(args.datadir, args.exp, startdate, length)
  File "/usr/people/reerink/cmorize/ece2cmor3/ece2cmor3/ece2cmorlib.py", line 176, in perform_nemo_tasks
    nemo2cmor.execute(nemo_tasks)
  File "/usr/people/reerink/cmorize/ece2cmor3/ece2cmor3/nemo2cmor.py", line 96, in execute
    execute_netcdf_task(dataset, task)
  File "/usr/people/reerink/cmorize/ece2cmor3/ece2cmor3/nemo2cmor.py", line 135, in execute_netcdf_task
    varid = create_cmor_variable(task, dataset, axes)
  File "/usr/people/reerink/cmorize/ece2cmor3/ece2cmor3/nemo2cmor.py", line 198, in create_cmor_variable
    original_name=str(srcvar))
  File "/usr/people/reerink/anaconda2/envs/ece2cmor3/lib/python2.7/site-packages/cmor/pywrapper.py", line 546, in variable
    missing_value, tolerance, positive, original_name, history, comment)
_cmor.CMORError: Problem with 'cmor.variable'. Please check the logfile (if defined).
treerink commented 5 years ago

Ah you pushed it in the ifsyearly branch. Intended for the master right?

goord commented 5 years ago

Shoot

goord commented 5 years ago

You can cherry-pick if you need it fast...

treerink commented 5 years ago

Ok did the cherry-pick, see #283.

treerink commented 5 years ago

One step further, but a next error about missing bounds:

Cmorizing only siitdconc still gives:

INFO:ece2cmor3.nemo2cmor:Cmorizing NEMO tasks...
INFO:ece2cmor3.nemo2cmor:Creating time axes for table SImon from data in /net/bhw463/nobackup/users/reerink/cmip6-ec-earth-output/t150/001/t150_1m_19900101_19901231_lim_grid_T_3D_ncatice.nc...
INFO:ece2cmor3.nemo2cmor:Creating depth axes for table SImon from data in /net/bhw463/nobackup/users/reerink/cmip6-ec-earth-output/t150/001/t150_1m_19900101_19901231_lim_grid_T_3D_ncatice.nc ...
INFO:ece2cmor3.nemo2cmor:Creating extra axes for table SImon from data in /net/bhw463/nobackup/users/reerink/cmip6-ec-earth-output/t150/001/t150_1m_19900101_19901231_lim_grid_T_3D_ncatice.nc ...

C Traceback:
! In function: cmor_axis
! 

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: axis: iceband (table: SImon) must have bounds, you did not pass any when creating it via cmor_axis
!
!!!!!!!!!!!!!!!!!!!!!!!!!

C Traceback:
! In function: cmor_treat_axis_values
! called from: cmor_axis
! 

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: axis 4 (iceband, table: SImon): cmor and user units are
! incompatible: m and 1
!
!!!!!!!!!!!!!!!!!!!!!!!!!

Traceback (most recent call last):
  File "/usr/people/reerink/anaconda2/envs/ece2cmor3/bin/ece2cmor", line 11, in <module>
    load_entry_point('ece2cmor3', 'console_scripts', 'ece2cmor')()
  File "/usr/people/reerink/cmorize/ece2cmor3/ece2cmor3/ece2cmor.py", line 115, in main
    ece2cmorlib.perform_nemo_tasks(args.datadir, args.exp, startdate, length)
  File "/usr/people/reerink/cmorize/ece2cmor3/ece2cmor3/ece2cmorlib.py", line 176, in perform_nemo_tasks
    nemo2cmor.execute(nemo_tasks)
  File "/usr/people/reerink/cmorize/ece2cmor3/ece2cmor3/nemo2cmor.py", line 95, in execute
    create_type_axes(dataset, task_list, table)
  File "/usr/people/reerink/cmorize/ece2cmor3/ece2cmor3/nemo2cmor.py", line 294, in create_type_axes
    axis_id = cmor.axis(table_entry=dim, coord_vals=axis_values, units=axis_unit)
  File "/usr/people/reerink/anaconda2/envs/ece2cmor3/lib/python2.7/site-packages/cmor/pywrapper.py", line 454, in axis
    type, cell_bounds, cbnds, interval)
_cmor.CMORError: Problem with 'cmor.axis'. Please check the logfile (if defined).
goord commented 5 years ago

Hmm what kind of bounds do they expect here? I mean, these icebands correspond to ice categories in our output...

treerink commented 5 years ago

1.2 MIP Variable: [siitdconc] Sea-ice area percentages in thickness categories:

description[i]: Actual (floe) thickness of sea ice in each category (NOT volume divided by grid area), (vector with one entry for each thickness category starting from the thinnest category, netcdf file should use thickness bounds of categories as third coordinate axis)

description[i]: "Thickness" means the vertical extent of a layer. "Sea ice" means all ice floating in the sea which has formed from freezing sea water, rather than by other processes such as calving of land ice to form icebergs.

goord commented 5 years ago

Not sure whether our ice categories are actually thickness categories... if they are, what are the corresponding thicknesses? I find all this very confusing...

goord commented 5 years ago

Looks like we do have thickness categories, but what are their values?

klauswyser commented 5 years ago

Maybe it would be good idea to get the advice from the sea-ice experts in the EC-Earth consortium, e.g. François (francois.massonnet@uclouvain.be). Not sure if he is following the ece2cmor3 development on GitHub, maybe you have to contact him and make him aware of the issue.

fmassonn commented 5 years ago

Hello my friends,

I have tagged myself in the issue to keep following it.

The siitdconc variable has lower and upper bounds of 0.0 and 1.0, respectively. This is because that variable is the fractional area of the grid cell occupied by ice in each category. It can be that the first category takes it all (1.0) and the other four take nothing (0.0), it can be an equal split (0.2 each), etc.

By contrast, the siitdthic variable (not sure about its name) has lower and upper bounds that do depend on category index.

Hope this helps, I'll be in Lisbonne on Monday in any case.

François

goord commented 5 years ago

Hi @fmassonn can you explain to us the ice categories? Are they like soil layers with a given thickness, because that is what I conclude from the data request's 'iceband' dimension...

fmassonn commented 5 years ago

Hi Gijs,

Let me try to summarize in a few lines what is certainly the most important concept of sea ice thermodynamics :-)

Sea ice thickness exhibits pronounced spatial variability: in a given region of, say, 100 square meters, you can find very thin ice that has been produced after re-freezing of a lead; medium ice (~2m) that has grown thermodynamically over the past days; and thick ice that resulted from dynamic rafting or ridging between different ice floes.

Accounting for this variability is key because many sea ice processes and fluxes depend non linearly on thickness. As an example: vertical heat conduction is inversely proportional to ice thickness. If a region is an even mixture of 1-m thick and 3-m thick ice, you cannot summarize this region as if it had a uniform 2-m thick ice cover, because heat conduction flux through a 2-m thick ice floe is not the simple average of fluxes through a 1-m and 3-m thick ice floes.

Obviously, large-scale sea ice models can't represent the high spatial variability of thickness given their nominal resolution of a few dozens of km.

To circumvent that, sea ice modelers invented the concept of "ice thickness distribution" (ITD). The ITD is simply the probability density function (PDF) of sea ice thickness in a given control region. In a model, it is the PDF of sea ice thickness in a given grid cell.

To illustrate this, look at this figure from my PhD thesis

image

This is a hypothetical profile of ice in a linear control region of 1 km. Thickness varies from a few centimeters to up to 4 meters. Instead of trying to represent this complex shape, the thickness space is discretized into five classes, or "categories": 0-1 m, 1-2 m, ..., 4-5 m. The ITD is the frequency of occurrence of each type of ice, that is, the area covered by each thickness range. In our example, ice in the 1-2 m range is the most abundant but instances of very thick ice can be found.

In the sea ice model used in EC-Earth, LIM, sea ice thickness is discretized in five categories as well. Unlike our example, the LIM categories are not evenly spaced but this is a detail. For each category, we have several variables:

  1. Sea ice concentration (units: fractional; ranging between 0 and 1). The sum of concentrations over the five categories must equal the grid cell ice concentration.

  2. Sea ice volume (units: m; ranging between the lower and upper bounds of the category). The volume is defined as the actual sea ice volume in the category (in m³) per unit grid cell area (in m²), thus in m. So, it is also equal to the thickness that would be obtained if ice in that category was spread evenly over the whole grid cell.

  3. There are many other state variables for each category: snow thickness, ice enthalpy, salt content, etc. No need to go in details, but you get the idea: for each type of ice, we have a description of its thermodynamic state.

So, in summary, ice thickness categories are not layers. The LIM also had layers (several for each category) to better resolve vertical profiles of temperature and salinity.

I hope this helps!

François

goord commented 5 years ago

Hi Francois, much thanks for your explanation, this carifies a lot. So you have discretized many PDF's in bins representing sea ice thicknesses 'bands' and I believe it is the bounds of these categories that is requested by CMOR. In your example above you have 5 ice bands with bounds [0-1,1-2,etc.], what is requested from us are the bounds used by LIM for defining their categories.

fmassonn commented 5 years ago

Hi Gijs,

The bounds can be found using this formula (from the LIM code):

image

where N is the number of categories (set to 5), \bar{h} is a prescribed typical mean thickness (set to 2.0 m), alpha = 0.05.

This gives the following boundaries:

>>> h = 2.0; N = 5; alpha = 0.05; [((N * (3 * h + 1) ** alpha) / ((N - i) * (3.0 * h + 1) ** alpha  + i)) ** (1.0 / alpha) - 1 for i in range(0, N + 1)]
[0.0, 0.4540167424155517, 1.1293138230836322, 2.1414589889479725, 3.670596409915544, 5.999999999999992]

Note that the upper bound of the last category is automatically overriden to 99.0 m, to allow arbitrarily thick ice to exist in that last category. So the categories limits are:

1 --> 0.0, 0.4540167424155517,
2 -->  0.4540167424155517, 1.1293138230836322
3 --> 1.1293138230836322, 2.1414589889479725
4 --> 2.1414589889479725, 3.670596409915544
5 --> 3.670596409915544, 99.0

The siitdthick variable for category i should be within the bounds for line i in the previous table.

François

goord commented 5 years ago

Thanks @fmassonn that's the info I was looking for, I will hard-code those numbers in ece2cmor3.

goord commented 5 years ago

@treerink can you have another go at your dataset with rev. 9c200861a09e3c516459823dcee8794520b0bd54

treerink commented 5 years ago

Hi Gijs,

Getting closer I think, but still a unit error (in https://github.com/EC-Earth/ece2cmor3/issues/281#issuecomment-429904654 I showed already the ncdump -h):

INFO:ece2cmor3.nemo2cmor:Cmorizing NEMO tasks...
INFO:ece2cmor3.nemo2cmor:Creating time axes for table SImon from data in /net/bhw463/nobackup/users/reerink/cmip6-ec-earth-output/t151/001/t151_1m_19900101_19901231_lim_grid_T_3D_ncatice.nc...
INFO:ece2cmor3.nemo2cmor:Creating depth axes for table SImon from data in /net/bhw463/nobackup/users/reerink/cmip6-ec-earth-output/t151/001/t151_1m_19900101_19901231_lim_grid_T_3D_ncatice.nc ...
INFO:ece2cmor3.nemo2cmor:Creating extra axes for table SImon from data in /net/bhw463/nobackup/users/reerink/cmip6-ec-earth-output/t151/001/t151_1m_19900101_19901231_lim_grid_T_3D_ncatice.nc ...

C Traceback:
! In function: cmor_treat_axis_values
! called from: cmor_axis
! 

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: axis 4 (iceband, table: SImon): cmor and user units are
! incompatible: m and 1
!
!!!!!!!!!!!!!!!!!!!!!!!!!

C Traceback:
! In function: cmor_treat_axis_values
! called from: cmor_axis
! 

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: axis 4 (iceband, table: SImon): cmor and user units are
! incompatible: m and 1
!
!!!!!!!!!!!!!!!!!!!!!!!!!

Traceback (most recent call last):
  File "/usr/people/reerink/anaconda2/envs/ece2cmor3/bin/ece2cmor", line 11, in <module>
    load_entry_point('ece2cmor3', 'console_scripts', 'ece2cmor')()
  File "/usr/people/reerink/cmorize/ece2cmor3/ece2cmor3/ece2cmor.py", line 112, in main
    ece2cmorlib.perform_nemo_tasks(args.datadir, args.exp, refdate)
  File "/usr/people/reerink/cmorize/ece2cmor3/ece2cmor3/ece2cmorlib.py", line 177, in perform_nemo_tasks
    nemo2cmor.execute(nemo_tasks)
  File "/usr/people/reerink/cmorize/ece2cmor3/ece2cmor3/nemo2cmor.py", line 99, in execute
    create_type_axes(dataset, task_list, table)
  File "/usr/people/reerink/cmorize/ece2cmor3/ece2cmor3/nemo2cmor.py", line 325, in create_type_axes
    axis_id = cmor.axis(table_entry=dim, coord_vals=axis_values, units=axis_unit, cell_bounds=bnds)
  File "/usr/people/reerink/anaconda2/envs/ece2cmor3/lib/python2.7/site-packages/cmor/pywrapper.py", line 454, in axis
    type, cell_bounds, cbnds, interval)
_cmor.CMORError: Problem with 'cmor.axis'. Please check the logfile (if defined).
fmassonn commented 5 years ago

Hey,

Since it seems to be a problem of units, just to clarify: siitdconc is expressed in units "1" (fractional) siitdthic is expressed in units "m"

François

goord commented 5 years ago

Hi, I added the units for the 'iceband' axis in cmor. I also added the 'coordinate_values' for the icebands axis to the midpoints of the bounds described above. These 'values' have to be given to the cmor library for technical reasons, physically speaking they are irrelevant.

treerink commented 5 years ago

Issue solved! Closing.