generating/checking climatological coordinate bounds

[taylor13]

The output requirements for CMIP6 ( https://goo.gl/neswPr ) provide guidance on how to treat the bounds on coordinates. It doesn't suggest that the bounds be treated differently if they represent a climatology (rather than a normal time axis) except the name of the bounds variable should be given by the CF "climatology" attribute rather than the CF "bounds" attribute. The guidance states:

Usually, the bounds name should be the same as the coordinate name, but 
with the suffix “_bnds” appended.

The only CF requirement concerning climatological bounds is that the variable name should be recorded by the "climatology" attribute.

Given this, should PrePARE skip checking what the name of the climatological bounds variable is, and we can let the CF checker determine if the file meets CF requirements?

Concerning CMOR3, it would be more consistent with the output requirements document if the name of climatological bounds were the same as the name of the regular bounds, so "time_bnds". Should we make that change?

[dnadeau4]

@taylor13 we now read the bounds from the bounds attribute in prepare, if the bounds attribute is not there, we try to get time_bnds.

[taylor13]

If the bounds attribute is not there, PrePARE should raise an error, shouldn't it?

[dnadeau4]

No it won't time_bnds is not an attribute, but a variable.

The bounds are only used by PrePARE to recreated the filename timestamp when we have a climatology dataset. In this case, the filename will be WRONG if bounds is not found.

If bounds are not found, start time bounds and end time bounds are set to 0.

[taylor13]

climatology bounds variable name is stored in attribute "climatology" not "bounds"

[taylor13]

Has this problem been addressed yet?

[taylor13]

@mauzey1 Could you generate the output files from test_non_monotonic_climo_bonds_ok.py and climatology_test_table_A, and one of the following: climatology_test.c climatology_test_code.f90. I want to check the climatology bounds produced by these tests. thanks.

[mauzey1]

@taylor13 test_non_monotonic_climo_bonds_ok_output.zip climatology_test_code_output.zip

climatology_test.c is using an outdated version of CMOR and uses CMIP5 tables. https://github.com/PCMDI/cmor/blob/master/Test/climatology_test.c What should we do with it?

[taylor13]

@mauzey1 Looks pretty good. It would be good to clean up test_non_monotonic_climo_bounds as follows: Set time coordinate values to: 18015, 18045, 18075, ... 18,345 Set clim. bounds to: (0, 36030), (30, 36060), (60, 36090), ... (330, 36360)

Also, and perhaps more importantly, I noticed that the longitude bounds values were 91, 89. The longitude dimension is supposed to be stored in increasing order as specified in the coordinates table "stored_direction": "increasing",. This means the bounds should also appear in increasing order (89, 91). Of course this is an unrealistic case since there is only one longitude value, so maybe CMOR normally handles things correctly. Does one of the test cases check whether CMOR is doing this correctly? If not, could you check what happens if in test_non_monotonic_climo_bonds you change longitude to length 2 and pass CMOR:

lon = 90, 88 bounds = (91, 89), (89, 87).

And for a second case: lon = 90, 88 bounds = (89, 91), (87, 89)

In both cases, you can set the co2 variable at lon = 90 to the value in the current file, and at lon = 88 to the negative of the value at lon=90.

thanks.

[taylor13]

@doutriaux1 @dnadeau4 Do you see any reason to keep the climatology_test.c code referenced above (https://github.com/PCMDI/cmor/issues/328#issuecomment-463832173)?

[taylor13]

When you have a moment, it would be nice to perform the test at https://github.com/PCMDI/cmor/issues/328#issuecomment-463849591 . It seems we haven't checked whether CMOR3 gets this right.

[mauzey1]

@taylor13

I tried switching the order of the bounds for the single longitude value case and the results were the same. However, the case with lon = 90, 88 only worked with bounds = (91, 89), (89, 87). When bounds = (89, 91), (87, 89) it causes an error.

C Traceback:
In function: cmor_check_monotonic
! called from: cmor_treat_axis_values
! called from: cmor_axis
! 

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Warning: Axis: 'longitude' (table: Amon), your bounds direction seems
! to be decreasing, but within cell 0 they are stored
! increasingly: you have [89.000000, 91.000000], but the next set
! is: [87.000000, 89.000000]
!
!!!!!!!!!!!!!!!!!!!!!!!!!

C Traceback:
! In function: cmor_check_values_inside_bounds
! called from: cmor_axis
! 

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: axis longitude has values not within bounds at indice:
! 0: 88.000000 not within: 91.000000, 89.000000
!
!!!!!!!!!!!!!!!!!!!!!!!!!

C Traceback:
! In function: cmor_check_values_inside_bounds
! called from: cmor_axis
! 

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: axis longitude has values not within bounds at indice:
! 1: 90.000000 not within: 89.000000, 87.000000
!
!!!!!!!!!!!!!!!!!!!!!!!!!

Traceback (most recent call last):
  File "Test/test_non_monotonic_climo_bounds.py", line 78, in <module>
    path_test()
  File "Test/test_non_monotonic_climo_bounds.py", line 62, in path_test
    axis_id = cmor.axis(**axis)
  File "/home/mauzey1/anaconda3/envs/cmor_dev/lib/python3.7/site-packages/CMOR-3.5.0-py3.7-linux-x86_64.egg/cmor/pywrapper.py", line 462, in axis
    str.encode(type), cell_bounds, cbnds, interval)
_cmor.CMORError: Problem with 'cmor.axis'. Please check the logfile (if defined).
[mauzey1@mauzey1lnx cmor]$ 

I have also updated the test and added it to the CI testing suite.

[mauzey1]

@taylor13 Here are the results from the test in the PR above.

co2_Amon_PCMDI-test-1-0_piControl-withism_r3i1p1f1_gn_200001-210012-clim.nc.zip

netcdf co2_Amon_PCMDI-test-1-0_piControl-withism_r3i1p1f1_gn_200001-210012-clim {
dimensions:
        time = UNLIMITED ; // (12 currently)
        plev = 19 ;
        lat = 1 ;
        lon = 2 ;
        bnds = 2 ;
variables:
        double time(time) ;
                time:climatology = "climatology_bnds" ;
                time:units = "days since 2000-01-01 00:00:00" ;
                time:calendar = "360_day" ;
                time:axis = "T" ;
                time:long_name = "time" ;
                time:standard_name = "time" ;
        double climatology_bnds(time, bnds) ;
        double plev(plev) ;
                plev:units = "Pa" ;
                plev:axis = "Z" ;
                plev:positive = "down" ;
                plev:long_name = "pressure" ;
                plev:standard_name = "air_pressure" ;
        double lat(lat) ;
                lat:bounds = "lat_bnds" ;
                lat:units = "degrees_north" ;
                lat:axis = "Y" ;
                lat:long_name = "Latitude" ;
                lat:standard_name = "latitude" ;
        double lat_bnds(lat, bnds) ;
        double lon(lon) ;
                lon:bounds = "lon_bnds" ;
                lon:units = "degrees_east" ;
                lon:axis = "X" ;
                lon:long_name = "Longitude" ;
                lon:standard_name = "longitude" ;
        double lon_bnds(lon, bnds) ;
        float co2(time, plev, lat, lon) ;
                co2:standard_name = "mole_fraction_of_carbon_dioxide_in_air" ;
                co2:long_name = "Mole Fraction of CO2" ;
                co2:comment = "Mole fraction is used in the construction mole_fraction_of_X_in_Y, where X is a material constituent of Y." ;
                co2:units = "mol mol-1" ;
                co2:original_units = "1.e-6" ;
                co2:history = "2019-08-22T23:01:50Z altered by CMOR: Converted units from \'1.e-6\' to \'mol mol-1\'. 2019-08-22T23:01:50Z altered by CMOR: Converted type from \'l\' to \'f\'. 2019-08-22T23:01:50Z altered by CMOR: Inverted axis: lon." ;
                co2:cell_methods = "area: mean time: mean within years time: mean over years" ;
                co2:cell_measures = "area: areacella" ;
                co2:missing_value = 1.e+20f ;
                co2:_FillValue = 1.e+20f ;

// global attributes:
                :Conventions = "CF-1.7 CMIP-6.2" ;
                :activity_id = "ISMIP6" ;
                :branch_method = "no parent" ;
                :branch_time_in_child = 59400. ;
                :branch_time_in_parent = 0. ;
                :contact = "Python Coder (coder@a.b.c.com)" ;
                :creation_date = "2019-08-22T23:01:50Z" ;
                :data_specs_version = "01.00.31" ;
                :experiment = "preindustrial control with interactive ice sheet" ;
                :experiment_id = "piControl-withism" ;
                :external_variables = "areacella" ;
                :forcing_index = 1 ;
                :frequency = "monC" ;
                :further_info_url = "https://furtherinfo.es-doc.org/CMIP6.PCMDI.PCMDI-test-1-0.piControl-withism.none.r3i1p1f1" ;
                :grid = "native atmosphere regular grid (3x4 latxlon)" ;
                :grid_label = "gn" ;
                :history = "2019-08-22T23:01:50Z ;rewrote data to be consistent with ISMIP6 for variable co2 found in table Amon.;\n",
                        "Output from archivcl_A1.nce/giccm_03_std_2xCO2_2256." ;
                :initialization_index = 1 ;
                :institution = "Program for Climate Model Diagnosis and Intercomparison, Lawrence Livermore National Laboratory, Livermore, CA 94550, USA" ;
                :institution_id = "PCMDI" ;
                :mip_era = "CMIP6" ;
                :nominal_resolution = "10000 km" ;
                :parent_activity_id = "no parent" ;
                :parent_experiment_id = "no parent" ;
                :parent_mip_era = "no parent" ;
                :parent_source_id = "no parent" ;
                :parent_time_units = "no parent" ;
                :parent_variant_label = "no parent" ;
                :physics_index = 1 ;
                :product = "model-output" ;
                :realization_index = 3 ;
                :realm = "atmos" ;
                :references = "Model described by Koder and Tolkien (J. Geophys. Res., 2001, 576-591).  Also see http://www.GICC.su/giccm/doc/index.html.  The ssp245 simulation is described in Dorkey et al. \'(Clim. Dyn., 2003, 323-357.)\'" ;
                :run_variant = "3rd realization" ;
                :source = "PCMDI-test 1.0 (1989): \n",
                        "aerosol: none\n",
                        "atmos: Earth1.0-gettingHotter (360 x 180 longitude/latitude; 50 levels; top level 0.1 mb)\n",
                        "atmosChem: none\n",
                        "land: Earth1.0\n",
                        "landIce: none\n",
                        "ocean: BlueMarble1.0-warming (360 x 180 longitude/latitude; 50 levels; top grid cell 0-10 m)\n",
                        "ocnBgchem: none\n",
                        "seaIce: Declining1.0-warming (360 x 180 longitude/latitude)" ;
                :source_id = "PCMDI-test-1-0" ;
                :source_type = "AOGCM ISM AER" ;
                :sub_experiment = "none" ;
                :sub_experiment_id = "none" ;
                :table_id = "Amon" ;
                :table_info = "Creation Date:(24 July 2019) MD5:206ddb8ba334b371d2d0b8010adc7910" ;
                :title = "PCMDI-test-1-0 output prepared for CMIP6" ;
                :tracking_id = "hdl:21.14100/fa338cf0-f260-44c4-9f4c-102b5bb77025" ;
                :variable_id = "co2" ;
                :variant_label = "r3i1p1f1" ;
                :license = "CMIP6 model data produced by Lawrence Livermore PCMDI is licensed under a Creative Commons Attribution ShareAlike 4.0 International License (https://creativecommons.org/licenses). Consult https://pcmdi.llnl.gov/CMIP6/TermsOfUse for terms of use governing CMIP6 output, including citation requirements and proper acknowledgment. Further information about this data, including some limitations, can be found via the further_info_url (recorded as a global attribute in this file) and at https:///pcmdi.llnl.gov/. The data producers and data providers make no warranty, either express or implied, including, but not limited to, warranties of merchantability and fitness for a particular purpose. All liabilities arising from the supply of the information (including any liability arising in negligence) are excluded to the fullest extent permitted by law." ;
                :cmor_version = "3.5.0" ;
data:

 time = 18015, 18045, 18075, 18105, 18135, 18165, 18195, 18225, 18255, 18285, 
    18315, 18345 ;

 climatology_bnds =
  0, 36030,
  30, 36060,
  60, 36090,
  90, 36120,
  120, 36150,
  150, 36180,
  180, 36210,
  210, 36240,
  240, 36270,
  270, 36300,
  300, 36330,
  330, 36360 ;

 plev = 100000, 92500, 85000, 70000, 60000, 50000, 40000, 30000, 25000, 
    20000, 15000, 10000, 7000, 5000, 3000, 2000, 1000, 500, 100 ;

 lat = 0 ;

 lat_bnds =
  -1, 1 ;

 lon = 88, 90 ;

 lon_bnds =
  87, 89,
  89, 91 ;

 co2 =
  1e-06, 0,
  3e-06, 2e-06,
  5e-06, 4e-06,
  7e-06, 6e-06,
  9e-06, 8e-06,
  1.1e-05, 1e-05,
  1.3e-05, 1.2e-05,
  1.5e-05, 1.4e-05,
  1.7e-05, 1.6e-05,
  1.9e-05, 1.8e-05,
  2.1e-05, 2e-05,
  2.3e-05, 2.2e-05,
  2.5e-05, 2.4e-05,
  2.7e-05, 2.6e-05,
  2.9e-05, 2.8e-05,
  3.1e-05, 3e-05,
  3.3e-05, 3.2e-05,
  3.5e-05, 3.4e-05,
  3.7e-05, 3.6e-05,
  3.9e-05, 3.8e-05,
  4.1e-05, 4e-05,
  4.3e-05, 4.2e-05,
  4.5e-05, 4.4e-05,
  4.7e-05, 4.6e-05,
  4.9e-05, 4.8e-05,
  5.1e-05, 5e-05,
  5.3e-05, 5.2e-05,
  5.5e-05, 5.4e-05,
  5.7e-05, 5.6e-05,
  5.9e-05, 5.8e-05,
  6.1e-05, 6e-05,
  6.3e-05, 6.2e-05,
  6.5e-05, 6.4e-05,
  6.7e-05, 6.6e-05,
  6.9e-05, 6.8e-05,
  7.1e-05, 7e-05,
  7.3e-05, 7.2e-05,
  7.5e-05, 7.4e-05,
  7.7e-05, 7.6e-05,
  7.9e-05, 7.8e-05,
  8.1e-05, 8e-05,
  8.3e-05, 8.2e-05,
  8.5e-05, 8.4e-05,
  8.7e-05, 8.6e-05,
  8.9e-05, 8.8e-05,
  9.1e-05, 9e-05,
  9.3e-05, 9.2e-05,
  9.5e-05, 9.4e-05,
  9.7e-05, 9.6e-05,
  9.9e-05, 9.8e-05,
  0.000101, 0.0001,
  0.000103, 0.000102,
  0.000105, 0.000104,
  0.000107, 0.000106,
  0.000109, 0.000108,
  0.000111, 0.00011,
  0.000113, 0.000112,
  0.000115, 0.000114,
  0.000117, 0.000116,
  0.000119, 0.000118,
  0.000121, 0.00012,
  0.000123, 0.000122,
  0.000125, 0.000124,
  0.000127, 0.000126,
  0.000129, 0.000128,
  0.000131, 0.00013,
  0.000133, 0.000132,
  0.000135, 0.000134,
  0.000137, 0.000136,
  0.000139, 0.000138,
  0.000141, 0.00014,
  0.000143, 0.000142,
  0.000145, 0.000144,
  0.000147, 0.000146,
  0.000149, 0.000148,
  0.000151, 0.00015,
  0.000153, 0.000152,
  0.000155, 0.000154,
  0.000157, 0.000156,
  0.000159, 0.000158,
  0.000161, 0.00016,
  0.000163, 0.000162,
  0.000165, 0.000164,
  0.000167, 0.000166,
  0.000169, 0.000168,
  0.000171, 0.00017,
  0.000173, 0.000172,
  0.000175, 0.000174,
  0.000177, 0.000176,
  0.000179, 0.000178,
  0.000181, 0.00018,
  0.000183, 0.000182,
  0.000185, 0.000184,
  0.000187, 0.000186,
  0.000189, 0.000188,
  0.000191, 0.00019,
  0.000193, 0.000192,
  0.000195, 0.000194,
  0.000197, 0.000196,
  0.000199, 0.000198,
  0.000201, 0.0002,
  0.000203, 0.000202,
  0.000205, 0.000204,
  0.000207, 0.000206,
  0.000209, 0.000208,
  0.000211, 0.00021,
  0.000213, 0.000212,
  0.000215, 0.000214,
  0.000217, 0.000216,
  0.000219, 0.000218,
  0.000221, 0.00022,
  0.000223, 0.000222,
  0.000225, 0.000224,
  0.000227, 0.000226,
  0.000229, 0.000228,
  0.000231, 0.00023,
  0.000233, 0.000232,
  0.000235, 0.000234,
  0.000237, 0.000236,
  0.000239, 0.000238,
  0.000241, 0.00024,
  0.000243, 0.000242,
  0.000245, 0.000244,
  0.000247, 0.000246,
  0.000249, 0.000248,
  0.000251, 0.00025,
  0.000253, 0.000252,
  0.000255, 0.000254,
  0.000257, 0.000256,
  0.000259, 0.000258,
  0.000261, 0.00026,
  0.000263, 0.000262,
  0.000265, 0.000264,
  0.000267, 0.000266,
  0.000269, 0.000268,
  0.000271, 0.00027,
  0.000273, 0.000272,
  0.000275, 0.000274,
  0.000277, 0.000276,
  0.000279, 0.000278,
  0.000281, 0.00028,
  0.000283, 0.000282,
  0.000285, 0.000284,
  0.000287, 0.000286,
  0.000289, 0.000288,
  0.000291, 0.00029,
  0.000293, 0.000292,
  0.000295, 0.000294,
  0.000297, 0.000296,
  0.000299, 0.000298,
  0.000301, 0.0003,
  0.000303, 0.000302,
  0.000305, 0.000304,
  0.000307, 0.000306,
  0.000309, 0.000308,
  0.000311, 0.00031,
  0.000313, 0.000312,
  0.000315, 0.000314,
  0.000317, 0.000316,
  0.000319, 0.000318,
  0.000321, 0.00032,
  0.000323, 0.000322,
  0.000325, 0.000324,
  0.000327, 0.000326,
  0.000329, 0.000328,
  0.000331, 0.00033,
  0.000333, 0.000332,
  0.000335, 0.000334,
  0.000337, 0.000336,
  0.000339, 0.000338,
  0.000341, 0.00034,
  0.000343, 0.000342,
  0.000345, 0.000344,
  0.000347, 0.000346,
  0.000349, 0.000348,
  0.000351, 0.00035,
  0.000353, 0.000352,
  0.000355, 0.000354,
  0.000357, 0.000356,
  0.000359, 0.000358,
  0.000361, 0.00036,
  0.000363, 0.000362,
  0.000365, 0.000364,
  0.000367, 0.000366,
  0.000369, 0.000368,
  0.000371, 0.00037,
  0.000373, 0.000372,
  0.000375, 0.000374,
  0.000377, 0.000376,
  0.000379, 0.000378,
  0.000381, 0.00038,
  0.000383, 0.000382,
  0.000385, 0.000384,
  0.000387, 0.000386,
  0.000389, 0.000388,
  0.000391, 0.00039,
  0.000393, 0.000392,
  0.000395, 0.000394,
  0.000397, 0.000396,
  0.000399, 0.000398,
  0.000401, 0.0004,
  0.000403, 0.000402,
  0.000405, 0.000404,
  0.000407, 0.000406,
  0.000409, 0.000408,
  0.000411, 0.00041,
  0.000413, 0.000412,
  0.000415, 0.000414,
  0.000417, 0.000416,
  0.000419, 0.000418,
  0.000421, 0.00042,
  0.000423, 0.000422,
  0.000425, 0.000424,
  0.000427, 0.000426,
  0.000429, 0.000428,
  0.000431, 0.00043,
  0.000433, 0.000432,
  0.000435, 0.000434,
  0.000437, 0.000436,
  0.000439, 0.000438,
  0.000441, 0.00044,
  0.000443, 0.000442,
  0.000445, 0.000444,
  0.000447, 0.000446,
  0.000449, 0.000448,
  0.000451, 0.00045,
  0.000453, 0.000452,
  0.000455, 0.000454 ;
}

[taylor13]

CMOR correctly reorders longitude, longitude bounds, and the data itself, so the test has been passed.

CMOR also correctly rejected a case where the two bounds on each longitude were ordered inconsistently with the ordering of the longitudes themselves.

Before closing this, I would like to check the output of "climatology" variables (first mentioned above: https://github.com/PCMDI/cmor/issues/328#issue-308113133) ? These variables are indicated in the CMOR tables with "climatology" set to "yes" (and the time dimension specified, I think, as "time2" or "time3". I would like to confirm that in place of a "bounds" attribute attached to "time", there is instead a "climatology" attribute. One option would be to run test_non_monotonic_climo_bonds_ok.py and post here the ncdump of the output.

thanks.

[mauzey1]

@taylor13 Wasn't that already in the ncdump I posted?

        double time(time) ;
                time:climatology = "climatology_bnds" ;
                time:units = "days since 2000-01-01 00:00:00" ;
                time:calendar = "360_day" ;
                time:axis = "T" ;
                time:long_name = "time" ;
                time:standard_name = "time" ;
        double climatology_bnds(time, bnds) ;

[taylor13]

Yes indeed. Sorry about that.
Regarding https://github.com/PCMDI/cmor/issues/328#issuecomment-380602315 , when creating the file name how does PrePARE know where the bounds are stored for a climatological variable vs. a regular variable. Does it know that for a climatological variable it should access the "climatology" attribute and then read the climatology bounds variable it points to, whereas for a regular variable it should access the "bounds" attribute and then read the regular bounds variable it points to?

[mauzey1]

@taylor13 PrePARE will try to find the "climatology"attribute for the variable if it can't find the "bounds" attribute.

PrePARE also determines if a variable is a "climatology" variable by checking for the -clim in the file name. If -clim is in the file name and the table is not Amon, it will append Clim to the end of the variable name. Lines 510-512 will get the variable name without the Clim at the end but it is not used for anything past those lines.

https://github.com/PCMDI/cmor/blob/84fd5213bf9fadd93beb228d689c3e0b825e25ed/LibCV/PrePARE/PrePARE.py#L301-L303 https://github.com/PCMDI/cmor/blob/84fd5213bf9fadd93beb228d689c3e0b825e25ed/LibCV/PrePARE/PrePARE.py#L505-L525

[taylor13]

It's not entirely clear whether the above code segment is correct, and it's not known whether any of the "Clim" variables that would be checked by PrePARE have yet been written. Let's assume that the coding is correct. If a user encounters a problem, we'll look into it further.