Catching invalid specified grids

[durack1]

@dnadeau4 the following file was generated using CMOR 2.7.1:

ncdump -c /cmip5_css02/data/cmip5/output1/BNU/BNU-ESM/historical/fx/atmos/fx/r0i0p0/sftlf/1/sftlf_fx_BNU-ESM_historical_r0i0p0.nc | more                                                      
netcdf sftlf_fx_BNU-ESM_historical_r0i0p0 {                                                                                             
dimensions:                                                                                                                             
        lat = 64 ;                                                                                                                      
        lon = 128 ;                                                                                                                     
        bnds = 2 ;                                                                                                                      
variables:                                                                                                                              
        double lat(lat) ;                                                                                                               
                lat:bounds = "lat_bnds" ;                                                                                               
                lat:units = "degrees_north" ;                                                                                           
                lat:axis = "Y" ;                                                                                                        
                lat:long_name = "latitude" ;                                                                                            
                lat:standard_name = "latitude" ;                                                                                        
        double lat_bnds(lat, bnds) ;                                                                                                    
        double lon(lon) ;                                                                                                               
                lon:bounds = "lon_bnds" ;                                                                                               
                lon:units = "degrees_east" ;                                                                                            
                lon:axis = "X" ;                                                                                                        
                lon:long_name = "longitude" ;                                                                                           
                lon:standard_name = "longitude" ;                                                                                       
        double lon_bnds(lon, bnds) ;                                                                                                    
        float sftlf(lat, lon) ;                                                                                                         
                sftlf:standard_name = "land_area_fraction" ;                                                                            
                sftlf:long_name = "Land Area Fraction" ;                                                                                
                sftlf:units = "%" ;                                                                                                     
                sftlf:cell_measures = "area: areacella" ;                                                                               
                sftlf:history = "2012-11-15T09:15:43Z altered by CMOR: replaced missing value flag (1e+30) with standard missing value (
1e+20)." ;                                                                                                                              
                sftlf:missing_value = 1.e+20f ;                                                                                         
                sftlf:_FillValue = 1.e+20f ;                                                                                            
                sftlf:associated_files = "baseURL: http://cmip-pcmdi.llnl.gov/CMIP5/dataLocation gridspecFile: gridspec_atmos_fx_BNU-ESM
_historical_r0i0p0.nc areacella: areacella_fx_BNU-ESM_historical_r0i0p0.nc" ;                                                           

// global attributes:
                :institution = "GCESS,BNU,Beijing,China" ;
                :institute_id = "BNU" ;                   
                :experiment_id = "historical" ;           
                :source = "BNU-ESM" ;                     
                :model_id = "BNU-ESM" ;                   
                :forcing = "Nat,Ant" ;                    
                :parent_experiment_id = "piControl" ;     
                :parent_experiment_rip = "r1i1p1" ;       
                :branch_time = 0. ;                       
                :contact = "Ji Duoying (duoyingji@bnu.edu.cn)" ;
                :initialization_method = 0 ;
                :physics_version = 0 ;
                :tracking_id = "07552e34-e98b-4e07-80e9-c6ca97fbb1e8" ;
                :product = "output" ;
                :experiment = "historical" ;
                :frequency = "fx" ;
                :creation_date = "2012-11-15T09:15:43Z" ;
                :history = "2012-11-15T09:15:43Z CMOR rewrote data to comply with CF standards and CMIP5 requirements." ;
                :Conventions = "CF-1.4" ;
                :project_id = "CMIP5" ;
                :table_id = "Table fx (26 July 2011) 491518982c8d8b607a58ba740689ea09" ;
                :title = "BNU-ESM model output prepared for CMIP5 historical" ;
                :parent_experiment = "pre-industrial control" ;
                :modeling_realm = "atmos" ;
                :realization = 0 ;
                :cmor_version = "2.7.1" ;
data:

 lat = -87.8638000488281, -87.8638000488281, -87.8638000488281,
    -87.8638000488281, -87.8638000488281, -87.8638000488281,
    -87.8638000488281, -87.8638000488281, -87.8638000488281,
    -87.8638000488281, -87.8638000488281, -87.8638000488281,
    -87.8638000488281, -87.8638000488281, -87.8638000488281,
    -87.8638000488281, -87.8638000488281, -87.8638000488281,
    -87.8638000488281, -87.8638000488281, -87.8638000488281,
    -87.8638000488281, -87.8638000488281, -87.8638000488281,
    -87.8638000488281, -87.8638000488281, -87.8638000488281,
    -87.8638000488281, -87.8638000488281, -87.8638000488281,
    -87.8638000488281, -87.8638000488281, -87.8638000488281,
    -87.8638000488281, -87.8638000488281, -87.8638000488281,
    -87.8638000488281, -87.8638000488281, -87.8638000488281,
    -87.8638000488281, -87.8638000488281, -87.8638000488281,
    -87.8638000488281, -87.8638000488281, -87.8638000488281,
    -87.8638000488281, -87.8638000488281, -87.8638000488281,
    -87.8638000488281, -87.8638000488281, -87.8638000488281,
    -87.8638000488281, -87.8638000488281, -87.8638000488281,
    -87.8638000488281, -87.8638000488281, -87.8638000488281,
    -87.8638000488281, -87.8638000488281, -87.8638000488281,
    -87.8638000488281, -87.8638000488281, -87.8638000488281, -87.8638000488281 ;

 lon = 0, 2.8125, 5.625, 8.4375, 11.25, 14.0625, 16.875, 19.6875, 22.5,
    25.3125, 28.125, 30.9375, 33.75, 36.5625, 39.375, 42.1875, 45, 47.8125,
    50.625, 53.4375, 56.25, 59.0625, 61.875, 64.6875, 67.5, 70.3125, 73.125,
    75.9375, 78.75, 81.5625, 84.375, 87.1875, 90, 92.8125, 95.625, 98.4375,
    101.25, 104.0625, 106.875, 109.6875, 112.5, 115.3125, 118.125, 120.9375,
    123.75, 126.5625, 129.375, 132.1875, 135, 137.8125, 140.625, 143.4375,
    146.25, 149.0625, 151.875, 154.6875, 157.5, 160.3125, 163.125, 165.9375,
    168.75, 171.5625, 174.375, 177.1875, 180, 182.8125, 185.625, 188.4375,
    191.25, 194.0625, 196.875, 199.6875, 202.5, 205.3125, 208.125, 210.9375,
    213.75, 216.5625, 219.375, 222.1875, 225, 227.8125, 230.625, 233.4375,
    236.25, 239.0625, 241.875, 244.6875, 247.5, 250.3125, 253.125, 255.9375,
    258.75, 261.5625, 264.375, 267.1875, 270, 272.8125, 275.625, 278.4375,
    281.25, 284.0625, 286.875, 289.6875, 292.5, 295.3125, 298.125, 300.9375,
    303.75, 306.5625, 309.375, 312.1875, 315, 317.8125, 320.625, 323.4375,
    326.25, 329.0625, 331.875, 334.6875, 337.5, 340.3125, 343.125, 345.9375,
    348.75, 351.5625, 354.375, 357.1875 ;
}

It seems the longitude values are repeated, and do not span the -90 to 90 range that you would expect. It would be useful for CMOR to throw a warning in this case.

When plotted VCS tries pretty hard

[durack1]

@dnadeau4 and here is another issue, upside down grid (although the problem file [top image] wasn't written using CMOR): So the ncdump looks like:

$ ncdump -c /cmip5_css01/data/cmip5/output1/ICHEC/EC-EARTH/historical/fx/atmos/fx/r0i0p0/v20130211/sftlf/sftlf_atmos_fx_EC-EARTH_historical_r0i0p0.nc | more                                                                                               
netcdf sftlf_atmos_fx_EC-EARTH_historical_r0i0p0 {                                                                                      
dimensions:                                                                                                                             
        lon = 320 ;                                                                                                                     
        lat = 160 ;                                                                                                                     
variables:                                                                                                                              
        double lon(lon) ;                                                                                                               
                lon:long_name = "longitude" ;                                                                                           
                lon:units = "degrees_east" ;                                                                                            
                lon:standard_name = "longitude" ;                                                                                       
        double lat(lat) ;                                                                                                               
                lat:long_name = "latitude" ;                                                                                            
                lat:units = "degrees_north" ;                                                                                           
                lat:standard_name = "latitude" ;                                                                                        
        float sftlf(lat, lon) ;                                                                                                         
                sftlf:standard_name = "land_area_fraction" ;                                                                            
                sftlf:units = "%" ;                                                                                                     
                sftlf:long_name = "Land Area Fraction" ;                                                                                
                sftlf:grid_type = "gaussian" ;                                                                                          

// global attributes:
                :CDI = "Climate Data Interface version 1.4.5 (http://code.zmaw.de/projects/cdi)" ;
                :Conventions = "CF-1.4" ;                                                         
                :history = "Mon Feb 11 13:04:52 2013: cdo -f nc copy output_a033_172_regGG.grib stflc.nc" ;
                :source = "EC-EARTH: (2011) ; atmosphere: IFS (cy31r1, 100); ocean: NEMO (v2, ORCA1, 4x8) ; coupler: OASIS3 (v3.3) ; sea
 ice: LIM2 ; + EC-Earth developments" ;                                                                                                 
                :title = "EC-Earth model output prepared for CMIP5 historical" ;                                                        
                :modeling_realm = "atmos" ;                                                                                             
                :institution = "EC-Earth (European Earth System Model)" ;                                                               
                :institute_id = "ICHEC" ;                                                                                               
                :model_id = "EC-EARTH" ;                                                                                                
                :comment = "generated from var172 (Land Sea Mask) of IFS by CDO" ;                                                      
                :references = "Model described by Hazeleger et al. (Bull. Amer. Meteor. Soc., 2010, 91, 1357-1363). Also see http://ecea
rth.knmi.nl." ;                                                                                                                         
                :product = "output" ;                                                                                                   
                :project_id = "CMIP5" ;                                                                                                 
                :parent_experiment_id = "piControl" ;                                                                                   
                :parent_experiment_rip = "r1i1p1" ;                                                                                     
                :initialization_method = 0 ;                                                                                            
                :realization = 0 ;                                                                                                      
                :physics_version = 0 ;                                                                                                  
                :frequency = "fx" ;                                                                                                     
                :tracking_id = "d2c4549d-e09e-4af4-b25b-261f36d24780" ;                                                                 
                :experiment_id = "historical" ;                                                                                         
                :branch_time = 1850. ;                                                                                                  
                :forcing = "Nat,Ant" ;                                                                                                  
                :table_id = "Table fx (27 April 2011) becb0e555e7f575d5973a833590cdfb1" ;                                               
                :cmor_table = "fx" ;                                                                                                    
data:                                                                                                                                   

 lon = 0, 1.125, 2.25, 3.375, 4.5, 5.625, 6.75, 7.875, 9, 10.125, 11.25, 
    12.375, 13.5, 14.625, 15.75, 16.875, 18, 19.125, 20.25, 21.375, 22.5, 
    23.625, 24.75, 25.875, 27, 28.125, 29.25, 30.375, 31.5, 32.625, 33.75, 
    34.875, 36, 37.125, 38.25, 39.375, 40.5, 41.625, 42.75, 43.875, 45,    
    46.125, 47.25, 48.375, 49.5, 50.625, 51.75, 52.875, 54, 55.125, 56.25, 
    57.375, 58.5, 59.625, 60.75, 61.875, 63, 64.125, 65.25, 66.375, 67.5,  
    68.625, 69.75, 70.875, 72, 73.125, 74.25, 75.375, 76.5, 77.625, 78.75, 
    79.875, 81, 82.125, 83.25, 84.375, 85.5, 86.625, 87.75, 88.875, 90,    
    91.125, 92.25, 93.375, 94.5, 95.625, 96.75, 97.875, 99, 100.125, 101.25, 
    102.375, 103.5, 104.625, 105.75, 106.875, 108, 109.125, 110.25, 111.375, 
    112.5, 113.625, 114.75, 115.875, 117, 118.125, 119.25, 120.375, 121.5,   
    122.625, 123.75, 124.875, 126, 127.125, 128.25, 129.375, 130.5, 131.625, 
    132.75, 133.875, 135, 136.125, 137.25, 138.375, 139.5, 140.625, 141.75,  
    142.875, 144, 145.125, 146.25, 147.375, 148.5, 149.625, 150.75, 151.875, 
    153, 154.125, 155.25, 156.375, 157.5, 158.625, 159.75, 160.875, 162,     
    163.125, 164.25, 165.375, 166.5, 167.625, 168.75, 169.875, 171, 172.125, 
    173.25, 174.375, 175.5, 176.625, 177.75, 178.875, 180, 181.125, 182.25,  
    183.375, 184.5, 185.625, 186.75, 187.875, 189, 190.125, 191.25, 192.375, 
    193.5, 194.625, 195.75, 196.875, 198, 199.125, 200.25, 201.375, 202.5,   
    203.625, 204.75, 205.875, 207, 208.125, 209.25, 210.375, 211.5, 212.625, 
    213.75, 214.875, 216, 217.125, 218.25, 219.375, 220.5, 221.625, 222.75,  
    223.875, 225, 226.125, 227.25, 228.375, 229.5, 230.625, 231.75, 232.875, 
    234, 235.125, 236.25, 237.375, 238.5, 239.625, 240.75, 241.875, 243,     
    244.125, 245.25, 246.375, 247.5, 248.625, 249.75, 250.875, 252, 253.125, 
    254.25, 255.375, 256.5, 257.625, 258.75, 259.875, 261, 262.125, 263.25,  
    264.375, 265.5, 266.625, 267.75, 268.875, 270, 271.125, 272.25, 273.375, 
    274.5, 275.625, 276.75, 277.875, 279, 280.125, 281.25, 282.375, 283.5,   
    284.625, 285.75, 286.875, 288, 289.125, 290.25, 291.375, 292.5, 293.625, 
    294.75, 295.875, 297, 298.125, 299.25, 300.375, 301.5, 302.625, 303.75,  
    304.875, 306, 307.125, 308.25, 309.375, 310.5, 311.625, 312.75, 313.875, 
    315, 316.125, 317.25, 318.375, 319.5, 320.625, 321.75, 322.875, 324,     
    325.125, 326.25, 327.375, 328.5, 329.625, 330.75, 331.875, 333, 334.125, 
    335.25, 336.375, 337.5, 338.625, 339.75, 340.875, 342, 343.125, 344.25,  
    345.375, 346.5, 347.625, 348.75, 349.875, 351, 352.125, 353.25, 354.375, 
    355.5, 356.625, 357.75, 358.875 ;                                        

 lat = 89.1415194264611, 88.0294288679515, 86.910770814124, 85.7906288836365, 
    84.6699240844465, 83.5489469125421, 82.4278175240078, 81.3065945226689,   
    80.1853098724772, 79.0639824814086, 77.9426242466729, 76.8212430271002,   
    75.6998442220113, 74.5784316632959, 73.4570081455833, 72.3355757549091,   
    71.214136079887, 70.0926903516245, 68.9712395389358, 67.8497844146698,    
    66.7283256028822, 65.6068636130102, 64.4853988650433, 63.3639317083405,   
    62.2424624358907, 61.1209912952521, 59.9995184970404, 58.8780442215827,   
    57.7565686241836, 56.6350918393302, 55.5136139840772, 54.3921351607921,   
    53.2706554593984, 52.1491749592204, 51.0276937305087, 49.906211835711,    
    48.784729330535, 47.6632462648426, 46.5417626834057, 45.4202786265483,    
    44.298794130694, 43.1773092288349, 42.0558239509352, 40.9343383242788,    
    39.8128523737712, 38.6913661222016, 37.5698795904715, 36.4483927977945,   
    35.3269057618723, 34.2054184990488, 33.0839310244466, 31.962443352088,    
    30.8409554950018, 29.7194674653187, 28.5979792743565, 27.4764909326964,   
    26.3550024502506, 25.2335138363243, 24.1120250996709, 22.9905362485413,   
    21.8690472907301, 20.747558233616, 19.6260690841993, 18.5045798491365,    
    17.3830905347709, 16.2616011471617, 15.1401116921107, 14.018622175186,    
    12.8971326017452, 11.7756429769564, 10.6541533058176, 9.53266359317569,   
    8.41117384374318, 7.28968406211511, 6.16819425278443, 5.04670442015713,   
    3.92521456856645, 2.80372470228675, 1.68223482554707, 0.560744942544222,  
    -0.560744942544222, -1.68223482554707, -2.80372470228675,                 
    -3.92521456856645, -5.04670442015713, -6.16819425278443,                  
    -7.28968406211511, -8.41117384374318, -9.53266359317569,                  
    -10.6541533058176, -11.7756429769564, -12.8971326017452,                  
    -14.018622175186, -15.1401116921107, -16.2616011471617,                   
    -17.3830905347709, -18.5045798491365, -19.6260690841993,                  
    -20.747558233616, -21.8690472907301, -22.9905362485413,                   
    -24.1120250996709, -25.2335138363243, -26.3550024502506,                  
    -27.4764909326964, -28.5979792743565, -29.7194674653187,                  
    -30.8409554950018, -31.962443352088, -33.0839310244466,                   
    -34.2054184990488, -35.3269057618723, -36.4483927977945,

The lat should be -90 (or closest) to 90 (or closest) as below:

$ ncdump -c /cmip5_css01/scratch/cmip5/output/CSIRO-QCCCE/CSIRO-Mk3-6-0/sstClim4xCO2/fx/atmos/fx/r0i0p0/v20111222/sftlf/sftlf_fx_CSIRO-Mk3-6-0_sstClim4xCO2_r0i0p0.nc | more                                                                               
netcdf sftlf_fx_CSIRO-Mk3-6-0_sstClim4xCO2_r0i0p0 {                                                                                     
dimensions:                                                                                                                             
        lat = 96 ;                                                                                                                      
        lon = 192 ;                                                                                                                     
        bnds = 2 ;                                                                                                                      
variables:                                                                                                                              
        double lat(lat) ;                                                                                                               
                lat:bounds = "lat_bnds" ;                                                                                               
                lat:units = "degrees_north" ;                                                                                           
                lat:axis = "Y" ;                                                                                                        
                lat:long_name = "latitude" ;                                                                                            
                lat:standard_name = "latitude" ;                                                                                        
        double lat_bnds(lat, bnds) ;                                                                                                    
        double lon(lon) ;                                                                                                               
                lon:bounds = "lon_bnds" ;                                                                                               
                lon:units = "degrees_east" ;                                                                                            
                lon:axis = "X" ;                                                                                                        
                lon:long_name = "longitude" ;                                                                                           
                lon:standard_name = "longitude" ;                                                                                       
        double lon_bnds(lon, bnds) ;                                                                                                    
        float sftlf(lat, lon) ;                                                                                                         
                sftlf:standard_name = "land_area_fraction" ;                                                                            
                sftlf:long_name = "Land Area Fraction" ;                                                                                
                sftlf:units = "%" ;                                                                                                     
                sftlf:original_name = "sftlf" ;                                                                                         
                sftlf:cell_measures = "area: areacella" ;                                                                               
                sftlf:missing_value = 1.e+20f ;                                                                                         
                sftlf:_FillValue = 1.e+20f ;                                                                                            
                sftlf:associated_files = "baseURL: http://cmip-pcmdi.llnl.gov/CMIP5/dataLocation gridspecFile: gridspec_atmos_fx_CSIRO-M
k3-6-0_sstClim4xCO2_r0i0p0.nc areacella: areacella_fx_CSIRO-Mk3-6-0_sstClim4xCO2_r0i0p0.nc" ;                                           

// global attributes:
...
                :cmor_version = "2.5.9" ;
                :version_number = "v20110518" ;
data:

 lat = -88.5721969604492, -86.7225036621094, -84.8619995117188,
    -82.9989013671875, -81.1350021362305, -79.2705993652344,
    -77.4058990478516, -75.5410995483398, -73.6761016845703,
    -71.8110961914062, -69.9460983276367, -68.0810012817383,
    -66.2158966064453, -64.350700378418, -62.4855995178223,
    -60.6203994750977, -58.755199432373, -56.8899993896484,
    -55.0247993469238, -53.1595993041992, -51.2943992614746, -49.42919921875,
    -47.5638999938965, -45.6986999511719, -43.8334999084473,
    -41.9682006835938, -40.1030006408691, -38.2377014160156,
    -36.372501373291, -34.5071983337402, -32.6419982910156,
    -30.7766990661621, -28.9115009307861, -27.046199798584,
    -25.1809997558594, -23.3157005310059, -21.4505004882812,
    -19.5851993560791, -17.7199993133545, -15.854700088501,
    -13.9893999099731, -12.1241998672485, -10.2588996887207,
    -8.39367008209229, -6.52840995788574, -4.6631498336792,
    -2.79788994789124, -0.932630002498627, 0.932630002498627,
    2.79788994789124, 4.6631498336792, 6.52840995788574, 8.39367008209229,
    10.2588996887207, 12.1241998672485, 13.9893999099731, 15.854700088501,
    17.7199993133545, 19.5851993560791, 21.4505004882812, 23.3157005310059,
    25.1809997558594, 27.046199798584, 28.9115009307861, 30.7766990661621,
    32.6419982910156, 34.5071983337402, 36.372501373291, 38.2377014160156,
    40.1030006408691, 41.9682006835938, 43.8334999084473, 45.6986999511719,
    47.5638999938965, 49.42919921875, 51.2943992614746, 53.1595993041992,
    55.0247993469238, 56.8899993896484, 58.755199432373, 60.6203994750977,
    62.4855995178223, 64.350700378418, 66.2158966064453, 68.0810012817383,
    69.9460983276367, 71.8110961914062, 73.6761016845703, 75.5410995483398,
    77.4058990478516, 79.2705993652344, 81.1350021362305, 82.9989013671875,
    84.8619995117188, 86.7225036621094, 88.5721969604492 ;

[taylor13]

I thought we trapped both types of errors: 1) coordinates that do not increase or decrease monotonically, and 2) coordinates increasing in the right direction. How do you explain the above results?

[durack1]

@taylor13 remember this is CMOR 2.7.1 so @dnadeau4 has no need to explain these results.. We just need to make sure that CMOR3 doesn't write such data without notifying the user..

[taylor13]

Understood, but I'm pretty sure both CMOR1 & CMOR2 included these checks, so it would be nice to double check those parts of the code are intact and working properly.

[dnadeau4]

@durack1 I cannot change the input values from the data providers. What you see is they did not care enough to round their lat/lon values to an appropriate values. For projections in "meters" it make senses to have such precise lat/lon or for land people working on a small area.

[durack1]

@dnadeau4 I never wanted you to change values, just throw a warning for such cases.. It's best to not let these kind of errors propagate without notifying the user

[dnadeau4]

@durack1 how many decimal should I used to trigger a warning?

[durack1]

@dnadeau4 I think the same number as your previous fix (for cdms), was that 4 (https://github.com/PCMDI/cmor/issues/146)?

[dnadeau4]

@durack1 your upside-down picture looks like a vcs issue, the lat/lon are find but they have 80S on the top. not sure why.

[durack1]

@dnadeau4 I think specifying another warning would be a good idea, so warning that the left, lower (index 0,0) value is starts at the southern-most bound.. That is uniform across all CMIP5 contributed data APART from this model

[dnadeau4]

@doutriaux1 why vcs did not flip the picture above?

[taylor13]

@dnadeau4 @durack1 Note that the first file above was not written my CMOR. CMOR would have trapped this error (both then and now.) PrePARE, on the other hand isn't yet checking the order of coordinates, so wouldn't notice this problem.

I don't know why VCS isn't smart enough to put the S. Pole at the bottom of the plot.

I don't think there is anything we need to do now as far as modifying CMOR or PrePARE, but sometime in the next 6 months it would be nice for PrePARE to trap this sort of thing.

[dnadeau4]

This is a VCS problem

[durack1]

@dnadeau4 the issue is that CMOR 2.7.1 allowed the invalid grid to be written to a file see above, I'll set it for the next milestone - I wanted to make sure that such issues were caught in CMOR 3.2.3+ and such invalid grids are not generated by CMOR

[durack1]

@dnadeau4 for the top problem, all lats are identical:

In [35]: a = [-87.8638000488281, -87.8638000488281, -87.8638000488281,
    ...:     -87.8638000488281, -87.8638000488281, -87.8638000488281,
    ...:     -87.8638000488281, -87.8638000488281, -87.8638000488281,
    ...:     -87.8638000488281, -87.8638000488281, -87.8638000488281,
...
    ...:     -87.8638000488281, -87.8638000488281, -87.8638000488281, -87.863800048828]
In [36]: 
In [37]: for count in range(0,len(a)):
    ...:     print a[count]-a[count+1]   
0.0
0.0
0.0
...

So that's a simple problem

[dnadeau4]

@taylor13 PrePARE was meant to be for metadata not data. This is beyond PrePARE scope in my opinion.

[durack1]

@dnadeau4 for the second one, the lat should be stored smallest (southern negative value) to largest (northern positive value), if the order is not this way an error could be thrown..

[taylor13]

@durack1 All versions of CMOR would trap (and I think correct) this error. PrePARE is currently checking file-level metadata (not variable or dimension stuff). There are lots of additional things for PrePARE to check, but for now we need to make sure it is working flawlessly on the things it checks.

[durack1]

@taylor13 if you look at the global_atts for the first file, it indicates that :cmor_version = "2.7.1" was used to write the file.. Admittedly this is not CMOR3, but I am curious if a test in 2.7.1 which appears to not have worked (with this file the issue is all lat values are identical) is now functioning correctly.. I agree that the EC-EARTH (upside down lat) data was not written using CMOR so for that reason the only way that you could catch such a problem is by augmenting PrePARE to also check these details

[taylor13]

Presence of "cmor_version="2.7.1" doesn't guarantee use of CMOR. That being said, perhaps we should check that the code checks monotonicity by requiring a coord. value to be ">" and not just ">=" the previous coord. value.

[dnadeau4]

CMOR check if every axis it creates is monotonic. https://github.com/PCMDI/cmor/blob/master/Src/cmor_axes.c#L1363-L1364

[taylor13]

Reviewing Charle's code, I see in line 737:
if( ( ( values[i] - values[i + 1] ) / ( values[i + 1] - values[i + 2] ) ) < 0. )

This leads to a divide by zero if consecutive values are equal. Could this explain why the error in the output file wasn't caught? One fix would be to replace "/" with "*" (multiply instead of divide) and replace "<" with "<=".

Also, will there be a problem if the length of the coordinate variable is less than 3? Won't values[I+2] be undefined in this case? Or is the "for loop" skipped in this case?

Denis, could you please review this and perhaps also prepare a test that would check what happens if you pass coordinate values that are identical?

[dnadeau4]

@doutriaux1

[dnadeau4]

Fixed monotonic issue in 3.2.6. @durack1 @doutriaux1 any idea on the reverse image?

[durack1]

@dnadeau4 the upside down image https://github.com/PCMDI/cmor/issues/148#issuecomment-289610732 above would just be a check to make sure that latitude values increase monotonically, rather than decreasing

[doutriaux1]

@durack1 @dnadeau4 data are stored from North to South, so vcs plots them reversed. That's what the user asked to plot.

[dnadeau4]

I think the user is right and wanted that picture reverse. I cannot change his decision, his latitudes are valid in a CF-1 compliance way.

[durack1]

@dnadeau4 @doutriaux1 this is against the standard convention of CMIP data (the 30+ models in the CMIP5 suite vs this one), it's not wrong, just not standard. As I noted above, I think a warning would be useful, so that the user is made aware they are going again the standard protocol, however the data are not an error

[doutriaux1]

@durack1 I was just talking in the context of vcs. CMIP does require South to North so I'm assuming CMOR will flip the latitudes. Correct @dnadeau4 ?

[dnadeau4]

Karl and I verified that CMOR check for South to North and CMOR will change the axis direction. There is metadata called "stored_direction" which is set to "increasing". @doutriaux1 must remember it.

[dnadeau4]

https://github.com/PCMDI/cmor/blob/master/Src/cmor_axes.c#L804-L831

[taylor13]

All CMOR issues raised here have been addressed. Let's close it.