IMOS check geospatial_vertical_min/max should take into account _FillValue

[ggalibert]

geospatial_vertical_max            :3:     1/ 2 :  
    match_data                     :3:     0/ 1 : geospatial_vertical_max
                                                  value (0.0309531) did not
                                                  match maximum value of any
                                                  vertical variable
                                                  {u'DEPTH': 999999.0,
                                                  u'NOMINAL_DEPTH': 3.0}

999999.0 is actually the defined _FillValue and so should not be included as the maximum value of DEPTH.

[mhidas]

@ggalibert Can you please provide a sample file that shows this issue?

I'm a bit puzzled because I thought the NetCDF4 library automatically maps fill values to NaNs... but maybe not.

[ggalibert]

http://thredds.aodn.org.au/thredds/catalog/IMOS/eMII/checker_test/LCJO/catalog.html?dataset=eMII/checker_test/LCJO/IMOS_SRS-OC-LJCO_KOSTUZ_20140109T052647Z_SRC_FV01_WQM-hourly_C-20151102T043252Z.nc

ncdump shows:

...
    float DEPTH(TIME) ;
        DEPTH:_FillValue = 999999.f ;
        DEPTH:ancillary_variables = "DEPTH_quality_control" ;
        DEPTH:comment = "depthPP: Depth computed using the Gibbs-SeaWater toolbox (TEOS-10) v3.02 from latitude and relative pressure measurements (calibration offset usually performed to balance current atmospheric pressure and acute sensor precision at a deployed depth)." ;
        DEPTH:coordinates = "TIME LATITUDE LONGITUDE NOMINAL_DEPTH" ;
        DEPTH:long_name = "actual depth" ;
        DEPTH:max_depth_mismatch = 15. ;
        DEPTH:max_knock_down_angle = 70. ;
        DEPTH:positive = "down" ;
        DEPTH:reference_datum = "sea surface" ;
        DEPTH:standard_name = "depth" ;
        DEPTH:units = "metres" ;
        DEPTH:valid_max = 12000. ;
        DEPTH:valid_min = -5. ;
...
 DEPTH = _, 0.0309531, 0.0309531, 0.0309531, 0.0309531, 0.0309531, 0.0309531, 
    0.0309531, 0.0309531, 0.0309531, 0.0309531, 0.0309531, 0.0309531, 
    0.0309531, 0.0309531, 0.0309531, 0.0309531, 0.0309531, 0.0309531, 
    0.0309531, 0.0309531, 0.0309531, 0.0309531, 0.0309531, 0.0309531, 
    0.0309531, 0.0309531, 0.0309531, 0.0309531, 0.0309531, 0.0309531, 
    0.0309531, 0.0309531, _, 0.0309531, 0.0309531, 0.0309531, 0.0309531, 
    0.0309531 ;
}

[ggalibert]

The NetCDF4 python library does map fill values to NaNs:

>>> import netCDF4
>>> ds = netCDF4.Dataset('http://thredds.aodn.org.au/thredds/dodsC/IMOS/eMII/checker_test/LCJO/IMOS_SRS-OC-LJCO_KOSTUZ_20140109T052647Z_SRC_FV01_WQM-hourly_C-20151102T043252Z.nc')
>>> print ds.variables['DEPTH']
<type 'netCDF4._netCDF4.Variable'>
float32 DEPTH(TIME)
    _FillValue: 999999.0
    ancillary_variables: DEPTH_quality_control
    comment: depthPP: Depth computed using the Gibbs-SeaWater toolbox (TEOS-10) v3.02 from latitude and relative pressure measurements (calibration offset usually performed to balance current atmospheric pressure and acute sensor precision at a deployed depth).
    coordinates: TIME LATITUDE LONGITUDE NOMINAL_DEPTH
    long_name: actual depth
    max_depth_mismatch: 15.0
    max_knock_down_angle: 70.0
    positive: down
    reference_datum: sea surface
    standard_name: depth
    units: metres
    valid_max: 12000.0
    valid_min: -5.0
unlimited dimensions: 
current shape = (39,)
filling on
>>> print ds.variables['DEPTH'][:]
[-- 0.030953101813793182 0.030953101813793182 0.030953101813793182
 0.030953101813793182 0.030953101813793182 0.030953101813793182
 0.030953101813793182 0.030953101813793182 0.030953101813793182
 0.030953101813793182 0.030953101813793182 0.030953101813793182
 0.030953101813793182 0.030953101813793182 0.030953101813793182
 0.030953101813793182 0.030953101813793182 0.030953101813793182
 0.030953101813793182 0.030953101813793182 0.030953101813793182
 0.030953101813793182 0.030953101813793182 0.030953101813793182
 0.030953101813793182 0.030953101813793182 0.030953101813793182
 0.030953101813793182 0.030953101813793182 0.030953101813793182
 0.030953101813793182 0.030953101813793182 -- 0.030953101813793182
 0.030953101813793182 0.030953101813793182 0.030953101813793182
 0.030953101813793182]

Now careful, when performing numpy.min/max function, the content of a NetCDF variable must be properly referred to:

>>> import numpy
>>> print numpy.nanmin(ds.variables['DEPTH'])
0.0309531
>>> print numpy.nanmax(ds.variables['DEPTH'])
999999.0
>>> print numpy.nanmin(ds.variables['DEPTH'][:])
0.0309531
>>> print numpy.nanmax(ds.variables['DEPTH'][:])
0.0309531

Will make the correction for check_geospatial_vertical_min_max but might need to double check any other use of a variable content...

[ggalibert]

Please see PR https://github.com/aodn/compliance-checker/pull/92

[mhidas]

@ggalibert This was an issue with the IMOS checks, wasn't it? (title says CF) If so, it's fixed in #92 and we can close it.

[ggalibert]

Yes, my bad, it is not related to CF but IMOS.