Closed ggalibert closed 8 years ago
@ggalibert Can you please provide a sample file that shows this issue?
I'm a bit puzzled because I thought the NetCDF4 library automatically maps fill values to NaNs... but maybe not.
ncdump shows:
...
float DEPTH(TIME) ;
DEPTH:_FillValue = 999999.f ;
DEPTH:ancillary_variables = "DEPTH_quality_control" ;
DEPTH:comment = "depthPP: Depth computed using the Gibbs-SeaWater toolbox (TEOS-10) v3.02 from latitude and relative pressure measurements (calibration offset usually performed to balance current atmospheric pressure and acute sensor precision at a deployed depth)." ;
DEPTH:coordinates = "TIME LATITUDE LONGITUDE NOMINAL_DEPTH" ;
DEPTH:long_name = "actual depth" ;
DEPTH:max_depth_mismatch = 15. ;
DEPTH:max_knock_down_angle = 70. ;
DEPTH:positive = "down" ;
DEPTH:reference_datum = "sea surface" ;
DEPTH:standard_name = "depth" ;
DEPTH:units = "metres" ;
DEPTH:valid_max = 12000. ;
DEPTH:valid_min = -5. ;
...
DEPTH = _, 0.0309531, 0.0309531, 0.0309531, 0.0309531, 0.0309531, 0.0309531,
0.0309531, 0.0309531, 0.0309531, 0.0309531, 0.0309531, 0.0309531,
0.0309531, 0.0309531, 0.0309531, 0.0309531, 0.0309531, 0.0309531,
0.0309531, 0.0309531, 0.0309531, 0.0309531, 0.0309531, 0.0309531,
0.0309531, 0.0309531, 0.0309531, 0.0309531, 0.0309531, 0.0309531,
0.0309531, 0.0309531, _, 0.0309531, 0.0309531, 0.0309531, 0.0309531,
0.0309531 ;
}
The NetCDF4 python library does map fill values to NaNs:
>>> import netCDF4
>>> ds = netCDF4.Dataset('http://thredds.aodn.org.au/thredds/dodsC/IMOS/eMII/checker_test/LCJO/IMOS_SRS-OC-LJCO_KOSTUZ_20140109T052647Z_SRC_FV01_WQM-hourly_C-20151102T043252Z.nc')
>>> print ds.variables['DEPTH']
<type 'netCDF4._netCDF4.Variable'>
float32 DEPTH(TIME)
_FillValue: 999999.0
ancillary_variables: DEPTH_quality_control
comment: depthPP: Depth computed using the Gibbs-SeaWater toolbox (TEOS-10) v3.02 from latitude and relative pressure measurements (calibration offset usually performed to balance current atmospheric pressure and acute sensor precision at a deployed depth).
coordinates: TIME LATITUDE LONGITUDE NOMINAL_DEPTH
long_name: actual depth
max_depth_mismatch: 15.0
max_knock_down_angle: 70.0
positive: down
reference_datum: sea surface
standard_name: depth
units: metres
valid_max: 12000.0
valid_min: -5.0
unlimited dimensions:
current shape = (39,)
filling on
>>> print ds.variables['DEPTH'][:]
[-- 0.030953101813793182 0.030953101813793182 0.030953101813793182
0.030953101813793182 0.030953101813793182 0.030953101813793182
0.030953101813793182 0.030953101813793182 0.030953101813793182
0.030953101813793182 0.030953101813793182 0.030953101813793182
0.030953101813793182 0.030953101813793182 0.030953101813793182
0.030953101813793182 0.030953101813793182 0.030953101813793182
0.030953101813793182 0.030953101813793182 0.030953101813793182
0.030953101813793182 0.030953101813793182 0.030953101813793182
0.030953101813793182 0.030953101813793182 0.030953101813793182
0.030953101813793182 0.030953101813793182 0.030953101813793182
0.030953101813793182 0.030953101813793182 -- 0.030953101813793182
0.030953101813793182 0.030953101813793182 0.030953101813793182
0.030953101813793182]
Now careful, when performing numpy.min/max function, the content of a NetCDF variable must be properly referred to:
>>> import numpy
>>> print numpy.nanmin(ds.variables['DEPTH'])
0.0309531
>>> print numpy.nanmax(ds.variables['DEPTH'])
999999.0
>>> print numpy.nanmin(ds.variables['DEPTH'][:])
0.0309531
>>> print numpy.nanmax(ds.variables['DEPTH'][:])
0.0309531
Will make the correction for check_geospatial_vertical_min_max
but might need to double check any other use of a variable content...
Please see PR https://github.com/aodn/compliance-checker/pull/92
@ggalibert This was an issue with the IMOS checks, wasn't it? (title says CF) If so, it's fixed in #92 and we can close it.
Yes, my bad, it is not related to CF but IMOS.
999999.0 is actually the defined _FillValue and so should not be included as the maximum value of DEPTH.